A runway flatness deviation progressive analysis method and system based on time sequence image feature recognition

Abstract

The application discloses a runway flatness deviation progressive analysis method and system based on time sequence image feature recognition. Multi-source data including historical image sequence, initial position of feature points and synchronous three-dimensional deviation true value are obtained; based on time sequence motion of feature points in the historical image, implicit surface parameters representing microcosmic three-dimensional deformation of a pavement are inversely calculated under smoothness constraint; a conditional variation encoder is trained by using the parameters and real deviation data, and a deterministic mapping model from visual features to three-dimensional deviation is established; further, the model is corrected in perspective normalization by fusing camera network geometry configuration; and real-time image stream is input into the corrected model, real-time calculation of flatness deviation and its change rate is realized, and a high-risk area of accelerated deterioration is identified according to a threshold. The application realizes automatic, low-cost and continuous monitoring of runway flatness by using an existing monitoring camera network.

Description

Technical Field

[0001] This invention relates to the field of infrastructure health monitoring and operation and maintenance, specifically to a method and system for progressive analysis of runway smoothness deviation based on time-series image feature recognition. Background Technology

[0002] Runway pavement smoothness is a key indicator for ensuring safe aircraft takeoff and landing, improving passenger comfort, and extending runway lifespan. Traditional manual inspections and periodic special tests are insufficient to meet the modern operation and maintenance requirements for high-frequency, comprehensive, and real-time monitoring of runway conditions. In particular, they cannot capture the gradual, microscopic deformation process of pavement smoothness under operational loads and environmental factors, making it impossible to detect and warn of potential safety hazards in a timely manner.

[0003] Currently, quantitative inspection of runway smoothness mainly relies on periodic inspections by manually driven specialized inspection vehicles (such as laser profilers and inertial measurement systems). While this method can acquire high-precision three-dimensional profile data, it is limited by factors such as expensive equipment, the need for runway closure for operations, and long inspection cycles, making continuous real-time status monitoring impossible. In addition, some research has attempted to utilize sensors deployed in fixed locations or drones for aerial mapping. However, the former has high deployment and maintenance costs and limited coverage, while the latter is greatly affected by weather and airspace control, and struggles to achieve continuous data stream acquisition.

[0004] The aforementioned technical solutions generally have significant limitations: high-precision mobile detection methods are costly and cannot provide continuous real-time monitoring; while using fixed surveillance camera networks can provide continuous images, there is a lack of effective means to robustly and quantitatively infer the microscopic three-dimensional physical deformation of the runway surface and its rate of change over time from a two-dimensional image sequence from a fixed perspective. This makes it difficult for existing methods to achieve early warning and accurate location of the progressive deterioration of runway smoothness under conditions of no-stop operation and low cost. Summary of the Invention

[0005] In view of the above-mentioned actual situation, this application proposes a progressive analysis method and system for runway smoothness deviation based on temporal image feature recognition, in order to solve the technical problems existing in the prior art, such as high detection cost, inability to achieve continuous real-time monitoring, lack of early warning capability for the progressive deterioration process of pavement smoothness, and difficulty in robustly and quantitatively inverting three-dimensional physical deformation and its rate of change from a two-dimensional image sequence with a fixed viewpoint.

[0006] A progressive analysis method for runway smoothness deviation based on temporal image feature recognition, the method comprising the following steps: S1, acquire the data to be processed, which includes historical time-series image sequence data, initial projection data of feature points, known 3D deviation dataset, camera network geometric configuration data, and real-time time-series image stream data; the initial projection data of feature points is a set of predefined reference positions of runway texture feature points in the image coordinate system; the known 3D deviation dataset contains runway surface 3D flatness deviation values ​​measured synchronously with the historical time-series images; the camera network geometric configuration data describes the geometric relationship between the spatial position, viewing direction, and internal optical parameters of each camera; S2, perform implicit deformation surface inversion processing on the historical time-series image sequence data and the initial projection data of feature points to obtain implicit deformation surface parameters; the implicit deformation surface inversion processing is based on the time-series changes of the positions of two-dimensional feature points under a fixed viewpoint, and under the preset road surface deformation manifold smoothness constraint, solve the parameterized surface that characterizes the microscopic three-dimensional morphological changes of the road surface. S3, perform multi-view latent space mapping processing on the latent deformable surface parameters and the known three-dimensional deviation dataset to obtain a three-dimensional deviation mapping model; the multi-view latent space mapping processing is to construct a generative model with latent deformable surface parameters as observation variables, and establish a cross-modal deterministic mapping relationship by inferring the latent space representation shared with the real three-dimensional deviation. S4, Perform geometric correction processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain the corrected deviation mapping model; the geometric correction processing is to use the camera geometric parameters to adjust the viewpoint of the mapping model to eliminate the influence of camera network configuration differences on deviation estimation. S5, perform progressive deviation analysis on the corrected deviation mapping model and real-time time-series image stream data to obtain high-risk progressive damage area identification; the progressive deviation analysis process involves inputting the real-time image stream into the correction model, calculating the runway smoothness deviation and its rate of change over time, and identifying progressive abnormal areas based on thresholds.

[0007] Furthermore, step S2 includes the following sub-steps: S201, the historical time-series image sequence data is processed to construct the optical flow field of feature points, thereby obtaining dense optical flow field data; the optical flow field construction process of feature points is based on the assumption of gray level conservation between adjacent image frames, and calculates the motion vector field formed by predefined pavement texture feature points in continuous time-series images. S202, the dense optical flow field data and the initial projection data of feature points are subjected to manifold embedding processing based on projection consistency to obtain implicit deformable surface parameters; the manifold embedding processing based on projection consistency regards the two-dimensional motion vector of each feature point as different observation projections of the same implicit three-dimensional deformable surface, and constructs and solves an optimization process with minimizing reprojection error and manifold curvature penalty term as the objective function, embedding and inverting the two-dimensional observations into a set of control point parameters of the three-dimensional deformable surface.

[0008] Furthermore, step S3 includes the following sub-steps: S301, perform spatiotemporal reference alignment processing on the implicit deformable surface parameters and the known three-dimensional deviation data to obtain a structured training sample set; the spatiotemporal reference alignment processing is to spatially stitch and temporally synchronize the implicit deformable surface parameters derived from different cameras at the same time section and the same road surface area, and associate them with the true value of the three-dimensional deviation of the section and the area to form a sample set with a unified spatiotemporal reference. S302, the structured training sample set is subjected to conditional variational encoder training to obtain a three-dimensional deviation mapping model; the conditional variational encoder training is based on the spliced ​​multi-view latent deformable surface parameters as conditions and the three-dimensional deviation true value as the reconstruction target, and the training model learns a conditional probability distribution. The decoder part of the model serves as a deterministic mapping function from latent parameters to three-dimensional deviation during inference.

[0009] Furthermore, step S4 includes the following sub-steps: S401, Perform parameter fusion processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain a fused parameter model; the parameter fusion processing involves embedding the camera geometric parameters as additional inputs into the mapping model, and adapting the model to a specific network geometry through retraining or parameter adjustment; S402, the fusion parameter model is validated to obtain the corrected bias mapping model; the validation process is to test the bias estimation accuracy of the fusion parameter model on an independent validation dataset to ensure its consistent performance under different perspectives.

[0010] Furthermore, step S5 includes the following sub-steps: S501, perform real-time feature extraction processing on the real-time time-series image stream data to obtain real-time feature point data; the real-time feature extraction processing is to apply a real-time feature detection algorithm to the current image frame to obtain the instantaneous image position of the runway surface texture feature points; S502, deviation estimation and rate detection processing are performed on the real-time feature point data and the corrected deviation mapping model to obtain the high-risk progressive damage area identification; the deviation estimation and rate detection processing is to use the corrected deviation mapping model to estimate the three-dimensional smoothness deviation from the real-time feature point data, calculate the rate of change of the deviation value within a continuous time window, and compare it with a preset threshold, and identify the pavement area with the rate exceeding the threshold as a high-risk progressive damage area.

[0011] Furthermore, the aforementioned feature point optical flow field construction process is based on the assumption of gray-level conservation between adjacent image frames. The calculation of the motion vector field formed by predefined pavement texture feature points in continuous temporal images includes optical flow constraint equation calculation and dense motion vector field optimization. The manifold embedding process based on projection consistency treats the two-dimensional motion vector of each feature point as different observation projections of the same implicit three-dimensional deformable surface. By constructing and solving an optimization process with minimizing reprojection error and manifold curvature penalty term as the objective function, the two-dimensional observation is embedded and inverted into a control point parameter set of the three-dimensional deformable surface, including deformable surface parameterization modeling and parameter inversion optimization.

[0012] Furthermore, the spatiotemporal reference alignment process involves spatially stitching and temporally synchronizing the latent deformable surface parameters derived from different cameras at the same time section and the same track area, and associating them with the three-dimensional deviation ground truth of that section and area to form a sample set with a unified spatiotemporal reference. This includes multi-source surface parameter spatial stitching and synchronization processing, as well as three-dimensional ground truth association and sample construction processing. The conditional variational encoder training process uses the stitched multi-view latent deformable surface parameters as conditions and the three-dimensional deviation ground truth as the reconstruction target to train the model to learn a conditional probability distribution. The decoder part of this model serves as a deterministic mapping function from latent parameters to three-dimensional deviations during inference, including variational inference and latent space learning processing, and deterministic mapping function extraction processing.

[0013] Furthermore, the parameter fusion process involves embedding camera geometric parameters as additional inputs into the mapping model, and adapting the model to a specific network geometry through retraining or parameter adjustment, including geometric parameter encoding and embedding processing and model adaptation training processing; the verification process involves testing the bias estimation accuracy of the fused parameter model on an independent verification dataset to ensure its consistent performance under different perspectives, including multi-view performance evaluation processing and model screening and confirmation processing.

[0014] Furthermore, the real-time feature extraction process applies a real-time feature detection algorithm to the current image frame to obtain the instantaneous image position of runway surface texture feature points, including real-time feature point detection and localization processing and feature coordinate serialization and encapsulation processing; the deviation estimation and rate detection processing uses a corrected deviation mapping model to estimate the three-dimensional smoothness deviation from real-time feature point data, calculates the rate of change of the deviation value within a continuous time window, compares it with a preset threshold, and identifies pavement areas with rates exceeding the threshold as high-risk progressive damage zones, including model-based real-time deviation estimation processing and temporal change rate analysis processing.

[0015] Furthermore, a progressive analysis system for runway smoothness deviation based on temporal image feature recognition is characterized in that the system comprises: The acquisition unit is used to acquire data to be processed, which includes historical time-series image sequence data, initial projection data of feature points, known 3D deviation dataset, camera network geometric configuration data, and real-time time-series image stream data. The initial projection data of feature points is a set of predefined reference positions of runway texture feature points in the image coordinate system. The known 3D deviation dataset contains runway surface 3D flatness deviation values ​​measured synchronously with the historical time-series images. The camera network geometric configuration data describes the geometric relationship between the spatial position, viewing direction, and internal optical parameters of each camera. The inversion unit is used to perform implicit deformation surface inversion processing on the historical time-series image sequence data and the initial projection data of feature points to obtain implicit deformation surface parameters. The implicit deformation surface inversion processing is based on the time-series changes of the positions of two-dimensional feature points under a fixed viewpoint, and under the preset road surface deformation manifold smoothness constraint, to solve the parameterized surface that characterizes the microscopic three-dimensional morphological changes of the road surface. The mapping unit is used to perform multi-view latent space mapping processing on the latent deformable surface parameters and the known three-dimensional deviation dataset to obtain a three-dimensional deviation mapping model. The multi-view latent space mapping processing is to construct a generative model with the latent deformable surface parameters as observation variables, and establish a cross-modal deterministic mapping relationship by inferring the latent space representation shared with the real three-dimensional deviation. The correction unit is used to perform geometric correction processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain the corrected deviation mapping model. The geometric correction processing is to use the camera geometric parameters to adjust the viewpoint of the mapping model to eliminate the influence of camera network configuration differences on deviation estimation. The analysis unit is used to perform progressive deviation analysis on the corrected deviation mapping model and real-time time-series image stream data to obtain high-risk progressive damage area identification; the progressive deviation analysis process involves inputting the real-time image stream into the correction model, calculating the runway smoothness deviation and its rate of change over time, and identifying progressive abnormal areas based on thresholds.

[0016] This application proposes a progressive analysis method and system for runway smoothness deviation based on temporal image feature recognition. This method enables automated, low-cost, and continuous monitoring of runway smoothness using existing surveillance camera networks. By inverting implicit deformation surfaces from temporal images and establishing their mapping relationship with three-dimensional physical deviations, it completes the transformation from two-dimensional visual observation to three-dimensional quantitative assessment. Furthermore, by analyzing the temporal rate of change of deviation values, it achieves early identification and precise positioning of high-risk progressive damage areas, providing data-driven decision support for runway preventive maintenance. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method flow for a progressive analysis method of runway smoothness deviation based on temporal image feature recognition proposed in this application; Figure 2 A schematic diagram of a progressive analysis system for runway smoothness deviation based on temporal image feature recognition provided in this application embodiment; Figure 3 A schematic diagram illustrating the relationship between multi-view observation projection and implicit deformation surface inversion; Figure 4 This is a schematic diagram illustrating the temporal variation of flatness deviation and the identification of gradual risks. Detailed Implementation

[0018] The simulation technology route in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The features and performance of the present invention will be further described in detail below with reference to embodiments. Please refer to the appendix. Figure 1 As shown, a progressive analysis method for runway smoothness deviation based on temporal image feature recognition is proposed. The method includes the following steps: S1, acquire the data to be processed, which includes historical time-series image sequence data, initial projection data of feature points, known 3D deviation dataset, camera network geometric configuration data, and real-time time-series image stream data; the historical time-series image sequence data is a set of continuous runway surface images acquired by a fixed camera network within a historical period; the initial projection data of feature points is a set of predefined reference positions of runway texture feature points in the image coordinate system; the known 3D deviation dataset contains runway surface 3D smoothness deviation values ​​measured synchronously with the historical time-series images; the camera network geometric configuration data describes the geometric relationship between the spatial position, viewing direction, and internal optical parameters of each camera; the real-time time-series image stream data is a current runway surface image sequence acquired in real-time or periodically by the same camera network.

[0021] In this embodiment, the historical time-series image sequence data is continuously acquired by a fixed camera network that has been spatially calibrated and time-synchronized, within a historical period covering the target area of ​​the runway surface. This data constitutes a time-indexed two-dimensional image set, essentially a digital sequence recording the optical reflection of the runway surface texture and micromorphology from a specific perspective as it changes over time. Furthermore, the initial projection data of the feature points is a set of predefined reference positions of runway texture feature points in the image coordinate system. It should be noted that these feature points are defined and detected based on the inherent texture of the runway surface or artificially placed markers; their image coordinates at the initial reference time or under ideal flat conditions constitute the geometric starting reference for subsequent analysis of all two-dimensional relative motions.

[0022] In some implementations, the known 3D deviation dataset is a set of runway pavement 3D smoothness deviation values ​​obtained through time and space-strictly synchronized measurements with historical time-series image data. This dataset is acquired using high-precision contact or non-contact 3D measuring instruments, such as laser profilers or total stations, providing the physical deviation between the actual 3D elevation of the pavement and the standard design elevation at the corresponding time and region in the historical images. The known 3D deviation dataset and the historical time-series image data form a one-to-one supervised learning sample pair, serving as a bridge to correlate visual observation information with physical world measurements, and is an indispensable source of ground truth labels for training subsequent 3D deviation mapping models.

[0023] In this embodiment, the camera network geometric configuration data is a mathematical description of the spatial arrangement and internal optical characteristics of the fixed camera network. Specifically, the data describes the spatial position coordinates of each camera in a unified world coordinate system, the spatial viewing angle direction of the optical principal axis, and internal optical parameters such as focal length, principal point, and distortion coefficient. It should be noted that this geometric configuration data is obtained in advance through a rigorous network calibration procedure. It defines the projective geometric relationship between each two-dimensional image pixel and a point in the three-dimensional track space, serving as the fundamental basis for subsequent multi-view data fusion, implicit surface inversion, and geometric correction processing.

[0024] Preferably, the real-time temporal image stream data consists of a sequence of current runway surface images continuously acquired in real-time or periodically triggered by a fixed camera network identical to the historical data acquisition network. This data is a continuation of the historical temporal image sequence data on the time axis, maintaining consistency in data format, spatial coverage, and acquisition perspective with the historical data. This ensures that the analysis model trained based on historical data can be directly applied to the real-time data stream. Specifically, the real-time temporal image stream data, as the online input of the entire analysis system, carries visual information reflecting the current state of the runway surface and serves as the direct data source for performing progressive deviation analysis and identifying high-risk areas. These five types of data form a whole: historical data is used to construct and train the analysis model, real-time data is used for online model inference, and feature point, 3D ground truth, and geometric configuration data provide the model with benchmarks, supervision, and constraints for transformation and learning from the 2D image domain to the 3D physical domain. Together, they constitute the data closed loop for implementing the progressive analysis method for runway smoothness deviation based on temporal image feature recognition.

[0025] S2, perform implicit deformation surface inversion processing on the historical time-series image sequence data and the initial projection data of feature points to obtain implicit deformation surface parameters; the implicit deformation surface inversion processing is based on the time-series changes of the positions of two-dimensional feature points under a fixed viewpoint, and under the preset road surface deformation manifold smoothness constraint, solve the parameterized surface that characterizes the microscopic three-dimensional morphological changes of the road surface. Please see the appendix Figure 3As shown in the figure, a unified world coordinate system is displayed, where the gray grid plane represents the ideal pavement reference plane. Above this reference plane, a continuous, undulating, blue translucent surface represents the microscopic three-dimensional deformation of the runway pavement that exists in reality but cannot be directly observed—the implicit deformation surface to be inverted. Multiple predefined texture feature points are distributed on this surface. Two cameras (Camera A and Camera B) fixed at different spatial locations observe this surface from their respective perspectives. Each camera projects the feature points on the three-dimensional surface onto its corresponding two-dimensional imaging plane through its imaging geometry, forming two-dimensional image coordinate observation points. The projection path is illustrated in the figure by a colored dashed line from the camera to the feature point and then to the imaging plane. The core of the implicit deformable surface inversion process lies in: based on the time-series changes of the two-dimensional feature point positions (i.e., the two-dimensional motion vector field) calculated from historical time-series image sequences, and according to the inherent geometric relationship of the camera, an optimization process is constructed and solved to minimize the positional error (reprojection error) between the two-dimensional observation point and the theoretical point obtained by reprojection from the three-dimensional deformable surface, while constraining the smoothness of the curvature change of the three-dimensional deformable surface (manifold curvature penalty) as the objective function. This process fuses and inverts discrete two-dimensional motion observations from multiple perspectives into a continuous, parameterized three-dimensional deformable surface estimate, thus intuitively revealing the complete logic of the process in embedding and inverting multi-view, discrete two-dimensional apparent motion into a continuous parameterized description of the hidden three-dimensional physical deformation morphology through projection consistency constraints and smoothness priors. Specifically, this step includes the following sub-steps: S201, the historical time-series image sequence data is processed to construct the optical flow field of feature points, thereby obtaining dense optical flow field data; the optical flow field construction process of feature points is based on the assumption of gray level conservation between adjacent image frames, and calculates the motion vector field formed by predefined pavement texture feature points in continuous time-series images. In some implementations, the feature point optical flow field construction process is based on the assumption of gray-level conservation between adjacent image frames. Calculating the motion vector field formed by predefined pavement texture feature points in continuous temporal images includes optical flow constraint equation calculation and dense motion vector field optimization. Specifically, the optical flow constraint equation calculation establishes a partial differential constraint relationship between image pixel brightness and motion vectors based on the gray-level conservation assumption; the dense motion vector field optimization, based on this constraint relationship, introduces spatial consistency or smoothness priors to solve for the motion vector corresponding to each pixel or feature point covering the entire region of interest.

[0026] In this embodiment, the feature point optical flow field construction process involves maintaining the spatial position and viewing direction of the fixed camera network constant within the analysis period. The apparent displacement of the pavement texture feature points observed in the image sequence does not originate from the macroscopic motion of the camera or the target object, but directly maps to the microscopic three-dimensional deformation of the runway pavement itself. This microscopic deformation causes subtle changes in the spatial position of local areas of the pavement, which, under a fixed imaging projection geometry, manifests as a temporal shift in the position of feature points on the image plane. The processing systematically captures and quantifies this projection trajectory change, driven by implicit three-dimensional physical changes and perceptible in the two-dimensional image sequence, into a complete and dense two-dimensional motion vector field. It should be noted that each motion vector obtained in this way is not an independent and isolated image displacement measurement value, but rather a component of the projection observation of the same underlying implicit three-dimensional deformation surface at different spatiotemporal positions. The dense optical flow field data constitutes the most direct and necessary input observation data basis for the subsequent steps of inverting and reconstructing the implicit three-dimensional deformation surface of the runway surface from multiple perspectives and multiple time series of two-dimensional projection observations.

[0027] In this embodiment, the gray-level conservation assumption forms the physical basis for the calculation and processing of the optical flow constraint equation. The gray-level conservation assumption states that for the same tiny surface region on the runway surface, its gray-level conservation is proportional to the gray level in two consecutive frames. and In this context, although the position of the area in the image changes due to pavement deformation or relative motion, its reflected brightness or grayscale value remains constant over a short period. This assumption is expressed as... ,in This represents the point at the time interval. The image plane displacement vector within.

[0028] Furthermore, based on the aforementioned gray-level conservation assumption, a first-order Taylor series expansion is performed on the right-hand side of the equation, and the optical flow constraint equation is calculated and processed to derive the fundamental optical flow equation. This equation is... ,in and These represent the spatial gray-level gradients of the image in the x and y directions, respectively. Represents the grayscale gradient of an image over time; and This refers to the velocity of the pixel in the x and y directions, i.e., the optical flow vector. It should be noted that a single equation contains two unknowns, u and v, and the equation itself is ill-conditioned, making it impossible to directly solve for a unique motion vector.

[0029] Preferably, to address the aforementioned issues and acquire the dense optical flow field data, a dense motion vector field optimization process is introduced. This process is performed in the image domain. Construct a vector field for motion. Global energy functional Optimization is performed to find the solution. The energy functional includes a data term and a smoothing term: ,in This represents the spatial gray-level gradient of the image in the x-direction. This represents the spatial gray-level gradient of the image in the y-direction. The gray-level gradient of the image changes over time, where u represents the component of the optical flow vector in the x-direction and v represents the component of the optical flow vector in the y-direction. Represents the regularization parameter; Represents the gradient operator; data item Enforcing the grayscale conservation assumption; smoothing term A spatial continuity constraint is introduced, which assumes that the motion vectors of adjacent pixels are similar; parameters The weights are used to control the smoothing term. This is achieved by minimizing the energy functional. This allows us to obtain a globally consistent dense optical flow field that achieves an optimal balance between grayscale conservation and spatial smoothness. The calculation or processing method is well known to those skilled in the art, and will not be elaborated upon here.

[0030] In some implementations, considering that the runway surface texture feature points have been predefined in step S1, the processing can be performed directly on the local image regions where these feature points are located. The dense optical flow field data obtained by the processing is characterized as a time-varying two-dimensional vector field function. ,in This refers to the set of coordinates of feature points in the image, where t is the time index. Specifically, for any predefined feature point, it will be tracked in the time-series image, and a motion trajectory will be generated. The motion vector between adjacent frames is... The collection of all these vectors constitutes the dense optical flow field data. In this embodiment, it is emphasized again that, under the geometric constraints of a fixed camera viewpoint, the apparent displacement of texture feature points caused by the microscopic three-dimensional deformation of the runway surface itself is transformed into a two-dimensional motion vector field that can be quantified and analyzed. This process does not detect the actual motion of the object, but rather captures the changes in the visual projection trajectory caused by implicit deformation, providing a direct observational data basis for the subsequent inversion of the three-dimensional deformed surface from the two-dimensional projection sequence.

[0031] S202, the dense optical flow field data and the initial projection data of feature points are subjected to manifold embedding processing based on projection consistency to obtain implicit deformable surface parameters; the manifold embedding processing based on projection consistency regards the two-dimensional motion vector of each feature point as different observation projections of the same implicit three-dimensional deformable surface, and constructs and solves an optimization process with minimizing reprojection error and manifold curvature penalty term as the objective function, embedding and inverting the two-dimensional observations into a set of control point parameters of the three-dimensional deformable surface.

[0032] In some implementations, the manifold embedding process based on projection consistency treats the two-dimensional motion vectors of each feature point as different observation projections of the same implicit three-dimensional deformable surface. It involves constructing and solving an optimization process with the objective function of minimizing reprojection error and manifold curvature penalty. Embedding and inverting the two-dimensional observations into a control point parameter set for the three-dimensional deformable surface includes deformable surface parametric modeling and parameter inversion optimization. Specifically, the deformable surface parametric modeling defines a mathematical representation of the implicit three-dimensional deformable surface to be solved; the parameter inversion optimization constructs an objective function that integrates projection geometric constraints and prior deformation smoothness, and solves for the control point parameters of the surface using numerical optimization methods.

[0033] In this embodiment, the parametric modeling of the deformable surface forms the geometric representation basis of the inversion problem. The implicit three-dimensional deformable surface represents the deviation of the runway surface micromorphology from an ideal reference plane. Preferably, the surface is parametrically modeled using a non-uniform rational B-spline surface. The surface is in a two-dimensional parameter domain. The above expression is: , where the parameter domain coordinates The parameter positions used to locate points on the surface have no direct correspondence with (u,v) representing optical flow components in step S201; The set of control point parameters for the implicitly deformable surface, i.e., all control points. A set of values, where m and n are non-negative integers, representing the control points in the parameter directions, respectively. and The quantity on the surface is reduced by one, where i and j are integer indices of the control points; p and q respectively define the B-spline surface on the surface. direction and The order of the polynomial in the direction; Corresponding control points of, order is The bivariate orthogonal rational B-spline basis functions, whose values ​​are about the parameters scalar functions; These are the coordinates of control points in three-dimensional space, constituting the set of control point parameters to be solved. The parametric modeling discretizes the continuous deformable surface into a system composed of a finite number of control points. Defined, smooth, and continuous mathematical objects.

[0034] In this embodiment, the parameter inversion optimization process aims to find a set of optimal control point parameters. This ensures that the deformable surface defined by this parameter can explain the motion of all two-dimensional feature points observed in step S201 to the greatest extent. Specifically, for the i-th feature point observed in the t-th frame of the image, its two-dimensional position on the image is... Assume that this feature point corresponds to a three-dimensional point on the pavement. At the initial reference time (corresponding to the initial projection data of the feature points), it projects the model through the camera. Projected onto image coordinates ,Right now When the pavement changes from a curved surface When describing deformation, the three-dimensional point The spatial position will shift, and the amount of shift will be determined by the surface coordinates of that point in the parameter domain. The deformation at that location is determined, i.e., moving to Correspondingly, its theoretical projection position on the t-frame image should be... It should be noted that the specific form of the projection model is a standard geometric mapping relationship that can be directly established by those skilled in the art based on the spatial position, viewing direction, and internal optical parameters provided in the camera network geometric configuration data. The process of mapping points in the three-dimensional world coordinate system to the two-dimensional image coordinate system is common knowledge in the field of computer vision, and its specific formulas will not be elaborated here. In the parameter inversion optimization process, this model is directly applied to calculate the projection from three-dimensional points to two-dimensional image points.

[0035] Furthermore, based on the above projection relationship, the process constructs an objective function aimed at minimizing the reprojection error and the manifold curvature penalty term. The objective function is defined as follows: ,in Represents the control point parameter set Overall objective function value; data items The expression for the difference between the theoretical projected positions and the measured image positions of all feature points at all observation times is: t represents the time frame index, and i represents the feature point index. This represents the measured two-dimensional image coordinates of the i-th feature point in the t-th frame. This represents the regularization hyperparameter. This indicates that the i-th feature point in the t-th frame of the image is determined by the deformation surface parameters. The theoretical projected two-dimensional image coordinates are calculated using the camera projection model. This data item directly enforces projection consistency constraints, ensuring that the inverted deformed surface matches the observed two-dimensional motion vector. It should be noted that the measured image position sequence in the formula... Its relative to the initial position The displacement is directly derived from the dense optical flow field data calculated in step S201. The cumulative integral results at the corresponding feature points and time.

[0036] In this embodiment, the smoothing term As a manifold curvature penalty term, it is used to introduce the physical prior that pavement deformation should have spatial continuity and smoothness. The smoothness term is typically based on the deformed surface. The curvature or second derivative can be used to construct it, for example, by using thin-plate spline energy: Where S represents the parameterized implicit three-dimensional deformable surface, i.e. , Let S represent the coordinates of the deformable surface S in its two-dimensional parameter domain. This penalty term tends to produce deformable surfaces with gentle curvature changes, suppressing non-physically drastic deformations caused by observation noise and enhancing the well-posedness of the inversion problem. Regularization hyperparameter Used to balance data fitting accuracy and surface smoothness.

[0037] The parameter inversion optimization process uses numerical optimization algorithms, such as the Levenberg-Marquardt algorithm or the Gauss-Newton method, to solve for the objective function. Minimize the set of control point parameters ,Right now The solution to this optimization problem is well-known to those skilled in the art, and will not be elaborated upon here. The implicit deformable surface parameters obtained are the set of optimal control point coordinates obtained after optimization convergence. The parameter set The three-dimensional deformation morphology of the runway surface inferred from historical time-series imagery was encoded in a compact mathematical form, forming a crucial intermediate representation connecting two-dimensional visual observation and three-dimensional physical deformation. This step completed the transformation from two-dimensional displacement observation of multiple feature points at different times to a continuous and parametric description of the overall implicit three-dimensional deformation morphology of the runway surface, providing a vital parametric intermediate representation for subsequently establishing a quantitative mapping relationship between two-dimensional image features and three-dimensional physical deviations.

[0038] S3, perform multi-view latent space mapping processing on the latent deformable surface parameters and the known three-dimensional deviation dataset to obtain a three-dimensional deviation mapping model; the multi-view latent space mapping processing is to construct a generative model with latent deformable surface parameters as observation variables, and establish a cross-modal deterministic mapping relationship by inferring the latent space representation shared with the real three-dimensional deviation. Specifically, this step includes the following sub-steps: S301, perform spatiotemporal reference alignment processing on the implicit deformable surface parameters and the known three-dimensional deviation data to obtain a structured training sample set; the spatiotemporal reference alignment processing is to spatially stitch and temporally synchronize the implicit deformable surface parameters derived from different cameras at the same time section and the same road surface area, and associate them with the true value of the three-dimensional deviation of the section and the area to form a sample set with a unified spatiotemporal reference. The spatiotemporal reference alignment process involves spatially stitching and temporally synchronizing implicit deformation surface parameters derived from different cameras at the same time section and on the same track surface region. This is then correlated with the 3D deviation ground truth of that section and region, forming a sample set with a unified spatiotemporal reference. This includes multi-source surface parameter spatial stitching and synchronization processing, as well as 3D ground truth correlation and sample construction processing. Specifically, the multi-source surface parameter spatial stitching and synchronization processing transforms and fuses the locally deformed surface parameters obtained from inversions in different camera coordinate systems into a unified world coordinate system. The 3D ground truth correlation and sample construction processing, within a unified spatiotemporal framework, precisely matches the fused surface parameter representations with known 3D flatness deviation measurements to construct paired samples for training the mapping model.

[0039] In some implementations, the splicing and synchronization of multi-source surface parameters is the primary step in achieving data alignment. (Hidden deformable surface parameters) It is obtained independently inverted in step S202 based on the viewpoint of the k-th camera and its corresponding image sequence, and its defined deformation surface is... Naturally, it resides in the local coordinate system of the camera or a reference system associated with the camera's imaging geometry. For comprehensive analysis across different viewpoints, all... The coordinates are transformed into a common world coordinate system W that covers the entire runway surface area. This process relies on the known extrinsic parameters of each camera, namely the rotation matrix. Translation vector For a set of control points The coordinates of the control points of the defined surface in the world coordinate system are calculated through a rigid transformation: Through this transformation, the local deformation surfaces derived from all cameras are unified to the same spatial reference. Furthermore, time synchronization processing, targeting the same time segment t, extracts the deformation surface parameters corresponding to that time segment t from the historical time-series image sequences of each camera. Since the acquisition systems of each camera have been synchronized in time, the extraction operation ensures that the surface parameters stitched together in space correspond to the pavement state at the same physical moment.

[0040] In this embodiment, it should be noted that the rotation matrix used here... Translation vector The static calibration parameters describe the inherent spatial properties of the observation system. Their role in this step is to place observation data from different perspectives within a unified geometric reference frame, constituting a spatial normalization operation in data preprocessing. This is fundamentally different in nature and function from the known 3D deviation dataset, which will be associated later as a monitoring target. The known 3D deviation dataset consists of dynamic physical measurement truth values ​​acquired strictly synchronously with historical imagery. It carries the true smoothness status information of the runway surface at the corresponding time, constituting the monitoring signal and fundamental basis for the subsequent model learning of the mapping law from visual observation features to physical quantities.

[0041] Preferably, after completing spatial stitching and time synchronization, a fused deformable surface parameter representation corresponding to time section t in the world coordinate system is obtained, denoted as The three-dimensional truth correlation and sample construction process then incorporates the fusion parameters. It is correlated with a known 3D deviation dataset. This known 3D deviation dataset contains the true values ​​of 3D deviations of a specific area of ​​the pavement, acquired using high-precision measuring equipment at the same time cross section t. The true value data is represented as a set of discrete points. ,in These are the horizontal position coordinates in the world coordinate system. This is the elevation-direction flatness deviation value measured at time t. To establish the correlation, the deformed surface parameters need to be... The described deformation field, at the same world coordinate position The evaluation is performed on the above. Specifically, for each true value measurement point... Its parametric surface There are corresponding parameter coordinates on it. and deformation To account for elevation deviation Correlate the deformation and extract its vertical component (Z-axis in the world coordinate system), denoted as... Thus, a training sample is constructed as a mapping from the parameters of the fused surface to the true value of the deviation at that point: the input is a parameterized representation that can characterize the deformation features of the local region (e.g., using...). A subset of control points for a surface block centered on the global parameters. Feature vectors extracted The output is the corresponding true deviation value. The feature vector The extraction method is a technical detail that can be flexibly designed by those skilled in the art based on the selected machine learning model, and will not be elaborated on here.

[0042] By performing the above operations on all available time sections t and all ground truth measurement points l, the resulting structured training sample set is represented as a set. Each sample in this dataset is rigorously established on a unified spatiotemporal reference, ensuring the representation of "latent deformation" features retrieved from visual observations. Compared with the true value of "explicit physical deviation" obtained by direct measurement There is an accurate correspondence between them. This processing solves the challenge of benchmark unification in multi-source heterogeneous data fusion, providing a reliable and consistent data foundation for subsequent steps to train a mapping model that can robustly infer physical deviations from complex visual observations.

[0043] S302, the structured training sample set is subjected to conditional variational encoder training to obtain a three-dimensional deviation mapping model; the conditional variational encoder training is based on the spliced ​​multi-view latent deformable surface parameters as conditions and the three-dimensional deviation true value as the reconstruction target, and the training model learns a conditional probability distribution. The decoder part of the model serves as a deterministic mapping function from latent parameters to three-dimensional deviation during inference.

[0044] The conditional variational encoder training process uses the spliced ​​multi-view latent deformation surface parameters as conditions and the three-dimensional deviation true value as the reconstruction target to train the model to learn a conditional probability distribution. During inference, the decoder part of this model serves as a deterministic mapping function from latent parameters to three-dimensional deviations, including variational inference and latent space learning processes, and deterministic mapping function extraction processes. Specifically, the variational inference and latent space learning processes aim to construct and optimize a probabilistic generation model that models the generation process of the three-dimensional deviation true value as a probability distribution determined by continuous latent variables, dependent on the latent deformation observation conditions. The deterministic mapping function extraction process, after model training, fixes the structure and parameters of its decoder part, making it a deterministic function that directly maps from the conditional input to the estimated three-dimensional deviation value.

[0045] In some implementations, the variational inference and latent space learning processes constitute the statistical learning foundation for training the mapping model. The processing targets the structured training sample set obtained in step S301. ,in This represents the conditional feature vector extracted from the multi-view fused deformable surface parameters. This represents the corresponding three-dimensional bias true value. The goal of a conditional variational encoder is to model the conditional probability distribution. This model introduces a continuous latent random variable z and assumes that y is generated jointly by z and condition c, i.e., the generation process is as follows: The latent variable z is assumed to follow a standard normal prior distribution. The model consists of two core components: an encoder network. Used to approximate the true posterior distribution and a decoder network This is used to define the likelihood distribution of y generated from the latent variable z and the condition c. It should be noted that... This represents the set of parameters of the decoder network. denoted as the parameter set of the encoder network, y represents the true value of the three-dimensional bias, which is the reconstruction target of the model, and c represents the conditional feature vector extracted from the parameters of the multi-view fused deformed surface.

[0046] Furthermore, the training objective of the process is to minimize the conditional log-likelihood. The variational lower bound. This is the conditional log-likelihood of the model generating true 3D bias data. This lower bound consists of two terms. The first is the expectation of the reconstruction term, which calculates the mean log probability that the decoder successfully reconstructs the true 3D bias value based on the latent variables and conditional inputs, given the latent variable distribution inferred by the encoder. This term directly drives the model's ability to accurately recover target data using latent variables and conditional information. The second term is the KL divergence term, which measures the difference between the posterior distribution approximated by the encoder and the pre-defined standard normal prior distribution. Its function is to impose a regularization constraint on the latent space, making the posterior distribution closer to the prior, thereby ensuring a good structure in the latent space and avoiding overfitting. The encoder network is implemented by taking the true 3D bias value and conditional features as input, and outputting the parameters of a multivariate Gaussian distribution through neural network operations, namely the mean vector and the variance values ​​of each dimension in the diagonal covariance matrix. This means that the encoder maps each specific input pair to a probability distribution in the latent space. The decoder network takes latent variable samples and conditional features sampled from the posterior distribution as input, and its output defines a Gaussian likelihood distribution with the network's predicted values ​​as the mean. The variance of this distribution can be set as a fixed hyperparameter to simplify the model, or the decoder network can output an additional variance prediction value, allowing the model to learn the inherent uncertainty in the data. By maximizing this variational lower bound, the model is jointly optimized to simultaneously achieve accurate data reconstruction and latent space regularization.

[0047] In this embodiment, the optimization process of the processing iteratively updates the encoder parameters using the stochastic gradient descent method. and decoder parameters To maximize the variational lower bound L. During training, a reparameterization technique is used to reduce the encoder distribution. Mid-sampling is used for gradient backpropagation, a technique well-known to those skilled in the art, and will not be elaborated upon here. Through this training, the model learns a structured latent space that can effectively encode all potential, uncertain factors that may produce the observed three-dimensional deviation y given a specific latent deformation observation feature c.

[0048] Preferably, the deterministic mapping function extraction process is performed after the model training converges. The core of this process lies in utilizing the trained decoder network. A deterministic mapping function is constructed. Since the true value of the latent variable z is unknown during the inference phase, following the conventions of variational inference, the expectation of the latent variable under its prior distribution is taken as the input for inference. Therefore, the three-dimensional bias mapping model is described. It is defined as a function that takes the conditional feature c as input and the output of the decoder network when the latent variables are zero vectors: ,in It is a trained decoder neural network. The mapping model M constitutes a mapping from the multi-view latent deformable surface feature c to the three-dimensional flatness deviation estimate. Deterministic mapping: .

[0049] It should be noted that the conditional variational encoder training process learns the potentially ambiguous mapping relationship between complex visual inversion features and physical deviations through a probabilistic generation framework, and ensures the robustness and generalization ability of the learned mapping by regularizing constraints on the latent space. The final extracted decoder function M discards the randomness in the generation process, providing a stable and efficient deterministic inference path. The acquisition of the three-dimensional deviation mapping model M marks the initial establishment of a quantitative and automated inference chain from the two-dimensional motion information contained in historical time-series images to the three-dimensional physical smoothness state of the pavement, providing a core inference engine for subsequent geometric correction and real-time progressive analysis.

[0050] S4, Perform geometric correction processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain the corrected deviation mapping model; the geometric correction processing is to use the camera geometric parameters to adjust the viewpoint of the mapping model to eliminate the influence of camera network configuration differences on deviation estimation. Specifically, this step includes the following sub-steps: S401, Perform parameter fusion processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain a fused parameter model; the parameter fusion processing involves embedding the camera geometric parameters as additional inputs into the mapping model, and adapting the model to a specific network geometry through retraining or parameter adjustment; The parameter fusion process involves embedding camera geometric parameters as additional inputs into the mapping model. Through retraining or parameter adjustment, the model adapts to a specific network geometry. This includes geometric parameter encoding and embedding, and model adaptation training. Specifically, the geometric parameter encoding and embedding process aims to transform the camera's spatial position, viewing direction, and internal optical parameters into a standardized feature representation suitable for neural network processing. The model adaptation training process uses these geometric features as additional conditions to fine-tune or supplement the existing 3D deviation mapping model, enabling the model's mapping behavior to explicitly adapt to and correct the systematic effects of different observation geometries.

[0051] In some implementations, the geometric parameter encoding and embedding processing are prerequisites for effective fusion. The camera network geometric configuration data describes the inherent attributes of each camera, including its extrinsic and intrinsic parameters. Preferably, the rotation information in the extrinsic parameters is converted into a low-dimensional numerical representation, while the translation information directly uses its three-dimensional coordinate values. The intrinsic parameters and lens distortion coefficients, after numerical normalization, are concatenated with the aforementioned extrinsic parameter representation to form a fixed-dimensional digital feature vector characterizing the geometric attributes of the camera. When the input data processed by the model fuses observation features from multiple cameras, the corresponding geometric input is an aggregated geometric feature formed by weighting or directly concatenating the geometric feature vectors of the involved cameras.

[0052] In this embodiment, the model adaptation training process updates the parameters of the 3D bias mapping model based on the expanded training data. The process expands the structured training sample set obtained in the previous steps, adding corresponding, encoded camera network geometric features to each sample, thus forming a new triplet training sample set containing conditional features, geometric features, and the true bias value. It should be noted that the geometric features introduced here represent the inherent, static spatial configuration attributes of the observation system itself, rather than dynamic observation data. The architecture of the 3D bias mapping model is modified accordingly, expanding its input from the original conditional feature vector to a joint feature vector formed by concatenating the conditional feature vector and the geometric feature vector. The functional form of the model is updated accordingly, and its internal adjustable parameter set is also adjusted to accommodate the new input dimension.

[0053] Furthermore, the optimization objective of the model adaptation training process is to minimize the difference between the predicted 3D flatness deviation value and the actual deviation value on the expanded training sample set. The process employs a supervised learning paradigm and preferably utilizes fine-tuning strategies from transfer learning: using the previously converged 3D deviation mapping model parameters as the initialization starting point, only the parameters of the network layers directly connected to the newly added geometric feature inputs and the parameters of several subsequent layers are updated and adjusted. Meanwhile, the parameters of the underlying layers of the model, which are abstract representations extracted from conditional features, are frozen, thereby preserving the core feature recognition capabilities about deformation patterns already learned by the model. The training process continuously adjusts the model parameters to reduce prediction errors through an iterative optimization algorithm. This optimization algorithm is well-known to those skilled in the art and its implementation will not be elaborated upon here.

[0054] Preferably, a fusion parameter model is obtained after the model adaptation training process. When receiving input, this model needs to simultaneously provide the implicit deformation feature representation obtained from image inversion and the corresponding camera geometric feature representation. The fusion parameter model systematically injects the inherent geometric properties of the camera as prior knowledge into the trained data-driven model, so that the general mapping relationship originally based on statistical association learning is explicitly modified into a physically constrained mapping dependent on specific observation geometric conditions. This process realizes the transformation from data fitting to physically guided model adaptation, ensuring that the same model can compensate for the difference in perspective by introducing corresponding geometric features when facing data collected by cameras from different spatial locations and perspectives, thereby producing a consistent and accurate estimate of the same physical deformation, improving the generalization ability and reliability of the entire analysis system in actual heterogeneous deployment environments.

[0055] S402, the fusion parameter model is validated to obtain the corrected bias mapping model; the validation process is to test the bias estimation accuracy of the fusion parameter model on an independent validation dataset to ensure its consistent performance under different perspectives.

[0056] The verification process involves testing the bias estimation accuracy of the fusion parameter model on an independent verification dataset to ensure its consistent performance across different viewpoints. This includes multi-view performance evaluation and model selection and verification. Specifically, the multi-view performance evaluation systematically quantifies the bias estimation error and inter-view estimation differences of the fusion parameter model on a spatiotemporally aligned verification dataset independent of the model training process. The model selection and verification process judges the evaluation results according to preset performance threshold standards to confirm whether the fusion parameter model meets the cross-view consistency requirements and obtains the corrected bias mapping model accordingly.

[0057] In some implementations, the independent validation dataset required for the multi-view performance evaluation process is constructed in a similar manner to the structured training sample set in step S301, but originates from completely different time periods or pavement regions, ensuring no overlap with the training data. The validation dataset includes latent deformation features obtained from multiple camera views, corresponding camera geometric coding features, and 3D deviation ground truth values. Further, the evaluation process first uses the fusion parameter model obtained in step S401 to predict the deviation for each sample in the validation set, obtaining predicted values. Subsequently, the process calculates two levels of performance metrics: global accuracy and inter-view consistency.

[0058] In this embodiment, the global accuracy index is used to measure the overall closeness between the model's predicted values ​​and the actual values. Preferably, the root mean square error (RMSE) is used as the quantification standard. The RMSE is calculated by taking the average of the sum of squares of the differences between the predicted and actual deviations of all samples in the validation set, and then taking the square root of this average. The inter-view consistency index is used to quantify the consistency of the model's estimation results for the same physical pavement area under different camera views. Specifically, for multiple samples from different cameras at the same world coordinate position in the validation set at the same time section, the statistical dispersion of the predicted values ​​corresponding to these samples is calculated. Preferably, the inter-view consistency error is used as the metric. This error is calculated by first calculating the average of all cross-view predicted values ​​for each pavement position observed by multiple cameras, then calculating the square of the deviation between the predicted value of each view and the average, then taking the square root of the average of these squared deviations, and finally averaging the results for all positions.

[0059] Preferably, the model screening and confirmation process makes decisions based on the root mean square error (RMSE) and inter-view consistency error (IPE) output by the multi-view performance evaluation process. The process presets two performance thresholds: a maximum permissible estimation error threshold and a maximum permissible inter-view difference threshold. The condition for the fused parameter model to pass verification is that its RMS error is not greater than the maximum permissible estimation error threshold, and its IPE is not greater than the maximum permissible inter-view difference threshold. It should be noted that this threshold determination logic is a setting method well-known to those skilled in the art based on engineering accuracy requirements, and will not be elaborated further here. When the model meets the conditions, the fused parameter model is confirmed as the corrected bias mapping model. If the model fails to meet both conditions simultaneously, it indicates that the model is overfitting the training viewpoint or that the fusion of geometric parameters is insufficient, requiring a return to step S401 to adjust the fusion strategy or supplement training data before re-verification.

[0060] The corrected bias mapping model maintains the same mathematical form as the fusion parameter model, but it has undergone rigorous validation to demonstrate its ability to perform stable and consistent 3D bias estimation on observation data under different camera geometries within a preset error tolerance. This step, by introducing an independent validation set and systematic evaluation, empirically elevates the model from a data-fit-based mapping relationship to a reliable metric tool with cross-view stability, providing a rigorously validated deterministic mapping relationship for subsequent progressive bias analysis on real-time temporal image streams.

[0061] S5, perform progressive deviation analysis on the corrected deviation mapping model and real-time time-series image stream data to obtain high-risk progressive damage area identification; the progressive deviation analysis process involves inputting the real-time image stream into the correction model, calculating the runway smoothness deviation and its rate of change over time, and identifying progressive abnormal areas based on thresholds. Specifically, this step includes the following sub-steps: S501, perform real-time feature extraction processing on the real-time time-series image stream data to obtain real-time feature point data; the real-time feature extraction processing is to apply a real-time feature detection algorithm to the current image frame to obtain the instantaneous image position of the runway surface texture feature points; The real-time feature extraction process applies a real-time feature detection algorithm to the current image frame to obtain the instantaneous image location of runway surface texture feature points. This includes real-time feature point detection and localization, and feature coordinate serialization and encapsulation. Specifically, the real-time feature point detection and localization process performs efficient feature detection operations on each real-time image frame to determine the precise sub-pixel coordinates of predefined texture features in the current image. The feature coordinate serialization and encapsulation process organizes the detected feature point coordinates according to a predefined identifier order and associates them with timestamps and camera identifiers to form a data structure that is strictly consistent with the format of the historical data processing stage.

[0062] In some implementations, the real-time detection and localization of feature points is crucial for achieving real-time perception. This processing is applied to each frame of image captured in real-time by the camera network. Where the subscript t represents the current timestamp. The processing employs the same type of feature detector as defined in step S1 ("initial projection data of feature points") and in step S201 (historical feature tracking) to ensure consistency in feature description. Preferably, a feature detection algorithm with high computational efficiency and meeting real-time requirements, such as FAST, ORB, or improved variants thereof, is used. The algorithm processes the input image... The system performs a scan and identifies a set of pixel locations that match preset corner or spot characteristics by evaluating the grayscale intensity variation patterns within the pixel neighborhood. Furthermore, to achieve higher positioning accuracy, the processing is performed at integer pixel coordinates. Subpixel-level fine-tuning is then performed on top of this. This operation is achieved by fitting a local gray-level distribution surface or through iterative optimization, resulting in the final coordinates of each feature point. This achieves sub-pixel precision, where This represents the two-dimensional coordinate vector in the image coordinate system of the i-th predefined runway texture feature point detected at timestamp t, where i is the index number of the feature point. This shows the sub-pixel coordinates of the i-th feature point at timestamp t, along the horizontal direction (usually defined as the row direction of the image) in the image coordinate system. This represents the sub-pixel coordinates of the i-th feature point at timestamp t, along the vertical direction (usually defined as the column direction of the image) in the image coordinate system. T represents the transpose operator of the vector indicating the coordinates. It is represented as a column vector. The calculation or processing method described is a conventional technique known to those skilled in the art for improving the accuracy of visual measurements, and will not be elaborated upon here.

[0063] In this embodiment, it should be noted that the "predefined runway texture feature points" sought by the real-time detection processing are physically identical to the historical feature points processed in steps S1 and S201. This means that, ideally, for the same camera in the same runway area, the set of feature points detected in real-time should correspond one-to-one with the set of feature points established in the historical data. This correspondence is ensured by using the same feature detector type and parameters, as well as a global feature point identifier mapping based on the runway texture map established during the initial system deployment. The processing transforms the real-time acquired image data stream into an instantaneous feature point coordinate sequence that is completely consistent with the data structure of the historical data stage, through a set of feature definition, detection, and localization rules that are strictly homogeneous with those of the historical training stage.

[0064] Preferably, the feature coordinate serialization and encapsulation process organizes the above positioning results into structured real-time feature point data. For the image acquired by the k-th camera at time t, the processing outputs a data unit. The data unit Represented as a tuple: Where t is the timestamp and k is the camera identifier. This is a unique identifier for the i-th feature point, and this identifier is consistent with the identifier system used in the "initial projection data of feature points" in step S1 and in all subsequent historical processing. This represents the two-dimensional sub-pixel coordinates of the feature point in the current frame image coordinate system, N is the number of feature points successfully detected under the viewpoint of this camera, and the subscript i is a traversal index used to refer to the i-th feature point successfully detected in the current frame image of the current camera k. The data unit These constitute the basic units of the real-time feature point data.

[0065] By repeatedly performing the above processing on the real-time image stream continuously generated by each camera in the camera network, a real-time feature point data stream that is continuously updated over time is obtained. The data stream maintains a high degree of homogeneity with the data used in the historical training phases in terms of feature definitions, coordinate accuracy, and data structure. This process ensures that the corrected bias mapping model, obtained empirically from step S402 and dependent on specific feature representations and coordinate systems, can be directly and reliably applied to current observations. This establishes an accurate and repeatable interface between the real-time data stream and the fully trained offline model, providing correctly formatted and physically meaningful real-time observation data for subsequent steps to perform instantaneous 3D bias estimation.

[0066] S502, deviation estimation and rate detection processing are performed on the real-time feature point data and the corrected deviation mapping model to obtain the high-risk progressive damage area identification; the deviation estimation and rate detection processing is to use the corrected deviation mapping model to estimate the three-dimensional smoothness deviation from the real-time feature point data, calculate the rate of change of the deviation value within a continuous time window, and compare it with a preset threshold, and identify the pavement area with the rate exceeding the threshold as a high-risk progressive damage area.

[0067] The deviation estimation and rate detection processing uses a corrected deviation mapping model to estimate the three-dimensional smoothness deviation from real-time feature point data, calculates the rate of change of the deviation value within a continuous time window, and compares it with a preset threshold. Areas of pavement with rates exceeding the threshold are identified as high-risk progressive damage zones. This includes model-based real-time deviation estimation and temporal rate of change analysis. Specifically, the model-based real-time deviation estimation process calls the corrected deviation mapping model obtained in step S402 to convert the real-time feature point data extracted in step S501 into estimated smoothness deviation values ​​in three-dimensional physical space. The temporal rate of change analysis process establishes a time series of deviation estimates for each local area of ​​the pavement and calculates the rate of change of this series within a sliding time window. By comparing this rate with a safety threshold, a quantitative assessment and spatial location of progressive damage risk are achieved.

[0068] In some implementations, the model-based real-time bias estimation processing is the core of achieving real-time conversion from image observation to physical quantities. This processing targets the real-time feature point data units output from step S501. The processing is based on the feature point identifier. Retrieve the corresponding initial image coordinates from the initial projection data of the feature points defined in step S1. Combined with current coordinates The cumulative two-dimensional displacement of the feature point since the initial moment is calculated. Furthermore, to adapt to the corrected deviation mapping model... The input requirement is that the processing needs to organize the feature point observations of the current frame into the input feature vector required by the model. The organization method described is the same as that used in step S301 when constructing training samples from implicit deformation surface parameters. Extracting feature vectors This is consistent with the logic. Specifically, for a three-dimensional pavement position to be estimated... The process collects displacement observations of neighboring feature points and encodes them into a feature vector characterizing the deformation pattern of the local region according to predetermined rules. Meanwhile, the geometric coding features of camera k from which this observation originated. The corrected bias mapping model was obtained. Called to calculate the estimated three-dimensional flatness deviation at that location at time t. : This allows the instantaneous feature vector representing the local texture deformation pattern of the pavement, extracted from the real-time image frame in step S501, to be used in the process. Encoding the inherent geometric properties of the corresponding camera All inputs are fed into the corrected deviation mapping model, which has undergone rigorous verification in step S402. In the middle; this model Essentially, it is a deterministic function trained on historical data and corrected by geometric prior knowledge. Its internal parameters encode a complex nonlinear mapping relationship between "multi-view deformation characteristics" and "observational geometry" and "three-dimensional physical deviation". Therefore, it performs forward calculation and inference on real-time input through this function and outputs a scalar value. This value is the world coordinate position of the pavement inferred by the model based on all input information at the current moment. By dividing the pavement into a regular grid or targeting key areas, the above calculations are repeated to obtain the deviation estimation field of the entire pavement or target area at time t. , where M represents the total number of estimated location points generated after spatial discretization of the runway surface, and m represents an index variable used to traverse and identify each independent spatial location of the runway surface.

[0069] In this embodiment, the time-series rate of change analysis is responsible for extracting dynamic risk information from the static deviation estimation field. This processing is performed on each point of interest on the pavement. Maintain a time-series buffer to store the sequence of bias estimates over a recent time period of length T. ,in This represents the two-dimensional horizontal coordinates of the m-th point of interest on the pavement in the unified world coordinate system. Indicates the location At any time The three-dimensional flatness deviation value is estimated using the corrected deviation mapping model. This represents the timestamp of the j-th sampling moment. This timestamp is related to the real-time feature point data unit in step S501. The included timestamps t are of the same origin, ensuring that the bias estimate is strictly synchronized with the observation time. j represents the position within the most recent time period T. The total number of accumulated effective deviation estimates, i.e., the length of the time series. T represents the preset sliding time window length. In this embodiment, For the current moment, and Preferably, the processing employs a linear regression method to fit the time series to quantify its changing trend. Specifically, for location... to estimate its deviation Treating it as a function of time t, a straight line is fitted using the least squares method. ,in These are coefficients obtained through least squares optimization; the subscript 'm' is used to emphasize that these parameters are specific to the m-th pavement position. The slope of the fitted line was calculated independently. This is defined as the rate of change of the flatness deviation at that location within the time window T. : ,in Indicates the position of the pavement The estimated rate of change of the flatness deviation at a given point is the slope of the fitted straight line; J represents the total number of sampling points (i.e., time frames) included in the time window used for linear regression analysis. This represents the time (timestamp) corresponding to the j-th sampling point within the time window, where j is an integer index from 1 to J. Indicates the location At, at a moment Through the corrected deviation mapping model The estimated three-dimensional flatness deviation value. This indicates that the summation operation is performed on index j from 1 to J; Represents the timestamps of all sampling points within the time window. The summation. In this embodiment, all summation operations are performed on J sampling points within the time window. The rate of change. The unit is (physical deviation unit / time unit), and its absolute value directly represents the severity and direction of the deformation of the pavement area.

[0070] Furthermore, the process will calculate the rate of change at each location. The comparison is made with a preset safety threshold. It should be noted that this threshold includes a positive threshold. and a negative threshold This is used to identify areas of rapid uplift and rapid subsidence, respectively. The identification logic is as follows: If... Then mark the location. This is a high-risk area for progressive destruction that is undergoing accelerated uplift; if If the threshold is specified, the location is identified as a high-risk zone for progressive damage, indicating accelerated subsidence. and The settings are based on engineering specifications, historical data statistics, or domain expert knowledge. This setting method is a familiar operation to those skilled in the art based on specific application scenarios, and will not be elaborated upon further here. All identified high-risk locations constitute the aforementioned set of high-risk progressive damage area markers. In summary, based on real-time acquired feature observations isomorphic to historical data, a deterministic mapping relationship, validated through geometric correction and performance verification, is applied to convert these observations into physically meaningful three-dimensional smoothness deviation values. Furthermore, differential analysis of the time series of these deviation values ​​captures their dynamic rate of change. This method shifts the focus of analysis from the absolute value of the deviation to its temporal gradient, thereby identifying pavement areas that are undergoing accelerated deformation even with small current absolute deviation values. This enables early, progressive warnings of potential structural damage, providing direct, quantitative spatial evidence for precise runway pavement maintenance decisions. Please refer to the appendix. Figure 4 As shown in the figure, the horizontal axis represents continuous time, and the vertical axis represents the three-dimensional smoothness deviation value estimated from real-time feature point data through the corrected deviation mapping model. The blue continuous curve represents the dynamic evolution trajectory of the smoothness deviation of a specific pavement area over time. The processing defines a continuous time analysis window on the curve and performs linear fitting based on the deviation estimation value sequence in the time buffer within this window. The slope of the resulting fitted line is quantified as the rate of change of smoothness deviation within that time period. The horizontal orange dashed line represents the positive and negative rate of change warning thresholds preset according to engineering specifications. The processing criterion is that when the absolute value of the slope of the fitted line exceeds the warning threshold, the corresponding time window and the associated pavement area are marked as a high-risk progressive damage zone, as shown in the red highlighted area in the figure. This technology shifts the focus of analysis from the absolute value of the deviation to its temporal gradient. Even if the current absolute deviation value is low, as long as its rate of change shows an accelerating trend and exceeds the safety threshold, it can be effectively identified and warned, thus demonstrating the ability of the technical solution to achieve early and progressive risk identification.

[0071] Based on the description of the above embodiments of the progressive analysis method for runway smoothness deviation based on time-series image feature recognition, this application also discloses a progressive analysis system for runway smoothness deviation based on time-series image feature recognition. This system can be a computer program (including program code) that runs the aforementioned progressive analysis method for runway smoothness deviation based on time-series image feature recognition. Please see the appendix. Figure 2As shown, the runway smoothness deviation progressive analysis system based on time-series image feature recognition can run the following units: The acquisition unit 110 is used to acquire data to be processed, which includes historical time-series image sequence data, initial projection data of feature points, known 3D deviation dataset, camera network geometric configuration data, and real-time time-series image stream data. The initial projection data of feature points is a set of predefined reference positions of runway texture feature points in the image coordinate system. The known 3D deviation dataset contains runway surface 3D flatness deviation values ​​measured synchronously with the historical time-series images. The camera network geometric configuration data describes the geometric relationship between the spatial position, viewing direction, and internal optical parameters of each camera. The inversion unit 120 is used to perform implicit deformation surface inversion processing on the historical time-series image sequence data and the initial projection data of feature points to obtain implicit deformation surface parameters. The implicit deformation surface inversion processing is based on the time-series changes of the positions of two-dimensional feature points under a fixed viewpoint, and under the preset road surface deformation manifold smoothness constraint, to solve the parameterized surface that characterizes the microscopic three-dimensional morphological changes of the road surface. The mapping unit 130 is used to perform multi-view latent space mapping processing on the latent deformable surface parameters and the known three-dimensional deviation dataset to obtain a three-dimensional deviation mapping model. The multi-view latent space mapping processing is to construct a generative model with latent deformable surface parameters as observation variables, and establish a cross-modal deterministic mapping relationship by inferring the latent space representation shared with the real three-dimensional deviation. The correction unit 140 is used to perform geometric correction processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain the corrected deviation mapping model; the geometric correction processing is to use the camera geometric parameters to adjust the viewpoint normalization of the mapping model to eliminate the influence of camera network configuration differences on deviation estimation. Analysis unit 150 is used to perform progressive deviation analysis on the corrected deviation mapping model and real-time temporal image stream data to obtain high-risk progressive damage area identification; the progressive deviation analysis process involves inputting the real-time image stream into the correction model, calculating the runway smoothness deviation and its rate of change over time, and identifying progressive anomaly areas based on thresholds. The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A progressive analysis method for runway smoothness deviation based on temporal image feature recognition, characterized in that, The method includes the following steps: S1, acquire the data to be processed, which includes historical time-series image sequence data, initial projection data of feature points, known 3D deviation dataset, camera network geometric configuration data, and real-time time-series image stream data; the initial projection data of feature points is a set of predefined reference positions of runway texture feature points in the image coordinate system; the known 3D deviation dataset contains runway surface 3D flatness deviation values ​​measured synchronously with the historical time-series images; the camera network geometric configuration data describes the geometric relationship between the spatial position, viewing direction, and internal optical parameters of each camera; S2, perform implicit deformation surface inversion processing on the historical time-series image sequence data and the initial projection data of feature points to obtain implicit deformation surface parameters; the implicit deformation surface inversion processing is based on the time-series changes of the positions of two-dimensional feature points under a fixed viewpoint, and under the preset road surface deformation manifold smoothness constraint, solve the parameterized surface that characterizes the microscopic three-dimensional morphological changes of the road surface. S3, perform multi-view latent space mapping processing on the latent deformable surface parameters and the known three-dimensional deviation dataset to obtain a three-dimensional deviation mapping model; the multi-view latent space mapping processing is to construct a generative model with latent deformable surface parameters as observation variables, and establish a cross-modal deterministic mapping relationship by inferring the latent space representation shared with the real three-dimensional deviation. S4, Perform geometric correction processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain the corrected deviation mapping model; the geometric correction processing is to use the camera geometric parameters to adjust the viewpoint of the mapping model to eliminate the influence of camera network configuration differences on deviation estimation. S5, perform progressive deviation analysis on the corrected deviation mapping model and real-time time-series image stream data to obtain high-risk progressive damage area identification; the progressive deviation analysis process involves inputting the real-time image stream into the correction model, calculating the runway smoothness deviation and its rate of change over time, and identifying progressive abnormal areas based on thresholds.

2. The method for progressive analysis of runway smoothness deviation based on temporal image feature recognition according to claim 1, characterized in that, Step S2 includes the following sub-steps: S201, the historical time-series image sequence data is processed to construct the optical flow field of feature points, thereby obtaining dense optical flow field data; the optical flow field construction process of feature points is based on the assumption of gray level conservation between adjacent image frames, and calculates the motion vector field formed by predefined pavement texture feature points in continuous time-series images. S202, the dense optical flow field data and the initial projection data of feature points are subjected to manifold embedding processing based on projection consistency to obtain implicit deformable surface parameters; the manifold embedding processing based on projection consistency regards the two-dimensional motion vector of each feature point as different observation projections of the same implicit three-dimensional deformable surface, and constructs and solves an optimization process with minimizing reprojection error and manifold curvature penalty term as the objective function, embedding and inverting the two-dimensional observations into a set of control point parameters of the three-dimensional deformable surface.

3. The method for progressive analysis of runway smoothness deviation based on temporal image feature recognition according to claim 1, characterized in that, Step S3 includes the following sub-steps: S301, perform spatiotemporal reference alignment processing on the implicit deformable surface parameters and the known three-dimensional deviation data to obtain a structured training sample set; the spatiotemporal reference alignment processing is to spatially stitch and temporally synchronize the implicit deformable surface parameters derived from different cameras at the same time section and the same road surface area, and associate them with the true value of the three-dimensional deviation of the section and the area to form a sample set with a unified spatiotemporal reference. S302, the structured training sample set is subjected to conditional variational encoder training to obtain a three-dimensional deviation mapping model; the conditional variational encoder training is based on the spliced ​​multi-view latent deformable surface parameters as conditions and the three-dimensional deviation true value as the reconstruction target, and the training model learns a conditional probability distribution. The decoder part of the model serves as a deterministic mapping function from latent parameters to three-dimensional deviation during inference.

4. A progressive analysis method for runway smoothness deviation based on temporal image feature recognition, as described in any one of claims 1-3, characterized in that... Step S4 includes the following sub-steps: S401, Perform parameter fusion processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain a fused parameter model; the parameter fusion processing involves embedding the camera geometric parameters as additional inputs into the mapping model, and adapting the model to a specific network geometry through retraining or parameter adjustment; S402, the fusion parameter model is validated to obtain the corrected bias mapping model; the validation process is to test the bias estimation accuracy of the fusion parameter model on an independent validation dataset to ensure its consistent performance under different perspectives.

5. The method for progressive analysis of runway smoothness deviation based on temporal image feature recognition according to claim 4, characterized in that, Step S5 includes the following sub-steps: S501, perform real-time feature extraction processing on the real-time time-series image stream data to obtain real-time feature point data; the real-time feature extraction processing is to apply a real-time feature detection algorithm to the current image frame to obtain the instantaneous image position of the runway surface texture feature points; S502, deviation estimation and rate detection processing are performed on the real-time feature point data and the corrected deviation mapping model to obtain the high-risk progressive damage area identification; the deviation estimation and rate detection processing is to use the corrected deviation mapping model to estimate the three-dimensional smoothness deviation from the real-time feature point data, calculate the rate of change of the deviation value within a continuous time window, and compare it with a preset threshold, and identify the pavement area with the rate exceeding the threshold as a high-risk progressive damage area.

6. The method for progressive analysis of runway smoothness deviation based on temporal image feature recognition according to claim 2, characterized in that, The optical flow field construction process for feature points is based on the assumption of gray-level conservation between adjacent image frames. It calculates the motion vector field formed by predefined pavement texture feature points in continuous temporal images, including optical flow constraint equation calculation and dense motion vector field optimization. The manifold embedding process based on projection consistency treats the two-dimensional motion vector of each feature point as different observation projections of the same implicit three-dimensional deformable surface. It constructs and solves an optimization process with the objective function of minimizing reprojection error and manifold curvature penalty term. It embeds and inverts the two-dimensional observations into a set of control point parameters for the three-dimensional deformable surface, including deformable surface parameterization modeling and parameter inversion optimization.

7. The method for progressive analysis of runway smoothness deviation based on temporal image feature recognition according to claim 3, characterized in that, The spatiotemporal reference alignment process involves spatially stitching and temporally synchronizing the implicit deformation surface parameters derived from different cameras at the same time section and the same track area, and associating them with the three-dimensional deviation truth value of that section and area to form a sample set with a unified spatiotemporal reference, including multi-source surface parameter spatial stitching and synchronization processing, as well as three-dimensional truth value association and sample construction processing. The conditional variational encoder training process uses the spliced ​​multi-view latent deformable surface parameters as conditions and the three-dimensional deviation true value as the reconstruction target to train the model to learn a conditional probability distribution. The decoder part of the model serves as a deterministic mapping function from latent parameters to three-dimensional deviation during inference, including variational inference and latent space learning processing and deterministic mapping function extraction processing.

8. The method for progressive analysis of runway smoothness deviation based on temporal image feature recognition according to claim 4, characterized in that, The parameter fusion process involves embedding camera geometric parameters as additional inputs into the mapping model, and adapting the model to a specific network geometry through retraining or parameter adjustment. This includes geometric parameter encoding and embedding processing, and model adaptation training processing. The validation process involves testing the bias estimation accuracy of the fused parameter model on an independent validation dataset to ensure its consistent performance under different perspectives. This includes multi-view performance evaluation processing and model screening and confirmation processing.

9. A progressive analysis method for runway smoothness deviation based on temporal image feature recognition according to claim 5, characterized in that, The real-time feature extraction process involves applying a real-time feature detection algorithm to the current image frame to obtain the instantaneous image position of the runway surface texture feature points, including real-time feature point detection and localization processing and feature coordinate serialization and encapsulation processing. The deviation estimation and rate detection processing uses a corrected deviation mapping model to estimate the three-dimensional smoothness deviation from real-time feature point data, calculates the rate of change of the deviation value within a continuous time window, and compares it with a preset threshold. The pavement area where the rate exceeds the threshold is identified as a high-risk progressive damage zone, including model-based real-time deviation estimation processing and time-series change rate analysis processing.

10. A progressive analysis system for runway smoothness deviation based on temporal image feature recognition, characterized in that, The system includes: The acquisition unit is used to acquire data to be processed, which includes historical time-series image sequence data, initial projection data of feature points, known 3D deviation dataset, camera network geometric configuration data, and real-time time-series image stream data. The initial projection data of feature points is a set of predefined reference positions of runway texture feature points in the image coordinate system. The known 3D deviation dataset contains runway surface 3D flatness deviation values ​​measured synchronously with the historical time-series images. The camera network geometric configuration data describes the geometric relationship between the spatial position, viewing direction, and internal optical parameters of each camera. The inversion unit is used to perform implicit deformation surface inversion processing on the historical time-series image sequence data and the initial projection data of feature points to obtain implicit deformation surface parameters. The implicit deformation surface inversion processing is based on the time-series changes of the positions of two-dimensional feature points under a fixed viewpoint, and under the preset road surface deformation manifold smoothness constraint, to solve the parameterized surface that characterizes the microscopic three-dimensional morphological changes of the road surface. The mapping unit is used to perform multi-view latent space mapping processing on the latent deformable surface parameters and the known three-dimensional deviation dataset to obtain a three-dimensional deviation mapping model. The multi-view latent space mapping processing is to construct a generative model with the latent deformable surface parameters as observation variables, and establish a cross-modal deterministic mapping relationship by inferring the latent space representation shared with the real three-dimensional deviation. The correction unit is used to perform geometric correction processing on the three-dimensional deviation mapping model and the camera network geometric configuration data to obtain the corrected deviation mapping model. The geometric correction processing is to use the camera geometric parameters to adjust the viewpoint of the mapping model to eliminate the influence of camera network configuration differences on deviation estimation. The analysis unit is used to perform progressive deviation analysis on the corrected deviation mapping model and real-time time-series image stream data to obtain high-risk progressive damage area identification; the progressive deviation analysis process involves inputting the real-time image stream into the correction model, calculating the runway smoothness deviation and its rate of change over time, and identifying progressive abnormal areas based on thresholds.

Images