Road asset patrol method based on machine vision and related equipment thereof

Abstract

The invention provides a road asset patrol method based on machine vision and related equipment thereof, which are used for improving the digital management level of road asset patrol. The method comprises the following steps: performing multi-source data dynamic synchronous acquisition on a road area to obtain an original road image matrix and corresponding shooting positioning data, and performing adaptive filtering and geometric correction on the original road image matrix to generate an enhanced image tensor, performing space-frequency spectrum joint feature extraction on the enhanced image tensor to obtain an asset feature vector set, performing multi-task classification on the asset feature vector set to obtain an asset category-position tensor, and performing tight coupling geographic positioning mapping on the asset category-position tensor according to the shooting positioning data to generate a geographic coordinate asset list. And performing spatio-temporal topology analysis on the geographic coordinate asset list to obtain a road asset map. And performing time sequence forward evolution prediction according to the road asset map to obtain an asset degradation prediction curve.

Description

Technical Field

[0001] This application relates to the field of visual analysis technology, and in particular to a machine vision-based method for inspecting highway assets and related equipment. Background Technology

[0002] Highway asset inspection is a core component of ensuring road safety and traffic efficiency, its importance lying in the real-time monitoring and maintenance of critical facilities such as signs, delineators, and traffic lights. These assets, acting as the "nerve endings" of highway infrastructure, directly impact vehicle safety and the orderliness of traffic flow. For example, blurred or missing signs may cause drivers to misjudge routes, worn lane markings reduce lane recognition efficiency, and malfunctioning traffic lights can lead to traffic accidents.

[0003] Current highway asset inspections heavily rely on personnel, primarily involving walking or driving along the road to visually inspect and record defects. However, this method is limited by physical labor and visual limitations, easily overlooking hidden defects. While specialized equipment (such as road radar) has been used to collect data in some inspections, improving accuracy in localized areas, its high cost and difficulty adapting to complex terrain have led to the emergence of drone inspections. Although drone inspections can achieve rapid, wide-area coverage, their image resolution is insufficient, requiring manual post-analysis and unable to provide real-time problem feedback.

[0004] Therefore, current inspection results are too subjective, and the data is severely fragmented, making it difficult to form a systematic data asset. Manual inspection results rely heavily on individual experience, and different personnel may have different judgments on the same disease, resulting in poor data consistency. While mechanical inspection and drone inspection can generate some structured data, they lack the ability to intelligently analyze the type, severity, and spatial distribution of diseases. Ultimately, the data is mostly stored in the form of images or videos, which is difficult to directly transform into quantifiable maintenance decision-making basis. Summary of the Invention

[0005] This application provides a machine vision-based highway asset inspection method and related equipment to improve the digital management level of highway asset inspection.

[0006] In a first aspect, this application provides a machine vision-based method for inspecting highway assets, the method comprising: Multi-source data is dynamically and synchronously collected in the highway area to obtain the original highway image matrix and the corresponding shooting and positioning data. The original highway image matrix is ​​subjected to adaptive filtering and geometric correction to generate an enhanced image tensor; Spatial-spectral joint feature extraction processing is performed on the enhanced image tensor to obtain an asset feature vector set; The asset feature vector set is subjected to multi-task classification processing through a preset first deep learning model to obtain the asset category-location tensor; Based on the captured location data, the asset category-location tensor is subjected to tight-coupled geolocation mapping processing to generate a list of geographic coordinate assets; Spatiotemporal topology analysis is performed on the geographic coordinate asset list to obtain a highway asset map; Using a pre-defined second deep learning model, a time-series forward evolution prediction process is performed based on the highway asset map to obtain an asset degradation prediction curve.

[0007] In one possible implementation, the adaptive filtering and geometric correction processing of the original highway image matrix to generate the enhanced image tensor includes: The original highway image matrix is ​​subjected to bilateral filtering to obtain a noise-optimized image matrix; The noise-optimized image matrix is ​​subjected to Retinex illumination compensation processing to obtain a contrast-enhanced image matrix; The contrast-enhanced image matrix is ​​subjected to Brown-Conrad distortion correction to obtain a geometrically corrected image matrix; The geometrically corrected image matrix is ​​scaled according to a preset size standard to generate an enhanced image tensor.

[0008] In one possible implementation, the spatial-spectral joint feature extraction process on the enhanced image tensor to obtain an asset feature vector set includes: The enhanced image tensor is subjected to convolutional neural network spatial feature extraction processing to obtain local texture feature vectors; The enhanced image tensor is subjected to Fast Fourier spectral feature extraction processing to obtain a global spectral feature vector; The local texture feature vector and the global spectral feature vector are concatenated and fused to obtain an asset feature vector set.

[0009] In one possible implementation, the step of performing multi-task classification processing on the asset feature vector set through a preset first deep learning model to obtain the asset category-location tensor includes: Construct an asset relationship topology graph based on the spatial distance of feature vectors in the asset feature vector set; The asset relationship topology graph is iteratively propagated using a pre-defined first deep learning model to obtain a node feature set. The first deep learning model is used to perform multi-task classification on the node feature set to obtain the asset class probability distribution and image location bounding box coordinates of each feature node. The asset class probability distribution is filtered according to a preset confidence threshold to obtain the asset class label for each feature node; The asset category label and the corresponding image location bounding box coordinates are encapsulated to obtain the asset category-location tensor.

[0010] In one possible implementation, the step of performing tightly coupled geolocation mapping processing on the asset category-location tensor based on the captured positioning data to generate a list of geographic coordinate assets includes: The image location bounding box coordinates are transformed using a first coordinate system based on preset camera calibration parameters to obtain the first physical coordinates of each asset. The collected inertial navigation data and the captured positioning data are subjected to Kalman filtering tight coupling processing to generate high-precision vehicle pose data. Based on the vehicle pose data, the first physical coordinates are transformed using the second coordinate system to obtain the geodetic coordinates of each asset. The geodetic coordinates are converted to latitude and longitude using WGS84 to obtain the latitude and longitude data of each asset. The latitude and longitude data and the corresponding asset category labels are then mapped and encapsulated to generate a list of geographic coordinate assets.

[0011] In one possible implementation, performing spatiotemporal topological analysis on the geographic coordinate asset list to obtain a highway asset map includes: A spatial association topology map is constructed based on a preset distance threshold and the geographic coordinate asset list; Historical detection datasets are obtained based on the geographic coordinate asset list, and the feature degradation rate of each asset is calculated based on the historical detection datasets and the geographic coordinate asset list to generate the current health assessment value for each asset. The current health assessment value and the spatial association topology map are structured and encapsulated to obtain a highway asset map.

[0012] In one possible implementation, the step of performing time-series forward evolution prediction processing based on the highway asset map using a preset second deep learning model to obtain an asset degradation prediction curve includes: Based on the time sequence, the health data of the target assets in the highway asset map is extracted and processed to obtain the historical health time series of each target asset. The historical health time series is modeled using a pre-defined second deep learning model to obtain an evolutionary model; The evolutionary model is subjected to regression decoding processing by the second deep learning model to obtain the health prediction value of the target asset; The historical health time series and the predicted health value are subjected to spline interpolation to obtain the asset degradation prediction curve.

[0013] Secondly, this application provides a machine vision-based highway asset inspection device, the device comprising: The data acquisition module is used to dynamically and synchronously acquire multi-source data in the highway area to obtain the original highway image matrix and the corresponding shooting and positioning data. The image enhancement module is used to perform adaptive filtering and geometric correction on the original highway image matrix to generate an enhanced image tensor. The feature extraction module is used to perform spatial-spectral joint feature extraction processing on the enhanced image tensor to obtain an asset feature vector set; The identification and classification module is used to perform multi-task classification processing on the asset feature vector set through a preset first deep learning model to obtain the asset category-location tensor. The asset location module is used to perform tight-coupled geolocation mapping processing on the asset category-location tensor based on the captured location data to generate a list of geographic coordinate assets. The spatiotemporal topology module is used to perform spatiotemporal topology analysis on the geographic coordinate asset list to obtain a highway asset map. The degradation prediction module is used to perform time-series forward evolution prediction processing based on the highway asset map using a preset second deep learning model to obtain an asset degradation prediction curve.

[0014] Thirdly, this application provides a computing device, the computing device comprising: At least one processor; and, A memory and a communication interface that are communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which implements the machine vision-based highway asset inspection method described above by executing the instructions stored in the memory.

[0015] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the machine vision-based highway asset inspection method described above.

[0016] In summary, this application includes at least the following beneficial technical effects: 1. Through spatiotemporal topology analysis, construct the spatial distribution network and temporal evolution patterns of assets to form a dynamically updated "digital twin" model, supporting global asset management. 2. Based on historical data and deep learning models, predict the degradation curve of highway assets and provide early warning of potential risks to promote maintenance from passive response to proactive prevention, thereby reducing maintenance costs.

[0017] 3. Through the deep integration of machine vision, deep learning and geographic information systems, high precision, full automation, predictability and intelligence of highway asset inspection have been achieved, solving the problems of low efficiency, incomplete coverage and delayed response of traditional manual inspection, and promoting the transformation of highway maintenance from "experience-driven" to "data-driven". Attached Figure Description

[0018] Figure 1 This is a diagram illustrating the operating environment of a machine vision-based highway asset inspection method provided in this application embodiment. Figure 2 This is a flowchart illustrating a machine vision-based highway asset inspection method provided in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of a machine vision-based highway asset inspection device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the computing device provided in the embodiments of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. With the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0020] The terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to those processes, methods, products, or apparatuses.

[0021] First, a definition of the terminology used in this application is provided: (1) Convolutional Neural Network (CNN) model CNN network models are used to simulate biological visual systems. They are specifically designed to process grid-like data such as images through structures such as convolutional layers and pooling layers. Convolutional layers can automatically extract local features such as edges and textures in images, while pooling layers simplify features and retain key information. This capability enables them to efficiently complete tasks such as image recognition and face recognition, for example, enabling mobile phone cameras to quickly recognize faces in photos or accurately read QR code information when scanning codes.

[0022] (2) Long Short-Term Memory Network (LSTM network) As an improved form of recurrent neural network, LSTM network solves the problem of traditional recurrent neural networks' inability to handle long sequence data through unique gating mechanisms (input gate, forget gate, output gate). It can remember important information in long sequence data. For example, when processing a text, it can remember the relationship between the people mentioned in the previous text and the actions in the following text. This makes it play an important role in fields such as language translation, speech recognition, and time series prediction. For example, it can help machine translation to understand the meaning of the context more accurately or predict the future trend of stock prices.

[0023] (3) Attention Mechanism The concept of attention mechanisms draws on the human tendency to selectively focus on key information when processing data. This allows models to automatically assign weights when processing data, focusing on information that is more important to the current task. For example, when translating "He likes to eat apples," the model will focus more on "apple" in the original text rather than "he" or "likes" when generating the word "apple." This mechanism significantly improves the performance of tasks such as machine translation, image description, and speech recognition, making the model's information processing more similar to human cognition.

[0024] (4) Retinex Retinex is an image enhancement theory based on the human visual system. It decomposes an image into a reflection component that reflects the characteristics of the object itself and a brightness component affected by lighting. By removing the effects of uneven lighting, it restores the true appearance of the object. This theory can be applied to process images taken under complex lighting conditions such as backlighting and shadows, making the details in the image clearer and the colors more natural. For example, it can make a face taken against the light no longer completely black, but clearly show the facial features.

[0025] (5) Brown-Conrady Distortion Correction The Brown-Conrad distortion model, proposed by Brown and Conrad, takes into account both radial and tangential distortion of the lens. These distortions can cause problems such as curved lines and distorted object shapes in the captured image. After correction by the Brown-Conrad distortion model, straight lines in the image can be restored to straightness and object shapes can be closer to reality. This is crucial in fields such as drone mapping and industrial inspection, for example, ensuring that road boundaries in aerial maps are straight lines, or accurately measuring the dimensions of industrial parts during inspection.

[0026] (6) World Geodetic System 1984 (WGS84) WGS84, proposed and promoted by the U.S. Department of Defense, is the coordinate system currently used by the Global Positioning System. It defines parameters such as the shape of the Earth (a rotating ellipsoid) and the location of the origin (the Earth's center of mass). It accurately describes the location of any point on Earth using latitude, longitude, and altitude. With the help of this system, people can obtain accurate location information when using navigation software, and vehicles, airplanes, and other vehicles can also achieve precise navigation.

[0027] (7) Earth Coordinate System The Earth coordinate system is a type of spatial coordinate system that maintains a fixed relationship with the Earth. Its establishment stemmed from the human need for precise positioning and description of objects on the Earth's surface and in near-Earth space, and it has been continuously improved with the development of surveying, navigation, and aerospace technologies. The core function of the Earth coordinate system is to provide a unified benchmark for determining the spatial position of any point on Earth (including, but not limited to, objects on the Earth's surface and within the atmosphere). By setting parameters such as the origin, coordinate axis directions, and scale, it makes the positional information of different objects comparable and operable. With the help of the Earth coordinate system, people can achieve a variety of practical effects, such as accurately marking the locations of cities and mountains on maps, enabling satellites and aircraft to determine their flight paths near Earth, allowing navigation equipment to guide vehicles and ships to the correct routes, and providing precise location references for geological exploration and engineering construction, ensuring the accuracy and consistency of spatial positioning for various activities.

[0028] (8) Residual Network (ResNet) The ResNet network addresses the vanishing gradient or performance degradation issues that occur during the training of deep neural networks by introducing residual connections. This design allows the network to be easily trained to dozens or even hundreds of layers, enabling it to extract more complex features from data such as images. This results in higher recognition accuracy in tasks such as image classification and object detection, allowing computers to more accurately distinguish between different types of animals or identify objects in complex scenes.

[0029] For ease of understanding, an exemplary runtime environment is provided below: like Figure 1As shown in the diagram, the operating environment provided in this embodiment is an inspection vehicle. A rigid bracket mounted on its top holds an integrated AI inspection device, which includes a core sensing unit and a communication module. The core component of the sensing unit is a stereo vision camera, including but not limited to a monocular stereo vision camera or a binocular stereo vision camera. The stereo vision camera uses a high-resolution camera equipped with autofocus and image stabilization to ensure clear image acquisition of the highway area under various lighting and weather conditions. In this embodiment, the AI ​​inspection device uses a monocular stereo vision camera to acquire images of the highway area during the vehicle's journey. For example, when the vehicle is traveling at 120 km / h, the camera synchronously captures images of the highway area at a rate of 30 frames per second. By calculating the horizontal displacement (parallax) of the target object (i.e., the highway asset in this application) in the image, and combining this with known lens spacing and focal length parameters, the spatial distance between the target and the camera can be accurately calculated. Simultaneously, the device also embeds a satellite positioning sensor using a multi-band GNSS receiver chip, continuously outputting the vehicle's latitude and longitude coordinates at a frequency of 10 Hz. It is worth noting that the positioning data acquired at this time only represents the vehicle's own position. To achieve precise positioning of highway assets, the system establishes a three-level coordinate mapping chain: first, the pixel coordinates of the highway assets in the image are converted into three-dimensional vectors in the camera coordinate system; then, through a pre-calibrated installation matrix (including parameters such as camera pitch angle and altitude), they are rotated and translated to the vehicle coordinate system; finally, real-time GNSS position data is superimposed to complete the geodetic coordinate transformation in the Earth coordinate system. It should be understood that, in another optional implementation, the AI ​​inspection equipment can also be installed on mobile inspection equipment such as drones.

[0030] Next, the AI ​​inspection equipment establishes a low-latency connection with the cloud server via a mobile network communication module. The transmission protocol employs a layered encapsulation strategy: the original image is appended with timestamps, location information, and other data, and then uploaded to the cloud server after encoding and compression. The distributed AI analysis engine deployed in the cloud includes a recognition module and a decision-making module working in parallel: the backbone network of the recognition module integrates a coordinate attention mechanism, significantly improving the detection rate of highway assets such as warning signs, directional signs, and anti-glare panels; the decision-making module integrates spatial analysis algorithms and a historical inspection database, automatically triggering asset health assessment and degradation prediction when highway assets are identified in a certain road section.

[0031] It should be noted that the above-mentioned equipment is exemplary, and the number and type of equipment can be adjusted in different scenarios or according to different needs.

[0032] The technical solution of this application is described below using an inspection vehicle as the main entity for inspection. It should be noted that the various steps of the technical solution in the embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0033] like Figure 2The diagram shown is a flowchart illustrating a machine vision-based highway asset inspection method provided in this embodiment of the application. The machine vision-based highway asset inspection method provided in this embodiment includes the following steps.

[0034] Step S1: Perform dynamic synchronous acquisition of multi-source data in the highway area to obtain the original highway image matrix and the corresponding shooting and positioning data.

[0035] It should be understood that the highway area specifically refers to the 120° cone-shaped three-dimensional space directly in front of the inspection vehicle, with its longitudinal coverage extending from 0 meters to 50 meters in front of the vehicle, and its lateral coverage covering both lanes and shoulders. This area includes the following three types of key targets: The fixed, integrated AI inspection equipment utilizes a monocular stereo vision camera (a camera that achieves stereo perception with only a single lens) as its core sensing unit. The camera's physical installation satisfies the geometric constraint of a 75° pitch angle, ensuring the lens's optical axis forms a specific angle with the horizontal plane, thus covering the three-dimensional space directly in front of the vehicle (i.e., the highway area). For example, when a vehicle is traveling on a highway, the 75° pitch angle can simultaneously capture nearby lane lines and distant cantilevered signs, avoiding asset omissions caused by the limited field of view of traditional head-up cameras.

[0036] The imaging process of a stereo vision camera is based on the principle of parallax ranging. Taking a driving scene at 120 km / h as an example, the camera captures an image sequence at a rate of 30 frames per second. When the same target (such as a delineator) produces a pixel displacement Δx between consecutive frames, the pre-calibrated camera intrinsic parameter matrix K (including focal length f, principal point coordinates C) is used to measure the distance between pixels. x C y The target depth can be calculated as: Z = fB / Δx. Where B is the equivalent baseline distance (virtual binocular distance), determined by the product of vehicle displacement and frame interval time. This process synchronously records the precise timestamp t of each frame. n (Nanosecond-level accuracy) and establish a mapping relationship with satellite positioning data. The satellite positioning sensor uses a multi-band GNSS receiver chip, and its output contains two key data: raw observation index (i.e., pseudorange ρ, carrier phase φ) and decoded location (e.g., latitude and longitude (λ, Φ) in the WGS84 coordinate system).

[0037] Meanwhile, to calibrate the spatiotemporal misalignment between the acquired "highway area" images and the positioning data, the system in this application embodiment deploys the following three-level synchronization mechanism: Time synchronization: The camera shutter trigger signal and the 1PPS (pulse per second) signal of the GNSS chip are aligned through FPGA hardware, and the time deviation is controlled within ±100μS; Spatial calibration: Establish the image coordinates (u,v) to the camera coordinate system (X) using the checkerboard calibration method.c ,Y c Z c The conversion relationship between () and (). Dynamic compensation: The built-in inertial measurement unit monitors the changes in vehicle pitch / yaw angle in real time. When the vehicle body bumps are detected, causing a pitch angle change Δα > 0.5°, the image reprojection algorithm is triggered to correct the coordinate mapping.

[0038] Ultimately, the generated original highway image matrix adopts a four-dimensional tensor structure: [timestamp, height H, width W, RGB channels]. For example, at a resolution of 1280×720, a single frame is stored as a uint8 type tensor [t n [720,1280,3]. Meanwhile, the corresponding shooting and positioning data is encapsulated as a structured array.

[0039] Subsequently, when transmitting data via mobile network communication, a layered encapsulation strategy is adopted: every 56 frames of image data (approximately 1.87 seconds of data) is packaged into a transmission unit, with an additional check frame and a location data packet header. The check frame uses Reed-Solomon encoding, which can repair data loss of ≤11 frames, ensuring an effective transmission rate of 45 frames / second even in weak signal areas such as tunnels.

[0040] Step S2: Perform adaptive filtering and geometric correction on the original highway image matrix to generate an enhanced image tensor.

[0041] First, an adaptive bilateral filtering algorithm is used to process the original highway image matrix, thereby solving image quality problems caused by vehicle vibration, sudden changes in illumination, and lens distortion under dynamic inspection environments. It should be understood that the original highway image matrix is ​​a data stream captured by a monocular stereo vision camera, essentially a digital array containing three-dimensional spatial information (height H × width W × RGB channels). When the inspection vehicle travels at 120 km / h, road bumps cause motion blur, and sudden changes in illumination at tunnel entrances and exits cause local overexposure or underexposure. In these situations, adaptive bilateral filtering is needed for noise optimization. The bilateral filtering algorithm differs from ordinary blurring algorithms in that it simultaneously considers the spatial proximity and color similarity of pixels: for each pixel in the image, the spatial weight (determined by a Gaussian function, with higher weights closer to the center) and color weight (determined by color difference, with higher weights for smaller color differences) of all pixels within its 50×50 neighborhood are calculated. Finally, the new pixel value at the center point is the weighted average of the neighboring pixels. For example, in a rainstorm scene, raindrops form dense white noise on the image. Traditional mean filtering will blur the edges of the outline, while bilateral filtering can preserve the sharp edges of the outline's metal frame while eliminating rain noise, which is equivalent to "intelligently smoothing" the image while preserving important details.

[0042] The noise-optimized image matrix (i.e., the noise-optimized image matrix) then enters the Retinex illumination compensation stage. The core of Retinex theory is to decompose an image into a reflection component (an inherent property of the object) and an illumination component (environmental interference). In practice, multi-scale Gaussian filtering is used: the noise-optimized image matrix undergoes three blurring processes with different radii (σ=15 / 80 / 200 pixels) to generate three base maps representing illumination distributions at different scales; the original image is subtracted from each base map in the logarithmic domain to obtain three sets of reflection component estimates; finally, these are weighted and fused into the final output. For example, when a vehicle exits a tunnel, strong backlighting causes the traffic light image area to appear as a silhouette. Retinex processing can separate the reflective characteristics of the traffic light housing, restoring the true color difference between red and green lights, similar to how professional photographers use HDR technology to restore backlit photos.

[0043] After obtaining an image with balanced illumination (i.e., a contrast-enhanced image matrix), it is necessary to correct the geometric distortion caused by the lens. The Brown-Conrad distortion model corrects both radial distortion (caused by lens curvature, manifested as curved straight lines at image edges) and tangential distortion (caused by lens mounting misalignment, manifested as trapezoidal shapes of rectangular objects). The correction process involves inverse kinematics and coordinate remapping: first, the camera calibration file is loaded to obtain the radial distortion coefficients k1=-0.21 / k2=0.03 and the tangential distortion coefficients p1=0.001 / p2=-0.002; then, the ideal coordinates (x,y) are established to the distorted coordinates (x,y). s y s Mapping relationship: Where, r 2 =x 2 +y 2 By inverse mapping, the pixel values ​​at the distorted coordinates are filled to the ideal coordinates. In the curve inspection scenario, the lane lines in the uncorrected image appear as convex arcs. This processing restores them to straight lines, ensuring that the accuracy error of subsequent AI measurement of the anti-glare panel spacing is less than 3 centimeters.

[0044] Finally, the stage performs size standardization. The size of the geometrically corrected image matrix (i.e., the geometrically corrected image matrix) may vary due to dynamic cropping (e.g., 1280×720 or 1920×1080), and needs to be uniformly scaled to a preset size of 1024×2048 pixels. Scaling employs a bicubic interpolation algorithm: for the target pixel (u,v), a 4×4 neighborhood is taken in its original image, and the weights of each neighboring pixel are calculated using the cubic polynomial S(x) = 1.5|x|³ - 2.5|x|² + 1 (when |x| < 1). A weighted average is then used to generate the new pixel value. This process not only standardizes the data size for subsequent convolutional neural network processing but also filters high-frequency noise through downsampling. For example, when identifying small delineators 5 meters away, the scaling process preserves the texture features of the reflector while eliminating granular noise from the asphalt pavement, improving feature extraction efficiency by 40%.

[0045] Step S3: Perform spatial-spectral joint feature extraction processing on the enhanced image tensor to obtain an asset feature vector set.

[0046] First, the enhanced image tensor is input into the first four residual blocks of the pre-trained ResNet backbone network. Through continuous convolutional operations and residual connections, multi-level extraction of texture, edge, and morphological information under different receptive fields is achieved. Specifically, the convolutional layers use trainable convolutional kernels to perform sliding window scanning on the tensor, automatically capturing the diamond-shaped edges of reflective contour markers, straight segments of lane markings, and columnar structures of overhead facility supports. The pooling layers then converge and downsample the local responses, removing redundancy and retaining key features to form a high-level spatial feature vector (i.e., a local texture feature vector) with 1024 dimensions at the output. This local texture feature vector accurately represents the geometric shape and texture distribution of each asset within the highway area.

[0047] Simultaneously, a Fast Fourier Transform (FFT) spectral feature extraction process is performed on the enhanced image tensor. By converting the image signal in the spatial domain to the frequency domain, the overall periodicity and texture structure distribution characteristics of the image are revealed. The core of this process lies in performing a two-dimensional discrete Fourier transform on the two-dimensional pixel matrix of each channel. The low-frequency components of the transformed result correspond to the overall contour and large-scale structural information of the image, such as wide road surface reflections and traffic light outlines, while the high-frequency components are associated with fine textures, such as raindrop spots and marking edge details. To avoid high-frequency noise interference, the top 10% of the low-frequency bands in the spectrum are extracted and their amplitude normalized, resulting in a set of 256-dimensional spectral descriptors (i.e., global spectral feature vectors). These global spectral feature vectors effectively reflect the shape characteristics and material consistency of the asset target globally, supplementing the multi-scale global information that is difficult to capture through spatial feature extraction.

[0048] Finally, the obtained local texture feature vectors and global spectral feature vectors are concatenated and fused. Specifically, the 1024-dimensional spatial vector output by the ResNet network and the 256-dimensional spectral vector obtained by FFT transformation are concatenated at the feature dimension level to form a joint feature vector of length 1280 (i.e., the asset feature vector set). The asset feature vector set retains the accurate local texture edge localization capability while also possessing the robustness brought by global spectral analysis, enabling high-precision target differentiation and classification in complex highway areas where small targets (such as reflective delineators) and large targets (such as cantilevered traffic lights) coexist.

[0049] Step S4: Perform multi-task classification processing on the asset feature vector set using a preset first deep learning model to obtain the asset category-location tensor.

[0050] It should be understood that the asset feature vector set contains a 1280-dimensional feature vector for each candidate region. For example, the feature vector of a delineator would highlight the high-frequency spectral characteristics of the reflector and the sharp edge texture of the metal frame.

[0051] To simulate the human cognitive process of recognizing objects by referring to their surroundings, the system in this embodiment first constructs an asset relationship topology graph—treating each feature vector as a graph node. If the spatial distance between two nodes in the original image is less than 200 pixels (approximately corresponding to a 5-meter range in the real world), a connection edge is established and assigned a weight. The weight reflects feature similarity; for example, adjacent delineators have small feature differences due to identical materials and installation angles, while delineators and distant traffic lights have significant feature differences. By assigning different weights, misidentification problems in dense scenes can be avoided. For instance, when heavy rain causes a water film to reflect off the surface of an anti-glare panel, traditional models easily misidentify it as a traffic light, but the topology can correct this misidentification based on the spatial constraint that "the object is located on a guardrail and adjacent to a delineator."

[0052] Subsequently, the asset relationship topology graph is input into a pre-trained CNN network model. The CNN network model iteratively propagates the features in the asset relationship topology graph to update the information of each node. The propagation process performed by the CNN network model consists of two layers of graph convolution operations as shown below: Neighborhood aggregation: A node receives a weighted sum of features from its neighbors. For example, a traffic light node integrates the features of the sign below (weight 0.7) and the features of the tree to the right (weight 0.1). Nonlinear transformation: The aggregation result is processed by the ReLU activation function, where the nonlinear transformation can be expressed by the following formula: in, Given an adjacency matrix with self-loops, This is a trainable weight matrix. After iteration, the node feature set integrates local and contextual information. For example, isolated milestones that are usually easily missed are detected, but because they are located above the lane line extension, the detection rate is improved by 35% after aggregating lane line features.

[0053] After obtaining the updated node feature set, the updated node feature set is input into the multi-task classification head for parallel decision-making. The classification head employs a dual-branch structure as shown below: Category branching: The node features are mapped to a 12-dimensional vector (corresponding to 12 asset classes) through a fully connected layer, and the asset class probability distribution is output through the Softmax function; Localization branch: Outputs bounding box parameters through coordinate regression layer .

[0054] For example, the feature of a node is output through the category branch. This indicates a 91% probability that the sign is a delineator; the localization branch simultaneously outputs its pixel coordinate range in the image. This multi-task design avoids the efficiency loss caused by the separation of classification and localization in traditional schemes, enabling sign recognition and localization to be completed simultaneously.

[0055] To filter out low-confidence results, a threshold is applied to truncate the probability distribution. In this embodiment, a confidence threshold is set. Only retain the highest probability value The nodes that are selected are considered background noise and are discarded. For example, the output of a certain node... (Highest probability 45% < 85%), identified as false targets caused by leaf shaking and discarded. The asset category labels and corresponding bounding box coordinates after this filtering are encapsulated as... The asset class-location tensor is formed by using N as the number of valid targets, with each row storing the class ID, confidence level, and bounding box coordinates.

[0056] Step S5: Perform tight-coupled geolocation mapping processing on the asset category-location tensor based on the captured positioning data to generate a list of geographic coordinate assets.

[0057] It should be understood that the asset class-location tensor contains asset class labels (such as "contour markers") and their bounding box coordinates in the image. For example, the bounding box coordinates of a delineator are [320, 580, 350, 620], indicating that the target is located within the range of columns 320 to 350 and rows 580 to 620 of the image. At this point, the first-level coordinate transformation—from image coordinates to the first physical coordinates in the vehicle coordinate system (i.e., the first coordinate system)—needs to be completed using camera calibration parameters. The camera calibration parameters are obtained in advance through checkerboard calibration and include the focal length. Main point and distortion coefficient. The conversion formula for this process is shown below: Where K is the intrinsic parameter matrix, R and T are the rotation and translation matrices from the camera to the vehicle coordinate system, and d is the depth value calculated through parallax. Taking a delineator as an example, its image center point (u,v)=(335,600) is transformed and output as vehicle coordinate system coordinates. The unit is meters, indicating that the delineator is located 2.3 meters to the right front of the vehicle, 0.8 meters below the ground (due to camera tilt angle), and 15.7 meters in front. The first-level coordinate transformation establishes the connection between image pixels and real physical space, much like converting markers on a photograph into measurement data on construction drawings.

[0058] The vehicle's physical coordinates need to be further transformed to the Earth coordinate system, but satellite positioning data has dynamic errors. When a vehicle travels at 120 km / h, the positioning error of ordinary GNSS can reach 3 meters, causing the delineator coordinates to drift to adjacent lanes. Therefore, a tightly coupled Kalman filter is introduced: fusing raw multi-band GNSS observations (pseudorange ρ, carrier phase φ) with inertial navigation data (angular velocity Ω, acceleration a). The Kalman filter includes two stages as shown below: prediction and update. Prediction: Based on the state at the previous time step The current state is calculated from (position + velocity) and IMU data: .in Here is the state transition matrix. For IMU input; Update: Using GNSS observations The predicted value is corrected using the formula shown below: in, For Kalman gain, To observe the noise matrix. For example, when a vehicle is turning, simple GNSS positioning will cause the coordinates of the delineator to shift off the road. However, by fusing IMU angular velocity data, the system can accurately track the vehicle's turning trajectory and compress the positioning error to within 0.05 meters, which is equivalent to accurately distinguishing delineators that are 30 centimeters apart on the same guardrail.

[0059] Subsequently, the second-level coordinate transformation is driven based on the high-precision vehicle pose data. This transforms the first physical coordinates in the first coordinate system. Transform the pose matrix into geodetic coordinates in the Earth coordinate system (i.e., the second coordinate system). The pose matrix consists of the vehicle's real-time latitude and longitude coordinates. And attitude angles (pitch / roll / yaw) are constructed. For example, when the vehicle is at latitude 31.2°, longitude 121.3°, and the front of the vehicle is facing north, the first physical coordinates (2.3, -0.8, 15.7) of the outline are converted into geodetic coordinates (-15.2, 28.7, -0.6), which means: rectangular coordinates with the Earth's center as the origin.

[0060] Furthermore, the geodetic coordinates need to be converted to the universal WGS84 latitude and longitude to be compatible with the Global Positioning System. This is calculated using the WGS84 ellipsoid parameters (semi-major axis a = 6378137 m, eccentricity f = 1 / 298.257): in , , The geodetic coordinates of the delineator (-15.2, 28.7, -0.6) are converted to output latitude and longitude. .

[0061] Finally, asset category labels and latitude / longitude coordinates are packaged into a geographic coordinate asset list. This geographic coordinate asset list can anchor virtual targets identified by the AI ​​model to the physical Earth. For example, when the system reports "the delineator at K202+300 on G15 Expressway is damaged," maintenance personnel can accurately locate the specific guardrail post number using latitude and longitude coordinates, avoiding misoperations caused by the vague description of "500 meters east" in traditional solutions.

[0062] Step S6: Perform spatiotemporal topology analysis on the geographic coordinate asset list to obtain a highway asset map.

[0063] It should be understood that the geographic coordinate asset list includes asset category labels and WGS84 latitude and longitude coordinates. For example, the record for a delineator is {"type": "delineator", "latitude": 31.2897, "longitude": 121.4732}. The geographic coordinate asset list only lists isolated assets and cannot reflect the spatial dependencies between facilities—for example, when a sign is tilted, its support pole may corrode, and adjacent delineators may also be damaged. Therefore, a spatial relationship topology is first constructed: with each asset as a node, if the surface distance between two assets is less than a preset threshold of 10 meters (calculated using the Haversine formula), an undirected connection edge is established. The distance threshold is set based on the layout patterns of highway assets; for example, the spacing within a group of anti-glare panels is typically 8-12 meters, and the independent interval between milestones is 1000 meters. A 10-meter threshold can accurately connect anti-glare panels within the same group without mistakenly connecting milestones. The topology uses an adjacency matrix storage structure. If asset A (sign) and asset B (support pole) are 3.2 meters apart, then the matrix elements... If asset C (milestone) is 105 meters from A, then... 0.

[0064] After constructing the topology map, the asset health assessment value needs to be calculated. Health represents the degree of asset deterioration, and its calculation relies on a historical inspection dataset—that is, an archive of feature vectors from the past 12 inspections of the same asset. The feature vector of the currently inspected asset... Retrieve its historical vector The feature degradation rate is quantified using the cosine similarity metric shown below: Current health assessment value obtained For example, the historical average feature vector of a certain delineator's reflective film is [0.92, 0.85, 0.78], while the current detection value is [0.82, 0.75, 0.70]. The cosine similarity is 0.94, the degradation rate is 0.06, and the current health assessment value H=0.94.

[0065] The final stage involves structured encapsulation. The spatial topology map and health assessment values ​​are merged into a JSON-formatted map. In the resulting highway asset map, the topology layer stores the connections between assets through a graph structure; the attribute layer attaches health and geographic coordinates to each node. For example, in a tunnel lighting system assessment, the map can reveal the correlation between "lighting lamp L22's health dropping to 0.68" and "adjacent lamp L23's health at 0.65," indicating that the entire group of lights needs replacement rather than single-point repair. This structured highway asset map breaks through the limitations of traditional tabular lists, essentially upgrading a two-dimensional paper map to a three-dimensional digital sandbox, making asset status and spatial relationships immediately clear.

[0066] Step S7: Using a preset second deep learning model, perform time-series forward evolution prediction processing based on the highway asset map to obtain the asset degradation prediction curve.

[0067] It should be understood that the second deep learning model is an LSTM network model with an attention mechanism. The highway asset map stores the spatial relationships and historical health records of assets in JSON format. For example, the data for contour marker C038 includes "health_history": [0.95, 0.93, 0.90, 0.88, 0.85] and a list of adjacent assets ["signboard_S12", "glare shield_A25"]. After obtaining the highway asset map, the first step is to extract the health time series of the target assets. That is, the monthly health values ​​for the past 12 months are extracted from the map in chronological order. If data for a certain month is missing, it is filled in using linear interpolation. For example, if the data for July 2024 is missing, the July value is calculated using the June value of 0.88 and the August value of 0.85: July value = 0.88 + (0.85 - 0.88) x (number of days in July / 30) = 0.86. Finally, a standardized historical health time series is generated. .

[0068] Subsequently, the historical health time series H is input into a pre-defined second deep learning model for evolutionary modeling. In the second deep learning model, the LSTM layer controls the information flow through forget gates, input gates, and output gates. Specifically, the forget gate determines which old memories to discard (e.g., ignoring accidental outliers from two years ago); the input gate filters new information (e.g., focusing on the impact of recent six months of heavy rainfall). Memory cells are updated using the formula shown: ,in For old memories, For new information. The final output gate generates the current state. In the second deep learning model, the attention layer automatically assigns weights to focus on key time periods. Specifically, the importance score of monthly health data is calculated using the formula shown: .in, The function is used to measure the difference between the state at time t and the global state h. s The correlation. For example, when predicting the degradation of anti-glare panels, the model assigns higher weights to summer months with frequent heavy rain (…). ), while reducing attention to dry winters ( ).

[0069] After modeling, regression decoding is performed on the evolutionary model. The context vector output by the attention layer is then used. Input to the fully connected layer of the second deep learning model: Output the future Monthly health prediction (e.g.) =3). Taking a profilometer as an example, the input sequence is... Model output , , This indicates that health will continue to decline over the next three months.

[0070] Finally, a smooth asset degradation prediction curve is generated through cubic spline interpolation. Historical data points are then connected. With prediction point Etc. Construct a continuous function using the piecewise cubic polynomial shown: S(t) = a i (tt i ) 3 +b i (tt i ) 2 +c i (tt i )+d i (t i ≤t<t i+1 ), where a i b i c i di The curve satisfies the condition that its first and second derivatives are continuous at the connection points. For example, the discrete values ​​[(12,0.85), (13,0.82), (14,0.80), (15,0.78)] are transformed into a continuous curve, and the interpolation value at t=13.5 is... The asset degradation prediction curve also marks warning thresholds (health level < 0.6 is marked in red), visually showing when the asset reaches a critical state.

[0071] For example, historical data from a group of anti-glare panels along a coastal highway shows that salt spray corrosion causes an 8% annual decline in health. Traditional linear forecasts recommend replacement after 5 years, but this model found that health drops sharply by 12% during the typhoon season of the third year. Based on this, the corresponding asset degradation prediction curve falls below the threshold at t=38 months, reminding maintenance departments to deploy replacements in advance to avoid accidents caused by mass failures.

[0072] Please see Figure 3 , Figure 3 This is a schematic diagram of a machine vision-based highway asset inspection device provided in an embodiment of this application. Figure 3 As shown, the machine vision-based highway asset inspection device 2 includes: a data acquisition module 21, an image enhancement module 22, a feature extraction module 23, a recognition and classification module 24, an asset positioning module 25, a spatiotemporal topology module 26, and a degradation prediction module 27.

[0073] The data acquisition module 21 is used to dynamically and synchronously acquire multi-source data in the highway area to obtain the original highway image matrix and the corresponding shooting and positioning data.

[0074] Image enhancement module 22 is used to perform adaptive filtering and geometric correction processing on the original highway image matrix to generate an enhanced image tensor.

[0075] In one possible implementation, the image enhancement module 22 is specifically used for: The original highway image matrix is ​​subjected to bilateral filtering to obtain a noise-optimized image matrix; The noise-optimized image matrix is ​​subjected to Retinex illumination compensation processing to obtain a contrast-enhanced image matrix; The contrast-enhanced image matrix is ​​subjected to Brown-Conrad distortion correction to obtain a geometrically corrected image matrix; The geometrically corrected image matrix is ​​scaled according to a preset size standard to generate an enhanced image tensor.

[0076] The feature extraction module 23 is used to perform spatial-spectral joint feature extraction processing on the enhanced image tensor to obtain an asset feature vector set.

[0077] In one possible implementation, the feature extraction module 23 is specifically used for: The enhanced image tensor is subjected to convolutional neural network spatial feature extraction processing to obtain local texture feature vectors; The enhanced image tensor is subjected to Fast Fourier spectral feature extraction processing to obtain a global spectral feature vector; The local texture feature vector and the global spectral feature vector are concatenated and fused to obtain an asset feature vector set.

[0078] The identification and classification module 24 is used to perform multi-task classification processing on the asset feature vector set through a preset first deep learning model to obtain the asset category-location tensor.

[0079] In one possible implementation, the identification and classification module 24 is specifically used for: Construct an asset relationship topology graph based on the spatial distance of feature vectors in the asset feature vector set; The asset relationship topology graph is iteratively propagated using a pre-defined first deep learning model to obtain a node feature set. The first deep learning model is used to perform multi-task classification on the node feature set to obtain the asset class probability distribution and image location bounding box coordinates of each feature node. The asset class probability distribution is filtered according to a preset confidence threshold to obtain the asset class label for each feature node; The asset category label and the corresponding image location bounding box coordinates are encapsulated to obtain the asset category-location tensor.

[0080] The asset positioning module 25 is used to perform tight-coupled geolocation mapping processing on the asset category-location tensor based on the captured positioning data to generate a list of geographic coordinate assets.

[0081] In one possible implementation, the asset location module 25 is specifically used for: The image location bounding box coordinates are transformed using a first coordinate system based on preset camera calibration parameters to obtain the first physical coordinates of each asset. The collected inertial navigation data and the captured positioning data are subjected to Kalman filtering tight coupling processing to generate high-precision vehicle pose data. Based on the vehicle pose data, the first physical coordinates are transformed using the second coordinate system to obtain the geodetic coordinates of each asset. The geodetic coordinates are converted to latitude and longitude using WGS84 to obtain the latitude and longitude data of each asset. The latitude and longitude data and the corresponding asset category labels are then mapped and encapsulated to generate a list of geographic coordinate assets.

[0082] The spatiotemporal topology module 26 is used to perform spatiotemporal topology analysis on the geographic coordinate asset list to obtain a highway asset map.

[0083] In one possible implementation, the spatiotemporal topology module 26 is specifically used for: A spatial association topology map is constructed based on a preset distance threshold and the geographic coordinate asset list; Historical detection datasets are obtained based on the geographic coordinate asset list, and the feature degradation rate of each asset is calculated based on the historical detection datasets and the geographic coordinate asset list to generate the current health assessment value for each asset. The current health assessment value and the spatial association topology map are structured and encapsulated to obtain a highway asset map.

[0084] The degradation prediction module 27 is used to perform time-series forward evolution prediction processing based on the highway asset map using a preset second deep learning model to obtain an asset degradation prediction curve.

[0085] In one possible implementation, the degradation prediction module 27 is specifically used for: Based on the time sequence, the health data of the target assets in the highway asset map is extracted and processed to obtain the historical health time series of each target asset. The historical health time series is modeled using a pre-defined second deep learning model to obtain an evolutionary model; The evolutionary model is subjected to regression decoding processing by the second deep learning model to obtain the health prediction value of the target asset; The historical health time series and the predicted health value are subjected to spline interpolation to obtain the asset degradation prediction curve.

[0086] The data acquisition module 21, image enhancement module 22, feature extraction module 23, recognition and classification module 24, asset location module 25, spatiotemporal topology module 26, and degradation prediction module 27 can all be implemented in software or hardware. For example, the implementation of the data acquisition module 21 will be described below. Similarly, the implementation of the image enhancement module 22, feature extraction module 23, recognition and classification module 24, asset location module 25, spatiotemporal topology module 26, and degradation prediction module 27 can refer to the implementation of the data acquisition module 21.

[0087] As an example of a software functional unit, the data acquisition module 21 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the data acquisition module 21 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0088] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0089] As an example of a hardware functional unit, the data acquisition module 21 may include at least one computing device, such as a server. Alternatively, the data acquisition module 21 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0090] The data acquisition module 21 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the data acquisition module 21 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the data acquisition module 21 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0091] See Figure 4 As shown, Figure 4 This is a schematic diagram of a computing device provided in this application. The computing device 100 includes: a processor 104, a communication interface 108, a bus 102, and a memory 106. The processor 104, the communication interface 108, and the memory 106 communicate via the bus 102. In practical applications, communication can also be achieved through other means such as wireless transmission; however, this is not limited here.

[0092] The computing device 100 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memory in the computing device 100.

[0093] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0094] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0095] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).

[0096] Memory 106 may include volatile memory, such as random access memory (RAM). Memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0097] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of the aforementioned data acquisition module 21, image enhancement module 22, feature extraction module 23, identification and classification module 24, asset positioning module 25, spatiotemporal topology module 26 and degradation prediction module 27, respectively, thereby realizing the machine vision-based highway asset inspection method. That is, the memory 106 stores instructions for executing the machine vision-based highway asset inspection method.

[0098] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a machine vision-based highway asset inspection method, or instruct the computing device to perform a machine vision-based highway asset inspection method.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for inspecting highway assets based on machine vision, characterized in that, The method includes: Multi-source data is dynamically and synchronously collected in the highway area to obtain the original highway image matrix and the corresponding shooting and positioning data. The original highway image matrix is ​​subjected to adaptive filtering and geometric correction to generate an enhanced image tensor; Spatial-spectral joint feature extraction processing is performed on the enhanced image tensor to obtain an asset feature vector set; The asset feature vector set is subjected to multi-task classification processing through a preset first deep learning model to obtain the asset category-location tensor; Based on the captured location data, the asset category-location tensor is subjected to tight-coupled geolocation mapping processing to generate a list of geographic coordinate assets; Spatiotemporal topology analysis is performed on the geographic coordinate asset list to obtain a highway asset map; Using a pre-defined second deep learning model, a time-series forward evolution prediction process is performed based on the highway asset map to obtain an asset degradation prediction curve.

2. The machine vision-based highway asset inspection method according to claim 1, characterized in that, The adaptive filtering and geometric correction process performed on the original highway image matrix to generate an enhanced image tensor includes: The original highway image matrix is ​​subjected to bilateral filtering to obtain a noise-optimized image matrix; The noise-optimized image matrix is ​​subjected to Retinex illumination compensation processing to obtain a contrast-enhanced image matrix; The contrast-enhanced image matrix is ​​subjected to Brown-Conrad distortion correction to obtain a geometrically corrected image matrix; The geometrically corrected image matrix is ​​scaled according to a preset size standard to generate an enhanced image tensor.

3. The machine vision-based highway asset inspection method according to claim 1, characterized in that, The step of performing spatial-spectral joint feature extraction processing on the enhanced image tensor to obtain an asset feature vector set includes: The enhanced image tensor is subjected to convolutional neural network spatial feature extraction processing to obtain local texture feature vectors; The enhanced image tensor is subjected to Fast Fourier spectral feature extraction processing to obtain a global spectral feature vector; The local texture feature vector and the global spectral feature vector are concatenated and fused to obtain an asset feature vector set.

4. The machine vision-based highway asset inspection method according to claim 1, characterized in that, The step of performing multi-task classification processing on the asset feature vector set using a preset first deep learning model to obtain the asset category-location tensor includes: Construct an asset relationship topology graph based on the spatial distance of feature vectors in the asset feature vector set; The asset relationship topology graph is iteratively propagated using a pre-defined first deep learning model to obtain a node feature set. The first deep learning model is used to perform multi-task classification on the node feature set to obtain the asset class probability distribution and image location bounding box coordinates of each feature node. The asset class probability distribution is filtered according to a preset confidence threshold to obtain the asset class label for each feature node; The asset category label and the corresponding image location bounding box coordinates are encapsulated to obtain the asset category-location tensor.

5. The machine vision-based highway asset inspection method according to claim 4, wherein the asset category-location tensor includes asset category labels and image location bounding box coordinates, characterized in that, The step of performing tight-coupled geolocation mapping processing on the asset category-location tensor based on the captured positioning data to generate a list of geographic coordinate assets includes: The image location bounding box coordinates are transformed using a first coordinate system based on preset camera calibration parameters to obtain the first physical coordinates of each asset. The collected inertial navigation data and the captured positioning data are subjected to Kalman filtering tight coupling processing to generate high-precision vehicle pose data. Based on the vehicle pose data, the first physical coordinates are transformed using the second coordinate system to obtain the geodetic coordinates of each asset. The geodetic coordinates are converted to latitude and longitude using WGS84 to obtain the latitude and longitude data of each asset. The latitude and longitude data and the corresponding asset category labels are then mapped and encapsulated to generate a list of geographic coordinate assets.

6. The machine vision-based highway asset inspection method according to claim 1, characterized in that, The process of performing spatiotemporal topological analysis on the geographic coordinate asset list to obtain a highway asset map includes: A spatial association topology map is constructed based on a preset distance threshold and the geographic coordinate asset list; Historical detection datasets are obtained based on the geographic coordinate asset list, and the feature degradation rate of each asset is calculated based on the historical detection datasets and the geographic coordinate asset list to generate the current health assessment value for each asset. The current health assessment value and the spatial association topology map are structured and encapsulated to obtain a highway asset map.

7. The machine vision-based highway asset inspection method according to claim 1, characterized in that, The step of performing time-series forward evolution prediction processing based on the highway asset map using a preset second deep learning model to obtain the asset degradation prediction curve includes: Based on the time sequence, the health data of the target assets in the highway asset map is extracted and processed to obtain the historical health time series of each target asset. The historical health time series is modeled using a pre-defined second deep learning model to obtain an evolutionary model; The evolutionary model is subjected to regression decoding processing by the second deep learning model to obtain the health prediction value of the target asset; The historical health time series and the predicted health value are subjected to spline interpolation to obtain the asset degradation prediction curve.

8. A machine vision-based highway asset inspection device, characterized in that, The device includes: The data acquisition module is used to dynamically and synchronously acquire multi-source data in the highway area to obtain the original highway image matrix and the corresponding shooting and positioning data. The image enhancement module is used to perform adaptive filtering and geometric correction on the original highway image matrix to generate an enhanced image tensor. The feature extraction module is used to perform spatial-spectral joint feature extraction processing on the enhanced image tensor to obtain an asset feature vector set; The identification and classification module is used to perform multi-task classification processing on the asset feature vector set through a preset first deep learning model to obtain the asset category-location tensor. The asset location module is used to perform tight-coupled geolocation mapping processing on the asset category-location tensor based on the captured location data to generate a list of geographic coordinate assets. The spatiotemporal topology module is used to perform spatiotemporal topology analysis on the geographic coordinate asset list to obtain a highway asset map. The degradation prediction module is used to perform time-series forward evolution prediction processing based on the highway asset map using a preset second deep learning model to obtain an asset degradation prediction curve.

9. A computing device, characterized in that, The computing device includes: At least one processor; and, A memory and a communication interface that are communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor implements the machine vision-based highway asset inspection method according to any one of claims 1 to 7 by executing the instructions stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the machine vision-based highway asset inspection method according to any one of claims 1 to 7.

Images