Rail transit inspection equipment intelligent system

Abstract

The application discloses a kind of rail transit inspection equipment intelligent system, it is related to equipment inspection technical field, including: handheld terminal, for collecting panoramic dataset, analysis panoramic dataset generates task planning scheme, completes initial calibration of projection by attitude solution algorithm, image generation fault preliminary screening list of reasoning detection point, adjust projection position by adaptive correction algorithm, reasoning generates secondary fault confirmation list;For suspected fault point position generates fault evidence dataset;Finally, reasoning fault evidence dataset, generates fault research result;Cloud server is used to show the result of the multiple handheld terminal generation summary.This application realizes visual data acquisition-real-time feature extraction-intelligent fault identification-position projection calibration-result visualization display, improves the fluency and efficiency of visual detection in the process of inspection;At the same time, handheld terminal is equipped with special projection lens, which solves the problem of inaccurate visual position calibration and large collection deviation in traditional scheme.

Description

Technical Field

[0001] This application relates to the field of equipment inspection technology, and in particular to an equipment inspection system based on a handheld terminal. Background Technology

[0002] Equipment inspection is a core and fundamental task in industrial operation and maintenance systems. Its purpose is to promptly detect potential faults such as wear, deformation, loosening, damage, and abnormal noises in equipment through routine condition monitoring, ensuring production continuity and equipment operational safety. Traditional manual inspection relies entirely on visual observation, experience-based judgment, and manual recording by inspectors. In practice, this method suffers from inherent drawbacks such as high missed inspection rates, inconsistent fault judgment standards, low efficiency, and significant risks in complex environments. Furthermore, the inspection data is fragmented and disorganized, failing to form a digital and traceable operational loop, and thus struggling to meet the demands of modern industrial development for intelligence, standardization, and high efficiency.

[0003] To replace traditional manual inspections, intelligent equipment such as drones, inspection robots, and automated testing platforms are gradually being applied to industrial scenarios. These devices, relying on technologies such as autonomous navigation, high-definition vision, and LiDAR, can achieve automated data collection and back-end analysis in open environments, reducing manual labor intensity under standardized working conditions. However, in practical applications, automated inspection equipment suffers from insurmountable general technical defects: First, its spatial adaptability is extremely poor. It cannot enter enclosed and confined spaces such as the bottom of the equipment, narrow gaps, internal cavities, and concealed areas of irregular structures due to limitations in its size, walking mechanism, and navigation conditions, creating numerous blind spots. Second, deployment and maintenance costs are high, requiring pre-set tracks, site modifications, and regular maintenance, making large-scale deployment difficult. Third, fault diagnosis is delayed; it can only complete data collection and needs to be transmitted back to the back-end for offline analysis, making it impossible to verify suspected faults on-site in real time.

[0004] Existing handheld inspection terminals, as manual auxiliary tools, suffer from core problems such as insufficient application of computer vision technology and low functional integration. Most terminals only have basic functions such as taking photos, recording videos, and text input, and lack artificial intelligence visual recognition models, making it impossible to achieve automatic fault detection and analysis. At the same time, existing terminals do not incorporate posture perception technology to achieve dynamic projection visual guidance. When working in confined spaces, they cannot accurately mark the fault location through visual projection, which easily leads to image acquisition deviations, missed detections, and false detections, seriously affecting the accuracy and reliability of visual recognition. Summary of the Invention

[0005] This application provides an intelligent system for rail transit inspection equipment to solve the aforementioned problems in the prior art.

[0006] This application provides an intelligent system for rail transit inspection equipment, including: The handheld terminal, held by inspection personnel, is used to acquire panoramic images, audio, and spatial coordinates via a camera, microphone, and inertial navigation module. After preprocessing, a panoramic dataset is generated. Image segmentation, feature matching, and improved A / B mapping are employed. The algorithm analyzes the panoramic dataset to generate the optimal task planning scheme. A visual rendering engine renders the task planning scheme, and an attitude calculation algorithm completes the initial projection calibration. Then, initial projection calibration is performed at the detection points. A lightweight AI vision model is used to infer the image of the initial projection calibration position in the task planning scheme, generating a preliminary fault screening list. A three-axis attitude sensor collects attitude data, and an adaptive correction algorithm adjusts the projection position to ensure it always matches the initial projection calibration position. Inspection personnel approach the projection position based on the preliminary fault screening list and use a handheld terminal to infer the collected images, generating a secondary fault confirmation list. Images of suspected fault locations in the secondary fault confirmation list are acquired using 4K ultra-high-definition acquisition, ESPCN super-resolution reconstruction, and multimodal feature fusion technology to generate a fault evidence dataset. A lightweight AI vision model is used to infer the fault evidence dataset, generating fault assessment results. The cloud server is used to summarize and display the fault assessment results generated by multiple handheld terminals.

[0007] The intelligent system for rail transit inspection equipment disclosed in this application has the following advantages: 1. Integrated edge vision functionality. High-definition visual acquisition, AI visual recognition, screen visual display, and posture-adaptive projection guidance are all integrated into the handheld terminal, achieving a closed-loop edge process from visual data acquisition to real-time feature extraction, intelligent fault identification, position projection calibration, and result visualization. Visual data processing latency is ≤80ms, significantly improving the smoothness and efficiency of visual inspection of industrial equipment.

[0008] 2. Posture-Adaptive Projection Visual Guidance. The handheld terminal is equipped with a dedicated projection lens and a built-in gyroscope and accelerometer to sense changes in handheld posture in real time. Through visual posture calculation algorithms, the projection angle and position are dynamically corrected. Regardless of whether the terminal is held upright, at an angle, or shooting from a low position, the fault acquisition location can be accurately projected onto the device surface in the form of a visual spot. The posture adaptation error is ≤1mm, which fundamentally solves the problems of inaccurate visual position calibration and large acquisition deviation in traditional solutions.

[0009] 3. Visualized Interaction on the Screen. The terminal display screen synchronously shows real-time visual acquisition images, AI visual recognition results, projection calibration positions, task planning paths, and fault details. The fully visual interactive interface is intuitive and simple, allowing standardized visual data acquisition to be completed without professional training, completely eliminating reliance on personnel experience, and enabling new employees to quickly get started. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating the composition of an intelligent system for rail transit inspection equipment, provided as an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] Figure 1 This is a schematic diagram illustrating the composition of an intelligent system for rail transit inspection equipment, provided as an embodiment of this application. This application provides an intelligent system for rail transit inspection equipment, including: The handheld terminal, held by inspection personnel, is used to acquire panoramic images, audio, and spatial coordinates via a camera, microphone, and inertial navigation module. After preprocessing, a panoramic dataset is generated. Image segmentation, feature matching, and improved A / B mapping are employed. The algorithm analyzes the panoramic dataset to generate the optimal task planning scheme. A visual rendering engine renders the task planning scheme, and an attitude calculation algorithm completes the initial projection calibration. Then, initial projection calibration is performed at the detection points. A lightweight AI vision model is used to infer the image of the initial projection calibration position in the task planning scheme, generating a preliminary fault screening list. A three-axis attitude sensor collects attitude data, and an adaptive correction algorithm adjusts the projection position to ensure it always matches the initial projection calibration position. Inspection personnel approach the projection position based on the preliminary fault screening list and use a handheld terminal to infer the collected images, generating a secondary fault confirmation list. Images of suspected fault locations in the secondary fault confirmation list are acquired using 4K ultra-high-definition acquisition, ESPCN super-resolution reconstruction, and multimodal feature fusion technology to generate a fault evidence dataset. A lightweight AI vision model is used to infer the fault evidence dataset, generating fault assessment results. The cloud server is used to summarize and display the fault assessment results generated by multiple handheld terminals.

[0014] For example, before formally carrying out the inspection of the equipment, the system needs to be initialized first. The initialization process is divided into hardware module self-test, software environment loading, AI model deployment, operation parameter configuration and core sensor calibration.

[0015] During the hardware module self-test, the handheld terminal's main controller sequentially issues test commands to perform a comprehensive check on the operating status, power supply voltage, and communication links of all core hardware components. This includes checking the focusing function, image sensor, and image transmission link of the high-definition visual acquisition camera; verifying the display resolution, color reproduction, and touch response speed of the high-definition display; checking the projection brightness, spot clarity, and angle adjustment motor of the attitude-adaptive projection lens; verifying the attitude data output frequency and zero-point drift value of the three-axis gyroscope and accelerometer; checking the signal strength and data transmission rate stability of the Wi-Fi / 5G communication module; checking the remaining capacity and read / write speed of the local storage module to meet data storage requirements; and checking the power supply voltage and battery life of the built-in battery. If any hardware module malfunctions, the handheld terminal immediately displays a fault message on the screen and terminates the initialization process; if all hardware functions normally, it proceeds to the next stage.

[0016] During the software environment loading phase, the handheld terminal automatically loads the embedded operating system, visual data preprocessing engine, posture calculation algorithm package, projection control driver, MQTT cloud communication protocol, and data storage management program. After all software components are loaded, process scheduling and memory allocation are automatically performed to ensure that the software environment is conflict-free and error-free, providing a stable operating environment for visual data processing and AI inference.

[0017] In the AI ​​model deployment stage, the handheld terminal loads the pre-trained lightweight AI visual recognition model from local storage to the running memory, completes the initialization of the model inference engine, operator optimization, and memory allocation, ensuring that the model can quickly respond to image input and realize real-time visual inference on the device.

[0018] During the parameter configuration phase, inspection personnel input the core parameters for this inspection through the touch screen of their handheld terminal: the scope of the work area, the type of inspection equipment, the type of visual recognition fault, the fault confidence threshold, the data storage path, and the cloud server address. After the parameters are entered, the handheld terminal automatically saves and applies the changes.

[0019] In the core sensor calibration stage, the handheld terminal automatically initiates the calibration process, which involves calibrating the focal length and correcting the distortion of the high-definition visual acquisition camera to eliminate lens optical errors; calibrating the initial angle of the attitude adaptive projection lens to determine the projection reference position; and calibrating the three-axis gyroscope and three-axis accelerometer to eliminate attitude perception drift errors and ensure accurate attitude data acquisition.

[0020] To quantify the initialization effect, this application introduces the overall initialization success rate: or total =0.6 or hard +0.4 or soft in, or total The overall initialization success rate of the system. or hard To improve the success rate of hardware module initialization or soft To improve the success rate of loading software and AI models.

[0021] When the overall system initialization success rate is ≥99%, the formal inspection process will begin.

[0022] After initialization, the inspection personnel carried handheld terminals into the work area and started the full-scene visual data acquisition process.

[0023] During the panoramic image acquisition phase, the inspection personnel slowly move around the equipment to be inspected. The handheld terminal's high-definition visual acquisition camera automatically activates the 360° panoramic acquisition mode, capturing images of the entire equipment area at a resolution of 1920×1080 and a frame rate of 30fps. This covers all inspection points, including the equipment's surface, bottom, narrow gaps, internal cavities, and irregularly shaped structures. The camera automatically adjusts the exposure, white balance, and focus distance according to the ambient light to ensure that the acquired visual images are clear, without blur, overexposure, or underexposure.

[0024] During the audio acquisition phase, the handheld terminal's built-in microphone is activated simultaneously to collect audio signals such as abnormal noises, friction sounds, and discharge sounds during device operation in real time. These signals serve as auxiliary features for visual recognition, compensating for acoustic fault features that cannot be covered by pure visual detection and improving the comprehensiveness of subsequent fault identification.

[0025] In the spatial coordinate acquisition stage, the inertial navigation module in the handheld terminal collects the three-dimensional spatial coordinates of the work area in real time, establishes the spatial position coordinate system of the equipment, and provides a position reference for subsequent projection guidance and spatial positioning. In a closed space without GPS signal, accurate position acquisition can be achieved by relying solely on inertial navigation.

[0026] Furthermore, the preprocessing of panoramic images and audio includes: using Gaussian filtering to remove noise and bilateral filtering to preserve edge features in panoramic images, eliminating image noise caused by environmental dust and light interference; using Wiener filtering to remove environmental noise in audio, retaining effective acoustic features related to equipment malfunctions; automatically screening the effective data after filtering, removing invalid image frames that are blurry, occluded, or abnormally exposed, and removing blank audio without effective acoustic features, ensuring that all acquired data meets the requirements of subsequent visual processing and AI recognition.

[0027] To evaluate the coverage effect of the data collection, this application also introduces spatial coverage in its embodiments: i =( S cap / S area )×100% in, i For the spatial coverage of the work area, S cap This represents the area of ​​the collected data. S area This refers to the total area of ​​the work area. When the spatial coverage... i If the result is ≥99%, proceed to the next step.

[0028] Furthermore, the method for generating a task planning scheme includes: the handheld terminal identifies the type, structural distribution, and component locations of the equipment to be inspected using an image segmentation algorithm; it compares the information with a visual fault knowledge base using a feature matching algorithm to mark fault-prone points and high-risk detection areas on the equipment; it establishes a three-dimensional visual model of the work area using spatial coordinates; and then, based on the improved A... The path planning algorithm constructs a dual-objective optimization model of minimum detection time and maximum detection coverage, and solves it to obtain the task planning scheme.

[0029] Specifically, during the model solving process, the detection sequence of high-risk fault points is prioritized to shorten ineffective movement time. During the planning process, the handheld terminal automatically integrates the detection point sequence, personnel movement path, and high-risk fault point list to form a complete visual inspection task planning scheme, which is adapted to the working habits of human handheld terminals and ensures that the subsequent inspection process is efficient and standardized.

[0030] Image segmentation algorithms are used to separate the main body, components, and background regions from panoramic images. The specific execution process is as follows: First, the panoramic image is uniformly scaled to a resolution of 512×512, and pixel normalization and channel alignment are completed. Then, the encoder enters the downsampling stage, using four sets of convolutional + pooling combination layers to gradually extract shallow contour features and deep semantic features of the image. After each convolutional layer, BatchNorm normalization and ReLU activation function are applied to compress the feature dimension and retain key segmentation features such as device edges, structures, and gaps. Next, the decoder enters the upsampling stage, using deconvolutional layers to restore the feature map size layer by layer. The feature maps of the corresponding layers of the encoder are stitched together through a skip connection structure to avoid the loss of edge information caused by downsampling. Finally, a single-channel segmentation probability map is generated through a 1×1 convolutional layer, and a binary segmentation mask is output through the Sigmoid activation function to divide the image into three semantic regions: device detection region, background region, and fault-prone hidden region. The segmentation results are directly used to label the detection points and spatial range.

[0031] The feature matching algorithm employs a visual feature matching algorithm combining SIFT feature extraction and FLANN fast matching to align the panoramic image with the equipment fault knowledge base. The specific execution process is as follows: First, SIFT key points are extracted from the segmented equipment region image. Scale-invariant feature points are detected using the Difference of Gaussian pyramid, and stable corner points, edge points, and structural abrupt change points are selected as feature points. A 128-dimensional feature descriptor is generated for each feature point to ensure feature invariance under rotation, scaling, and illumination changes. Then, the real-time image feature descriptor and the standard equipment feature descriptor from the knowledge base are input into the FLANN approximate nearest neighbor matching library to construct a random KD-Tree index structure. This allows for rapid retrieval of the optimal matching feature pair, initially completing the feature matching between the image and the standard template. Next, the RANSAC random sampling consensus algorithm is used to eliminate mismatched feature pairs, and noisy matching points are filtered through homography matrix fitting, retaining the accurately matched feature point set. Finally, based on the positional distribution of the matched feature points, faulty components and key detection points in the image are labeled.

[0032] Improvement A Path planning algorithms in traditional A Based on the path planning algorithm, fault risk weights are integrated to generate the optimal detection path and point sequence. The specific execution process is as follows: First, initialize the Open and Close lists of the algorithm, add the starting point of the work area to the Open list, and set the target point to the set of all high-risk detection points; then construct an improved heuristic function, which weights and integrates Euclidean distance and fault risk priority. The heuristic function is h(n) = α·d(n) + β·p(n), where d(n) is the Euclidean distance from the node to the target point, p(n) is the fault risk level of the point, and α = 0.7 and β = 0.3 are weight coefficients, prioritizing the expansion of high-risk point nodes; next, traverse the nodes in the Open list, calculate the cost function f(n) = g(n) + h(n) for each node, where g(n) is the actual movement cost from the starting point to the current node, select the node with the lowest cost as the current expansion node, and move it to the Close list; then traverse the 8 feasible neighboring nodes of the current node, update the node cost and parent node pointer, and repeat the expansion until all high-risk points are included in the path; finally, generate the optimal detection path by backtracking through the parent node, and output the detection point sequence in the order of the path. Furthermore, in the initial projection calibration, the handheld terminal activates the attitude-adaptive projection lens, and combines the initial attitude data collected by the three-axis gyroscope and three-axis accelerometer. The initial projection angle of the projection lens is calculated through the attitude calculation algorithm, and the high-risk detection points in the task planning scheme are projected onto the surface of the equipment in the form of circular light spots. This is the initial projection calibration, which completes the preliminary matching between the projection light spots and the detection points.

[0033] Specifically, before projection begins, the handheld terminal's visual rendering engine converts the task planning scheme into a visual interface, displaying the following in full dimensions on a high-definition screen: the optimal travel path is drawn with a solid green line, regular inspection points are marked with blue dots, and high-risk, fault-prone points are marked with red highlighted boxes. The point number, inspection priority, and operation prompts are displayed simultaneously, allowing inspection personnel to quickly obtain inspection task information without needing to memorize complex inspection lists. After reviewing this information, inspection personnel can begin moving along the path and collect images of each inspection point, including regular and high-risk points, in the order of their movement.

[0034] The attitude calculation algorithm used in this embodiment is a quaternion-based gradient descent attitude calculation algorithm, deployed in the embedded chip of a handheld terminal for real-time calculation of the terminal's spatial attitude. The specific execution process is as follows: First, raw inertial data is collected at high frequency using a three-axis gyroscope and a three-axis accelerometer, with a sampling frequency set to 100Hz. Outlier removal and sliding window filtering are performed on the raw data to eliminate noise interference caused by vibration and jitter. Then, the quaternion q = [q0, q1, q2, q3] is initialized, where q is a quaternion vector, q0 is the real part of the quaternion, and q1-q3 are the imaginary vector components of the X, Y, and Z axes, respectively. Accelerometer measurements are used as observations to construct an attitude error function, which is then iteratively corrected using the gradient descent method. Positive quaternions are used to minimize attitude estimation errors. Then, the quaternions are updated based on gyroscope angular velocity data, and the first-order Runge-Kutta method is used to solve the quaternion differential equations, ensuring the real-time and continuous nature of attitude updates. The corrected quaternions are then converted into roll, pitch, yaw, and Euler angle outputs to complete the full solution of the terminal's spatial attitude. Finally, a zero-rate update mechanism is introduced to automatically correct attitude drift when the terminal is stationary. The attitude solution accuracy is ≤0.5°, and the response delay is ≤10ms, providing accurate attitude data support for projection position correction.

[0035] Furthermore, the lightweight AI vision model adopts a lightweight architecture based on MobileNetV3-Lite and YOLOv8-Nano. This model is specifically designed for low-computing-power environments on the edge, featuring a small number of parameters, fast inference speed, and millisecond-level response. Specifically, the model includes sequentially connected convolutional layers, depthwise separable convolutional layers, inverted residual bottleneck layers, and a detection head. The model preprocesses the input image, adjusting the image size and normalizing pixel values. Then, it extracts image features through convolutional layers, depthwise separable convolutional layers, and inverted residual bottleneck layers. The detection head outputs fault confidence, bounding boxes, and fault categories, quickly completing full-area fault scanning and locating high-risk suspected fault locations.

[0036] Specifically, the lightweight AI vision model structure consists of three parts: the backbone network adopts MobileNetV3-Lite, which is built based on depthwise separable convolution to compress the number of parameters and computation. It improves the feature expression capability through the Hard-Swish activation function and outputs five sets of feature maps at different scales; the neck network adopts the PANet lightweight feature fusion structure, which upsamples, downsamples and concatenates the multi-scale feature maps to fuse shallow edge features and deep semantic features; the detection head adopts a coupled head structure to simultaneously complete target classification and bounding box regression, and output fault category, confidence and location box. The training process of this model is as follows: 120,000 visual fault images of industrial equipment are used as the dataset, covering 80 fault types. The dataset is divided into training, validation, and test sets in a 7:2:1 ratio. Random cropping, flipping, color gamut transformation, and mosaic enhancement are used to improve generalization. The AdamW optimizer is used with an initial learning rate of 1e-4 and a weight decay of 5e-4. The loss function is CIoU bounding box loss + cross-entropy classification loss + DFL distribution loss. The batch size is 32, the iteration is 200 epochs, and cosine annealing is used for learning rate scheduling. The final model has 1.8M parameters, 2.2 GFLOPs of computation, a single-frame inference speed of ≤80ms, and a fault recognition accuracy of ≥0.94.

[0037] After the initial fault screening is completed, the handheld terminal initiates the secondary visual fault confirmation process. In this process, while the handheld terminal adjusts its projection position, the three-axis gyroscope and three-axis accelerometer collect the handheld terminal's attitude data at a frequency of 100Hz, calculate the changes in roll, pitch, and yaw angles in real time, and determine whether the handheld terminal has changed its attitude. The attitude data is transmitted to the projection control module in real time. After receiving the attitude data, the projection control module calculates the projection offset through an adaptive correction algorithm and automatically controls the angle adjustment motor of the projection lens to correct the projection angle and position in real time. No matter how the terminal moves or tilts, the projection spot always firmly locks onto the suspected fault location, solving the calibration failure problem caused by attitude changes in traditional guidance solutions.

[0038] Specifically, the adaptive correction algorithm is used to dynamically calibrate the position of the projected light spot. Its execution process is as follows: First, the terminal Euler angle data output by the attitude calculation algorithm is acquired in real time, and the attitude offset Δθ is calculated by comparing it with the initial calibration attitude, including roll offset, pitch offset, and yaw offset. Then, a projection position correction model is constructed, converting the attitude offset into an angle correction value for the projection lens. The correction model establishes a linear mapping relationship between the terminal attitude and the projection angle based on spatial geometric mapping, and the correction coefficient K is obtained through offline calibration. Next, the dual-axis micro-servo motor inside the projection lens is driven to adjust the horizontal and vertical projection angles according to the correction value, compensating for the light spot deviation caused by the attitude offset in real time. Simultaneously, the real-time images of the projected light spot and the fault location are acquired through the terminal camera, performing visual closed-loop feedback, calculating the pixel deviation between the light spot and the target location, and fine-tuning the correction value a second time, forming a closed-loop control of attitude perception, deviation calculation, angle correction, and visual feedback. The algorithm correction response time is ≤20ms, and the projection light spot positioning error is ≤1mm, maintaining accurate target location locking of the light spot under any handheld terminal attitude.

[0039] Following the guidance of the projected light spot, the inspection personnel approached the suspected fault location within 0.5m. The handheld terminal automatically collected 1080P high-definition visual images and auxiliary audio data, which were then transmitted to the local lightweight AI visual model for secondary inference. The model used the detailed features of the high-definition images to eliminate misjudged points caused by light interference, angle deviation, and foreign object obstruction, retaining only the real suspected faults.

[0040] For confirmed suspected fault locations at level two, the terminal initiates a high-definition visual forensics and multimodal feature fusion process. In this process, the handheld terminal acquires fault images of the suspected fault locations marked by projected light spots in 4K ultra-high-definition acquisition mode. These images are uncompressed, high-resolution, and detailed, clearly displaying minute fault features such as cracks, wear, and looseness in the equipment. Simultaneously, audio and spatial coordinates are acquired to supplement fault features beyond visual data. Then, the acquired fault images are optimized using the lightweight ESPCN super-resolution algorithm. Finally, the optimized fault images, along with the simultaneously acquired audio and spatial coordinates, are deeply fused and packaged into a unified format fault forensics dataset.

[0041] Specifically, this application employs the lightweight ESPCN subpixel convolutional super-resolution algorithm, deployed on a handheld terminal, to reconstruct high-definition faulty images into 4K super-resolution images. The model structure consists of three layers: the feature extraction layer uses three sets of 3×3 convolutional layers to extract shallow texture features and deep detail features from low-resolution images, and then applies the PReLU activation function after convolution to avoid gradient vanishing; the subpixel convolutional layer is the core structure, generating multi-channel feature maps through r² sets of convolutions, and converting the channel features into spatial resolution through subpixel rearrangement to achieve 2x super-resolution reconstruction without the checkerboard artifacts caused by deconvolution; the image reconstruction layer uses a 1×1 convolutional layer to fuse features and output a super-resolution image. The model is trained using a high-resolution image dataset of equipment failures, containing 50,000 pairs of low / high resolution images. The loss function uses MSE (mean squared error) loss plus perceptual loss, balancing pixel accuracy and visual quality. The optimizer is Adam, with an initial learning rate of 5e-4, a batch size of 16, and 150 epochs. The model has only 0.8M parameters, a single-frame super-resolution time of ≤50ms, and a peak signal-to-noise ratio of ≥32dB for the reconstructed image, which can clearly restore the minor fault features of equipment such as cracks and wear.

[0042] The process of deep fusion of fault images, audio, and spatial coordinates is as follows: First, for the 4K super-resolution fault image, a 256-dimensional visual feature vector is extracted using a lightweight AI vision model, preserving the shape, texture, and edge visual features of the fault. Second, for the audio, 40-dimensional MFCC audio features are extracted using Mel-frequency transform to characterize the frequency and amplitude acoustic features of the fault noise. Then, for the spatial coordinates, a position encoding algorithm is used to convert the three-dimensional coordinates (X,Y,Z) into a 64-dimensional position feature vector, realizing the characteristic representation of spatial location. Subsequently, the visual features, audio features, and position features are concatenated by channel to generate a 360-dimensional joint feature vector, and batch normalization is used to eliminate the dimensional differences of features from different modalities. Next, a 1×1 convolutional layer is used for feature dimensionality reduction and interactive learning to extract inter-modal correlation features and filter redundant noise features. Finally, a fully connected layer outputs a 128-dimensional fused feature vector as the input features for fault assessment.

[0043] For the fault evidence dataset, this application embodiment adopts a local inference method. In local inference, the handheld terminal inputs the fault evidence dataset into a lightweight AI vision model. The model quickly extracts fault features and outputs preliminary judgment results of fault type, location, and confidence level. The inference time is ≤80ms, ensuring real-time performance on site.

[0044] After the local inference is completed, the handheld terminal determines whether the fault is valid based on the confidence threshold, and finally determines the specific type, precise location and severity (general, moderate, severe) of the fault. The judgment results are displayed on the terminal screen in real time.

[0045] Furthermore, the fault assessment results include the fault type, location, and severity for each fault point. The handheld terminal uses image data acquired through binocular vision and triangulation principles to calculate the actual depth distance between the fault point and the handheld terminal, converting this into three-dimensional world coordinates of the work area. The positioning accuracy is ≤1mm, allowing for precise labeling of the fault's exact location on the equipment without manual measurement. The handheld terminal then integrates the three-dimensional world coordinates, fault type, fault severity, evidence images from the fault evidence dataset, and recognition confidence levels to generate a standardized fault information table containing all core fault-related information, facilitating subsequent storage, retrieval, and maintenance access.

[0046] The calculation process for depth distance is as follows: Z =( f · B ) / d x Where Z represents the depth distance of the fault. f For camera focal length, B The binocular baseline distance. d x This represents the image disparity value.

[0047] After obtaining the fault information table, the handheld terminal stores all the data generated during the inspection process into a local SQLite database in a standardized format. The database is divided into five data tables: task information table, visual acquisition data table, fault analysis data table, spatial positioning data table, and operation log table. All data is stored in categories and linked with indexes for easy retrieval and querying. When connected to the network, the handheld terminal automatically identifies newly added data and uploads unsynchronized visual data, fault information, and incremental task logs to the cloud server via the MQTT protocol, avoiding bandwidth waste caused by full transmission. When offline, the data is temporarily stored locally and automatically resumed upon reconnection, ensuring complete consistency between the end-to-end and cloud data.

[0048] Finally, the handheld terminal automatically reads data from the database and generates a report according to a standardized template. This report includes the following: 1) Basic inspection information, specifying the operation time, area, personnel, and equipment type; 2) Inspection overview, displaying spatial coverage, operation duration, and total number of inspection points; 3) Fault details, listing fault type, spatial coordinates, severity, and high-resolution images; 4) Handling suggestions, providing maintenance guidance for faults of different severity levels; and 5) Data statistics, displaying quantitative indicators such as fault identification accuracy, missed detection rate, and operation efficiency. Ultimately, the handheld terminal generates a PDF report, which can be directly stored, printed, and uploaded to the maintenance platform, meeting the needs of industrial maintenance archiving and repair reporting.

[0049] After the handheld terminal completes fault assessment and generates fault assessment results and a fault information table, it immediately uploads data such as fault type, fault spatial coordinates, fault severity, high-definition fault evidence images, detection point information, operators, and operation time to the cloud server via Wi-Fi / 5G module using the MQTT encryption protocol. Upon receiving the data from the handheld terminal, the cloud server first performs data integrity verification, removing abnormal data with format errors or missing fields. Then, it categorizes and summarizes the data according to four dimensions: detection equipment type, fault severity, operation area, and operation time, calculating the total number of faults in each area, the proportion of faults of different levels, and the distribution of faults on individual devices. Subsequently, based on the summarized data, it generates three types of visualized data: fault statistics reports, equipment fault spatial distribution heatmaps, and operation completion records.

[0050] The cloud server then displays the summary results through a web management interface. The interface presents a detailed list of faults, spatial coordinate positioning marks, and multi-dimensional statistical charts in real time. It supports filtering and searching by region, time, and fault level, and can export summary reports and ledger data.

[0051] The following is a brief description of the usage process of the inspection system proposed in this application, taking the inspection task in a rail transit scenario as an example: 1. The handheld terminal completes initialization, calibration of the vision sensor and projection lens, and configuration of the detection parameters for rail transit equipment; 2. Handheld terminals collect panoramic images, audio, and spatial coordinates of rail transit vehicles, tracks, electromechanical equipment, and signaling equipment to generate a full-scene panoramic dataset; 3. The handheld terminal parses the panoramic dataset to generate a sequence of inspection points, travel paths, and a list of high-risk points for rail transit equipment; 4. The handheld terminal renders task information on the screen, and the projection lens completes the initial calibration of high-risk points such as the undercarriage, gaps, and bogies; 5. The local lightweight AI vision model on the handheld terminal completes the initial screening of faults and projects a red light spot indicating a suspected fault. 6. The handheld terminal senses changes in posture in real time, dynamically corrects the projection position, and completes secondary fault confirmation by collecting data at close range; 7. The handheld terminal initiates 4K high-definition acquisition to complete visual evidence collection of faults and multimodal feature fusion; 8. Use handheld terminals to analyze and diagnose faults in rail transit equipment, determining the type, location, and severity of the fault; 9. Use binocular vision to calculate the three-dimensional spatial coordinates of the fault and generate a fault information table with precise location; 10. Structured storage of visual data throughout the entire process, synchronized to the cloud-based operation and maintenance platform after networking; 11. Automatically generate visual inspection reports for rail transit equipment, including fault details, high-definition images, and handling suggestions.

[0052] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0053] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An intelligent system for rail transit inspection equipment, characterized in that, include: The handheld terminal, held by inspection personnel, is used to acquire panoramic images, audio, and spatial coordinates via a camera, microphone, and inertial navigation module. After preprocessing, a panoramic dataset is generated. Image segmentation, feature matching, and improved A / B mapping are employed. The algorithm analyzes the panoramic dataset to generate the optimal task planning scheme; the visual rendering engine renders the task planning scheme; the initial projection calibration is completed through the pose calculation algorithm; and the initial projection calibration is performed at the detection points. A lightweight AI vision model is used to infer the image of the initially projected and calibrated position in the task planning scheme to generate a preliminary fault screening list. A three-axis attitude sensor is used to collect attitude data. An adaptive correction algorithm is used to adjust the projection position so that the adjusted projection position is always the same as the position calibrated by the initial projection. After the inspection personnel approach the projection position according to the initial fault screening list, the handheld terminal infers the collected image to generate a secondary fault confirmation list. Images of suspected fault locations in the secondary fault confirmation list are acquired using 4K ultra-high-definition acquisition, ESPCN super-resolution reconstruction, and multimodal feature fusion technology to generate a fault evidence dataset; the lightweight AI vision model is then used to infer the fault evidence dataset to generate fault assessment results. A cloud server is used to summarize and display the fault assessment results generated by multiple handheld terminals. In the initial projection calibration, the handheld terminal activates the attitude-adaptive projection lens, combines the initial attitude data collected by the three-axis gyroscope and three-axis accelerometer, calculates the initial projection angle of the projection lens through the attitude calculation algorithm, and projects the high-risk detection points in the task planning scheme onto the device surface in the form of a circular light spot, thus completing the initial matching between the projection light spot and the detection point. During the process of adjusting the projection position, the handheld terminal uses a three-axis gyroscope and a three-axis accelerometer to collect attitude data of the handheld terminal at a frequency of 100Hz, calculates the changes in roll, pitch and yaw angles in real time, and determines whether the handheld terminal has changed its attitude. The attitude data is transmitted to the projection control module in real time. After receiving the attitude data, the projection control module calculates the projection offset through an adaptive correction algorithm, automatically controls the angle adjustment motor of the projection lens, and corrects the projection angle and position in real time.

2. The intelligent system for rail transit inspection equipment according to claim 1, characterized in that, Preprocessing of the panoramic image and the audio includes: The panoramic image is processed by Gaussian filtering to remove noise and bilateral filtering to preserve edge features, eliminating image noise caused by environmental dust and light interference; the audio is processed by Wiener filtering to remove environmental noise and preserve effective acoustic features related to equipment malfunction.

3. The intelligent system for rail transit inspection equipment according to claim 1, characterized in that, The method for generating the task planning scheme includes: The handheld terminal uses an image segmentation algorithm to identify the type, structural distribution, and component locations of the device to be inspected. It then uses a feature matching algorithm to compare against a visual fault knowledge base, marking fault-prone points and high-risk detection areas on the device. Combined with the spatial coordinates, it establishes a three-dimensional visual model of the work area, and then, based on an improved A... The path planning algorithm constructs a dual-objective optimization model of minimum detection time and maximum detection coverage, and solves it to obtain the task planning scheme.

4. The intelligent system for rail transit inspection equipment according to claim 1, characterized in that, The lightweight AI vision model adopts a lightweight architecture based on MobileNetV3-Lite and YOLOv8-Nano, including a convolutional layer, a depth-separable convolutional layer, an inverted residual bottleneck layer, and a detection head connected in sequence.

5. The intelligent system for rail transit inspection equipment according to claim 1, characterized in that, In the process of generating the fault evidence dataset, the handheld terminal acquires fault images at suspected fault locations marked by projected light spots in 4K ultra-high-definition acquisition mode, while simultaneously acquiring the audio and spatial coordinates; then, the acquired fault images are optimized using the lightweight ESPCN super-resolution algorithm; finally, the optimized fault images, along with the simultaneously acquired audio and spatial coordinates, are deeply fused and packaged into the fault evidence dataset in a unified format.

6. The intelligent system for rail transit inspection equipment according to claim 1, characterized in that, The fault assessment results include the fault type, fault location, and fault severity for each fault point. The handheld terminal uses image data acquired through binocular vision to calculate the actual depth distance between the fault point and the handheld terminal using the principle of triangulation, and then converts it into three-dimensional world coordinates of the work area. The handheld terminal then integrates the three-dimensional world coordinates, the fault type, the fault severity, the evidence images in the fault evidence dataset, and the recognition confidence level to generate a standardized fault information table.

Images