Special equipment inspection and detection method and system based on machine vision and medium

Abstract

The invention discloses a special equipment inspection and detection method and system based on machine vision and a medium, and belongs to the technical field of special equipment detection. The method comprises the following steps: an equipment variety determination stage: automatically identifying the type and basic information of current inspection and detection equipment; in the equipment image acquisition stage, equipment images are acquired through multispectral fusion; an edge calculation pre-processing stage: pre-processing the equipment image to obtain pre-processed tensor data; a lightweight model reasoning stage; a component ID identification stage; a test item and clause matching stage: converting the TSG clauses into executable logic to obtain a test item list; an AR prompting stage; an original data recording stage; in the blockchain evidence storage and submission stage, the check record data is linked; and an inspection report automatic generation stage: finally automatically generating a report. According to the method, edge calculation, multi-modal interaction and inspection automation technologies are fused, and the problems that traditional inspection depends on manpower, efficiency is low and standardization is insufficient are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of special equipment inspection and testing, and particularly relates to a method, system and medium for special equipment inspection and testing based on machine vision. Background Art

[0002] Special equipment refers to electromechanical or pressure-bearing equipment involving life safety and greater risks, including ten categories such as elevators, boilers, pressure vessels, and lifting machinery. According to the requirements of laws and regulations such as the Special Equipment Safety Law and the Special Equipment Safety Supervision Regulations, equipment that has not been inspected and tested or fails the inspection and testing is prohibited from being used. Each type of special equipment has a dedicated inspection regulation. For example, elevators have the Rules for Elevator Supervision and Periodic Inspection (TSG T7001-2023) and the Rules for Elevator Self-Inspection (TSG T7008-2023), and lifting machinery has the Safety Technical Code for Lifting Machinery (TSG 51-2023).

[0003] According to the Special Equipment Catalog, it can be divided into 10 major categories, including: elevators, lifting machinery, special-purpose motor vehicles within factories and yards, large-scale amusement facilities, passenger ropeways, boilers, pressure vessels, pressure pipelines, pressure pipeline components, and safety accessories. Further subdivided, it is divided into 48 small categories and 99 varieties in total. Taking one variety, "traction-driven passenger elevator", as an example, according to its structural composition and position in the elevator inspection regulation, it can be divided into: technical data, machine space, hoistway, electrical equipment (devices) and control, drive host, suspension compensation device and rotating components, car and counterweight, landing doors and car doors, etc. units. Taking the landing doors and car doors unit as an example, 5 key components such as landing doors, car doors, automatic closing landing door devices, emergency unlocking devices, and door lock devices need to be identified.

[0004] When inspecting and testing special equipment, inspection and testing personnel need to conduct inspections one by one according to each requirement of the inspection regulation, fill in the original inspection records, and finally generate an inspection report. Therefore, it is very inconvenient for inspectors to operate the test while recording, and the inspection efficiency is low. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method, system and medium for special equipment inspection and testing based on machine vision.

[0006] The purpose of the present invention is achieved through the following technical solutions: In the first aspect of the present invention, there is provided: A method for special equipment inspection and testing based on machine vision, including the following steps: In the equipment variety determination stage, by obtaining the equipment location information or inputting the equipment number, combined with the equipment database pre-stored in the cloud, automatically identify the type and basic information of the current inspection and testing equipment; During the device image acquisition stage, different acquisition strategies are adopted according to different light intensities, and device images are acquired through multispectral fusion; During the edge computing preprocessing stage, combined with hardware acceleration, the device images are preprocessed to obtain preprocessed tensor data; During the lightweight model inference stage, the improved lightweight model is used to process the preprocessed tensor data to obtain component bounding boxes, component IDs, and component confidence levels; During the component ID recognition stage, according to the component bounding boxes, component IDs, and component confidence levels, the component attribute database is queried for fine-grained classification to obtain the unique component identifier; During the stage of matching inspection items and clauses, according to the unique component identifier, the TSG clauses are obtained from the knowledge graph, and the TSG clauses are converted into executable logic through the rule engine to obtain the inspection item list; During the AR prompt stage, different prompts are given to different inspection items using AR hierarchical display, and the inspection is completed to obtain the inspection results; During the original data recording stage, the inspection results are recorded using a hash chain and abnormal data is processed to obtain structured inspection records; During the blockchain evidence storage and submission stage, the inspection records are uploaded to the blockchain, and the validity of the signature is automatically verified through a smart contract. Finally, the status receipt is automatically written to the blockchain through RPA; During the automatic inspection report generation stage, the inspection records are read, the HTML intermediate file is rendered using a template engine, and then the format is converted to obtain the inspection report in the preset format.

[0007] Preferably, the device location information is obtained through the GPS module integrated in the AR glasses or the QR code scanning function; the AR glasses are equipped with a camera; the device image acquisition stage further includes the following steps: The attitude data is collected in real time using an IMU sensor, the jitter is compensated through Kalman filtering, and the exposure parameters are dynamically adjusted. The HDR mode is enabled in strong light environments, and the infrared camera is synchronously activated to assist imaging in weak light environments.

[0008] Preferably, the hardware acceleration is to call the Hexagon DSP of the Qualcomm XR2 chip to perform matrix operations; the edge computing preprocessing stage further includes the following steps: The device images are normalized, the RGB and infrared data are merged through an early feature fusion network, a 4-channel tensor is output, and motion blur compensation is performed. The jitter trajectory is estimated based on OpenCV, and an inverse filter is applied to restore the clear image.

[0009] Preferably, the improved lightweight model is the improved YOLOv8n model; the backbone network of the improved YOLOv8n model is replaced with MobileNetV3-Small, the CBAM attention module is embedded in the Neck layer, and the detection head uses dynamic convolution; when deploying the improved YOLOv8n model, TensorRT is quantized to INT8, the model volume is compressed to 12MB, and Hexagon DSP parallel computing is used to accelerate inference.

[0010] Preferably, the component ID recognition stage further includes the following steps: For the same type of components, subclass labels are output through the fully connected layer of MobileNetV3, and a confidence threshold is set, and components with a confidence lower than the threshold are rechecked.

[0011] Preferably, the AR hierarchical display includes displaying primary information and secondary information. The primary information is whether the component name and key indicators are qualified; the secondary information is to trigger a pop-up window through gestures to display the original clause and the historical data line chart; the prompt for the inspection item includes a voice prompt, and the voice prompt is dynamically broadcast by the Edge-TTS engine and the speech rate is adaptively adjusted according to the ambient noise.

[0012] Preferably, the template engine is the Jinja2 template engine, and the preset format is the PDF format; the inspection report automatic generation stage further includes the following steps: signing with the SM2 national cryptography algorithm and writing the signature data into the inspection report in the preset format.

[0013] The second aspect of the present invention provides: A special equipment inspection and detection system based on machine vision for implementing any one of the above-mentioned special equipment inspection and detection methods based on machine vision, including: An equipment type determination module for automatically identifying the type and basic information of the current inspection and detection equipment by obtaining equipment location information or inputting an equipment number and combining with the equipment database pre-stored in the cloud; An equipment image acquisition module for adopting different acquisition strategies according to different light intensities and acquiring equipment images through multi-spectral fusion; An edge computing preprocessing module for preprocessing the equipment image in combination with hardware acceleration to obtain preprocessing tensor data; A lightweight model inference module for using the improved lightweight model to process the preprocessing tensor data to obtain component bounding boxes, component IDs, and component confidences; A component ID recognition module for querying the component attribute database according to the component bounding box, component ID, and component confidence, and performing fine-grained classification to obtain the unique identifier of the component; A matching inspection item and clause module, which is used to obtain TSG clauses from a knowledge graph according to a component unique identifier, and convert the TSG clauses into executable logic through a rule engine to obtain a list of inspection items; An AR prompt module, which is used to perform different prompts for different inspection items by using AR hierarchical display, and complete the inspection to obtain an inspection result; A raw data recording module, which is used to record the inspection result using a hash chain and perform abnormal data processing to obtain a structured inspection record; A blockchain evidence storage and submission module, which is used to upload the inspection record to the blockchain, automatically verify the signature validity through a smart contract, and finally automatically write the status receipt to the blockchain through RPA; An inspection report automatic generation module, which is used to read the inspection record, render an HTML intermediate file using a template engine, and then perform format conversion to obtain an inspection report in a preset format.

[0014] The third aspect of the present invention provides: A computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned special equipment inspection and detection methods based on machine vision is implemented.

[0015] The beneficial effects of the present invention are: 1) Integrating edge computing, multimodal interaction and inspection regulation automation technologies, realizing a full-process closed loop of "acquisition - recognition - prompt - record - report", solving the problems of traditional inspection relying on manual labor, low efficiency and insufficient standardization, and still maintaining a real-time processing ability of 25FPS in an offline environment, significantly improving the accuracy and efficiency of special equipment inspection.

[0016] 2) Lightweight model + edge computing: Realize real-time detection with 95% mAP@0.5 on the AR glasses side, breaking through the traditional limitation of relying on the cloud.

[0017] 3) Programmability of inspection regulations: Convert text clauses into executable logic through NLP + rule engine, supporting dynamic updates.

[0018] 4) Multimodal closed-loop interaction: AR display, voice, and gesture cooperate to realize a full-process paperless operation of "acquisition - recognition - prompt - record - report".

[0019] 5) Industrial-level reliability: Dust-proof and waterproof design, offline operation, anti-interference algorithm, adapting to harsh environments such as elevator shafts. Description of the Drawings

[0020] Figure 1 It is a flowchart of a special equipment inspection and detection method based on machine vision. Detailed Embodiments

[0021] The technical solution of the present invention will be clearly and completely described below in conjunction with embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0022] With the development and maturity of portable and wearable devices, especially the gradual application of AR glasses, it is possible to combine machine vision, virtual (augmented) reality technology with special equipment inspection and testing, providing the possibility to establish an efficient inspection mode and inspection system based on machine vision.

[0023] Referring to Figure 1 , the first aspect of the present invention provides: A method for inspecting and testing special equipment based on machine vision, including the following steps: Equipment variety determination stage: By obtaining equipment location information or inputting an equipment number, combined with the equipment database pre-stored in the cloud, automatically identify the type and basic information of the current inspection and testing equipment; Equipment image acquisition stage: Adopt different acquisition strategies according to different light intensities, and collect equipment images through multi-spectral fusion; Edge computing preprocessing stage: Combine hardware acceleration to preprocess the equipment image to obtain preprocessed tensor data; Lightweight model inference stage: Use an improved lightweight model to process the preprocessed tensor data to obtain component bounding boxes, component IDs, and component confidence levels; Component ID identification stage: Query the component attribute database according to the component bounding box, component ID, and component confidence level, and perform fine-grained classification to obtain the unique component identifier; Matching inspection items and clauses stage: Obtain TSG clauses from the knowledge graph according to the unique component identifier, and convert the TSG clauses into executable logic through a rule engine to obtain a list of inspection items; AR prompt stage: Use AR hierarchical display to give different prompts for different inspection items, and complete the inspection to obtain the inspection result; Original data recording stage: Use a hash chain to record the inspection result and perform abnormal data processing to obtain a structured inspection record; Blockchain evidence storage and submission stage: Upload the inspection record to the blockchain, automatically verify the validity of the signature through a smart contract, and finally automatically write the status receipt into the blockchain through RPA; Inspection report automatic generation stage: Read the inspection record, use a template engine to render an HTML intermediate file, and then perform format conversion to obtain an inspection report in a preset format.

[0024] In this embodiment, blockchain evidence storage and submission are performed first, and then an inspection and testing report is automatically generated. The purpose of doing this is to ensure the consistency between the original records and data and the site. The device location information is obtained through a vision terminal. The functions of the vision terminal are, firstly, to collect videos or pictures; secondly, to interact with the operator, and the interaction methods include text, pictures, voice, etc. The vision terminal can be a wearable device, such as an AR glasses, a wearable helmet, etc.; it can also be a portable device, such as a mobile phone, a watch, a tablet computer, etc.; or even a combination of a fixed camera + a display. The embodiment of the present invention will be described with an AR glasses as the vision terminal.

[0025] Data collection and preprocessing mainly include: (1) Multi-source data collection, real-scene data: Use AR glasses to take images of components at different perspectives and illuminations (strong light / weak light / stroboscopic light) of the elevator landing door and car door units, covering key components such as the landing door, car door, automatic closing landing door device, emergency unlocking device, door lock device, etc. Synchronously record IMU sensor data (attitude, acceleration) to assist subsequent spatial positioning and image alignment. Synthetic data generation: Use Blender to build a 3D elevator model, simulate complex scenarios such as metal reflection, oil stain adhesion, and component occlusion (such as cable occlusion of the door lock), and render high-resolution images. Randomly adjust texture, illumination angle, and camera parameters through Domain Randomization to improve the generalization ability of the model. (2) Data annotation and enhancement, annotation rules: Use Label Studio to annotate the bounding box of the components and bind a unique ID (such as door_lock_001), which is associated with the provisions of TSG T7001-2023. Add fine-grained labels (such as door_lock_OTIS, door_lock_Mitsubishi) to easily confused components (such as door locks of different brands). Data enhancement is carried out by the following methods. Physical simulation enhancement: Add simulated metal reflection (OpenCV specular spot generation), motion blur (jitter during shooting when the elevator is running). Adversarial enhancement: Use GAN to generate images under extreme illuminations (such as strong light directly shining on the camera in the machine room).

[0026] Model construction and training mainly include: (1) Lightweight object detection model, baseline model selection: YOLOv8n: Balancing speed and accuracy, with a default input resolution of 640×640, suitable for edge devices. The design can also be improved through the following measures. Backbone network: Replace the original Darknet53 with MobileNetV3 to reduce the computational load (FLOPs reduced by 40%). Attention mechanism: Embed CBAM (Convolutional Block Attention Module) in the Neck layer to suppress the interference of metal reflection. Detection head optimization: Use dynamic convolution to replace fixed convolution to adapt to different-sized components (such as small-sized door lock contacts vs large-sized car guide rails). Model training: Loss function: Use Focal Loss to alleviate class imbalance (such as the number of guide rail samples is much more than that of safety clamps). Transfer learning: Pre-train based on the COCO dataset and then fine-tune with elevator component data (freeze the first 50% of the Backbone layers). Knowledge distillation: Use YOLOv8x as the teacher model to guide the training of the lightweight student model and improve the recognition accuracy of small-sample components. (2) Anti-interference design, multispectral fusion: Fusion of RGB camera and infrared imaging data, processed by an early feature fusion network for low-light scenarios. Infrared data can penetrate some oil stains to assist in identifying components with severe rust. Dynamic inference: Automatically switch the model branch according to the ambient light intensity (such as using YOLOv8n under normal light and switching to the infrared-enhanced sub-model under low light).

[0027] Edge deployment and real-time optimization mainly include: (1) Model compression and acceleration, quantization: Use TensorRT to quantize the model weights from FP32 to INT8, with the accuracy loss controlled within 2% (fine-tune through the calibration dataset). Pruning: Remove redundant convolutional kernels based on channel importance scoring (such as L1-norm), reducing the model size by 30%. Hardware acceleration: Call the Hexagon DSP of the Qualcomm XR series chips to accelerate convolutional calculations (5 times faster inference speed compared to CPU). (2) Real-time guarantee, multi-threaded pipeline: Image acquisition → preprocessing (normalization / denoising) → inference → post-processing (NMS) are executed in separate threads, with the frame rate stable above 25FPS. Adaptive resolution: When detecting concurrent multi-components (such as when the car door is fully open), temporarily reduce the input resolution to 480×480 to prioritize real-time performance.

[0028] Multimodal fusion and post-processing mainly include: (1) Spatial alignment and AR integration, SLAM positioning: Use ARCore / ARKit to construct a sparse point cloud map of the elevator shaft in real time, and map the 2D detection box to the 3D space through the PnP algorithm. IMU-assisted calibration: When the rapid movement of the camera causes image blurring, interpolate and compensate for the pose offset based on IMU data to reduce AR display jitter. (2) False detection filtering and logical verification, geometric constraints: Set rules according to the prior knowledge of the elevator structure (such as "door locks cannot appear in the machine room") to filter abnormal detection results. Temporal consistency: Track the position of components based on Kalman filtering. If the detection results are inconsistent for 5 consecutive frames, trigger a re-detection.

[0029] The inspection and detection of on-site adaptability mainly include: (1) Robustness in extreme environments, anti-shake algorithm: Adopt electronic image stabilization (EIS) and multi-frame fusion technology to eliminate the influence of head or hand jitter during manual inspection.

[0030] Anti-overexposure processing: Perform local brightness suppression (CLAHE algorithm) on high-light areas (such as the reflection of stainless steel guide rails) to avoid detection box drift. (2) Maintainability design, incremental learning: Support online model update (such as adding a new door lock model of a certain brand), only need to upload a small number of samples to the edge device for fine-tuning, without retraining the entire model. Fault self-diagnosis: When no valid results are detected for 10 consecutive frames, automatically switch to the backup model or prompt the user to clean the lens.

[0031] Regulatory structuring and knowledge base construction mainly include: (1) Semantic parsing of regulatory texts, technology selection: Rule engine + NLP: Combine regular expressions to extract key fields of clauses (such as "meshing depth ≥ 7mm"), and supplement with the BERT model for context semantic understanding (distinguish between mandatory / suggestive requirements such as "shall" and "should"). Entity relationship extraction: Use the BiLSTM-CRF model to identify inspection objects (such as "door locks"), inspection parameters (such as "meshing depth"), thresholds (such as "7mm"), and detection methods (such as "measured with a vernier caliper") in the clauses. (2) Knowledge graph construction, graph structure design: Nodes: Component types (door locks), inspection clauses, parameters, detection tools. Edges: Relationship types (such as "needs to be detected", "depends on tool"). Tools: Use Neo4j or AWS Neptune to construct the graph, supporting multi-hop queries (such as "door lock → TSGT7001-2023_3.8.1 → meshing depth → vernier caliper").

[0032] Dynamic clause matching and logic execution include: (1) Establish a component-clause association mapping table, and manually label the relationship between components and clauses. Incremental update: When a new clause is added, the associated components are automatically recommended through keyword matching (such as "door lock"). (2) Programmable inspection logic, rule engine integration: Use a Drools or Python-based rule engine to convert clauses into executable logic. Dynamic loading: According to the currently recognized component_id, load the corresponding rule set from the knowledge base into memory for execution.

[0033] Multimodal interaction and status management mainly include: (1) AR visualization and voice interaction, and information layering is rendered according to the following method. First-level prompt: The core inspection items (such as "Engagement depth: to be measured") are floatingly displayed beside the component, and the status is color-coded (gray = not inspected, green = qualified, red = unqualified). Second-level details: After the user clicks with a gesture, the full text of the clause and the historical inspection records (such as "Detection value on January 5, 2023: 7.2mm") are expanded. The voice engine includes the following: Text-to-Speech (TTS): Use Edge-TTS or VITS to generate natural voice prompts (such as "Please use a vernier caliper to measure the engagement depth of the door lock"). Automatic Speech Recognition (ASR): Integrate Porcupine wake word + Whisper speech-to-text, and support command control (such as "Show all clauses"). (2) Inspection process recording, state machine design: Persistent storage: Use SQLite to record the inspection time, operator, and original data (such as measurement photos, voice remarks), and support exporting PDF reports (generated through the Jinja2 template engine).

[0034] Offline support and version management mainly include: (1) Localized deployment, data synchronization: During the initial deployment, pre-install the regulatory knowledge base, voice model, etc. into the storage of the AR glasses. Support incremental updates via USB or Wi-Fi (such as pushing a new version of the knowledge base after a regulatory amendment). Caching strategy: The recently accessed clause data is retained in memory to reduce database query latency. (2) Version control, Git-style management: Generate a version hash (such as SHA-256) for each modification to the knowledge base to ensure that the specified version (such as TSGT7001-2023_v1.2.3) is used for on-site inspections. Support version rollback to prevent the system from becoming unavailable due to update errors. Template engine: Adopt the Jinja2 template, support dynamic rendering of tables with loop statements, and the pagination logic is controlled by CSS page-break. Font embedding: Pre-install the "FangSong_GB2312" font file to ensure normal Chinese display in the offline environment.

[0035] In some embodiments, the device location information is obtained through the GPS module integrated in the AR glasses or the QR code scanning function; the AR glasses are equipped with a camera; the device image acquisition stage further includes the following steps: Collect attitude data in real time using an IMU sensor, compensate for jitter through Kalman filtering, and dynamically adjust the exposure parameters. Enable the HDR mode in strong light environments and synchronously activate the infrared camera to assist imaging in low light environments.

[0036] In this embodiment, the device location information is obtained by integrating a GPS module or QR code scanning function into the AR glasses, or the device number is input by means of voice, etc. Combining with the device database pre-stored in the cloud, the type and basic information of the current inspection and testing device are automatically identified. For example, after scanning the QR code of the elevator use sign and obtaining the device number, query the SQLite local cache database to match the device variety (such as "traction drive passenger elevator") and other basic information, including the name of the using unit, the contact phone number of the contact person of the using unit, the name of the maintenance unit, the contact phone number of the contact person of the maintenance unit, the manufacturing unit, the manufacturing date, the elevator floor station, the running speed, etc. In an offline environment, load information based on the most recent access records. Support offline-online hybrid mode recognition, manage the local cache through the LRU algorithm, and ensure that more than 95% of the scenarios do not require network interaction.

[0037] The AR glasses are equipped with a 1080P camera (frame rate 30fps) to capture component images, and synchronously activate the infrared camera (850nm wavelength) to assist in low light scenarios (such as the elevator pit). The IMU sensor (accelerometer + gyroscope) collects attitude data in real time (in JSON format) and compensates for jitter through Kalman filtering. Dynamically adjust the exposure parameters: enable the HDR mode (3-frame synthesis) in strong light environments, and prioritize infrared imaging in low light. The ability of infrared to penetrate oil stains improves the visibility of components.

[0038] In some embodiments, the hardware acceleration is to call the Hexagon DSP of the Qualcomm XR2 chip to perform matrix operations; the edge computing preprocessing stage further includes the following steps: Normalize the device image, merge RGB and infrared data through an early feature fusion network, output a 4-channel tensor, and perform motion blur compensation. Estimate the jitter trajectory based on OpenCV and apply an inverse filter to restore a clear image.

[0039] In this embodiment, the image is normalized to a resolution of 640×640, and the pixel values are normalized (0-1). Multispectral fusion: Merge RGB and infrared data through an early feature fusion network to output a 4-channel tensor (R, G, B, IR). Motion blur compensation: Estimate the jitter trajectory based on OpenCV and apply an inverse filter to restore a clear image. Hardware acceleration: Call the Hexagon DSP of the Qualcomm XR2 chip to perform matrix operations (such as FFT denoising), with a delay ≤ 10ms. Under the acceleration of the Hexagon DSP, the preprocessing speed is increased by 4 times.

[0040] In some embodiments, the improved lightweight model is the improved YOLOv8n model; the backbone network of the improved YOLOv8n model is replaced with MobileNetV3-Small, the CBAM attention module is embedded in the Neck layer, and the detection head uses dynamic convolution; when deploying the improved YOLOv8n model, TensorRT is quantized to INT8, the model volume is compressed to 12MB, and Hexagon DSP parallel computing is used to accelerate inference.

[0041] In this embodiment, the YOLOv8n model is improved as follows: the backbone network is replaced with MobileNetV3-Small (FLOPs = 2.8B), the CBAM attention module (channel + spatial attention) is embedded in the Neck layer, and the detection head uses dynamic convolution (switching between 3×3 / 5×5 kernels according to the target size). Deployment optimization: TensorRT is quantized to INT8 (1000 calibration dataset images), and the model volume is compressed to 12MB. Inference acceleration: Hexagon DSP parallel computing, single-frame inference time ≤ 30ms. The dynamic convolution adapts to multi-size components, and the detection accuracy of small targets is increased by 15%.

[0042] In some embodiments, the component ID recognition stage further includes the following steps: For components of the same type, subclass labels are output through the fully connected layer of MobileNetV3, and a confidence threshold is set, and components with a confidence lower than the threshold are rechecked.

[0043] In this embodiment, the component attribute database (SQLite) stores unique identifiers (such as door_lock_OTIS_X1), and the fields include brand, model, and historical inspection records. Fine-grained classification: for components of the same type (such as door locks of different brands), subclass labels are output through the last fully connected layer of MobileNetV3. The confidence threshold is set to 0.8, and values lower than this trigger manual review (red frame prompt in the AR interface). The fine-grained classification error rate ≤ 2%, and incremental learning is supported for online updating of subclasses.

[0044] The Neo4j knowledge graph stores TSG terms. The nodes include component types, inspection parameters, and thresholds, and the edges define the association relationships (such as door_lock → engagement depth ≥ 7mm). The knowledge graph supports multi-hop queries (such as "door lock → engagement depth → vernier caliper"). The Drools rule engine converts the terms into executable logic:

[0045] In some embodiments, the AR hierarchical display includes displaying primary information and secondary information. The primary information is the part name and whether the key indicators are qualified. The secondary information is to trigger a pop-up window through gestures to display the original clause and a line chart of historical data. The prompt for the inspection item includes a voice prompt, which is dynamically broadcast using the Edge-TTS engine and adaptively adjusts the speech rate according to the ambient noise.

[0046] In this embodiment, the AR hierarchical display: The primary information is the part name + key indicators (green qualified / red unqualified), with a transparency of 70% and a font size of 24pt. The secondary information triggers a pop-up window through gestures (MediaPipe recognizes the pinch action) to display the original clause and a line chart of historical data. Voice prompt: Dynamically broadcast by the Edge-TTS engine (such as "Please measure the meshing depth"), and the speech rate is adaptively adjusted according to the ambient noise (50 - 70dB). The gesture interaction delay ≤ 100ms, and the voice command recognition accuracy rate ≥ 95%.

[0047] The SQLite database table structure is shown in Table 1. Table 1

[0048] For the processing of abnormal data: When the confidence level < 0.8, it is marked as the status to be rechecked, and the original image is uploaded back to the cloud. The hash chain technology is used to ensure data integrity and prevent tampering.

[0049] In some embodiments, the template engine is the Jinja2 template engine, and the preset format is the PDF format. The automatic generation stage of the inspection report further includes the following steps: Sign using the SM2 national cryptography algorithm and write the signature data into the inspection report in the preset format.

[0050] In this embodiment, use the Jinja2 template engine to render HTML, including a dynamic table (generating a list of detection items through loop statements) and a Matplotlib chart (historical trend comparison). PDF conversion: The WeasyPrint engine supports CSS pagination (@mediaprint) and embeds the FangSong_GB2312 font file. Digital signature: Sign using the SM2 national cryptography algorithm and write the signature data into the PDF metadata. Generate 50-page reports within 10 seconds in an offline environment, meeting the TSG format specification.

[0051] Hyperledger Fabric chain code design: ① Data on the chain: Report summary (MD5), signature, timestamp. ② Smart contract: Automatically verify the validity of the signature and trigger email notifications. ③ RPA automation: The UiPath script logs in to the regulatory platform (automatically fills in forms and uploads PDFs), and the status receipt is written into the blockchain. RPA + blockchain realizes zero manual intervention submission with a 100% compliance rate.

[0052] The second aspect of the present invention provides: A special equipment inspection and testing system based on machine vision, which is used to implement any of the above-mentioned special equipment inspection and testing methods based on machine vision, and includes: An equipment type determination module, which is used to automatically identify the type and basic information of the current inspection and testing equipment by obtaining equipment location information or inputting an equipment number and combining with the equipment database pre-stored in the cloud; An equipment image acquisition module, which is used to adopt different acquisition strategies according to different light intensities and acquire equipment images through multi-spectral fusion; An edge computing preprocessing module, which is used to preprocess the equipment image in combination with hardware acceleration to obtain preprocessed tensor data; A lightweight model inference module, which is used to process the preprocessed tensor data using an improved lightweight model to obtain part bounding boxes, part IDs, and part confidence levels; A part ID recognition module, which is used to query the part attribute database according to the part bounding box, part ID, and part confidence level, and perform fine-grained classification to obtain a unique part identifier; A matching inspection items and clauses module, which is used to obtain TSG clauses from the knowledge graph according to the unique part identifier, and convert the TSG clauses into executable logic through a rule engine to obtain a list of inspection items; An AR prompt module, which is used to perform different prompts for different inspection items by using AR hierarchical display, and complete the inspection to obtain inspection results; A raw data recording module, which is used to record the inspection results using a hash chain and perform abnormal data processing to obtain a structured inspection record; A blockchain evidence storage and submission module, which is used to upload the inspection record to the blockchain, automatically verify the validity of the signature through a smart contract, and finally automatically write the status receipt to the blockchain through RPA; An inspection report automatic generation module, which is used to read the inspection record, render an HTML intermediate file using a template engine, and then perform format conversion to obtain an inspection report in a preset format.

[0053] The third aspect of the present invention provides: A computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, any of the above-mentioned special equipment inspection and testing methods based on machine vision is implemented.

[0054] The above are only the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be changed within the scope of the concept described herein through the above teachings or the technology or knowledge in the relevant field. Any changes and modifications made by those skilled in the art without departing from the spirit and scope of the present invention shall fall within the protection scope of the appended claims of the present invention.

Claims

1. A special equipment inspection and testing method based on machine vision, characterized in that: It includes the following steps: In the device variety determination stage, by obtaining the device location information or inputting the device number, and combining with the device database pre-stored in the cloud, the type and basic information of the current inspection and testing device are automatically identified; In the device image acquisition stage, different acquisition strategies are adopted according to different light intensities, and the device image is acquired through multi-spectral fusion; In the edge computing preprocessing stage, combined with hardware acceleration, the device image is preprocessed to obtain preprocessed tensor data; In the lightweight model inference stage, the improved lightweight model is used to process the preprocessed tensor data to obtain the part bounding box, part ID, and part confidence; In the part ID recognition stage, according to the part bounding box, part ID, and part confidence, the part attribute database is queried, and fine-grained classification is performed to obtain the unique part identifier; In the stage of matching inspection items and clauses, the TSG clauses are obtained from the knowledge graph according to the unique part identifier, and the TSG clauses are transformed into executable logic through the rule engine to obtain the inspection item list; In the AR prompt stage, different prompts are given to different inspection items by using AR hierarchical display, and the inspection is completed to obtain the inspection result; In the original data recording stage, the inspection result is recorded using a hash chain and abnormal data is processed to obtain a structured inspection record; In the blockchain evidence storage and submission stage, the inspection record is uploaded to the chain, the validity of the signature is automatically verified through a smart contract, and finally the status receipt is automatically written into the blockchain through RPA; In the automatic inspection report generation stage, the inspection record is read, the HTML intermediate file is rendered using a template engine, and then format conversion is performed to obtain the inspection report in the preset format.

2. The method for inspection and testing of special equipment based on machine vision according to claim 1, characterized in that: The device location information is obtained through the GPS module integrated in the AR glasses or the QR code scanning function; the AR glasses are equipped with a camera; the device image acquisition stage further includes the following steps: The attitude data is collected in real time using an IMU sensor, the jitter is compensated through Kalman filtering, and the exposure parameters are dynamically adjusted. The HDR mode is enabled in strong light environments, and the infrared camera is synchronously activated to assist imaging in low light environments.

3. The special equipment inspection and testing method based on machine vision according to claim 1, characterized in that: The hardware acceleration is to call the Hexagon DSP of the Qualcomm XR2 chip to perform matrix operations; the edge computing preprocessing stage further includes the following steps: The device image is normalized, the RGB and infrared data are merged through an early feature fusion network, a 4-channel tensor is output, and motion blur compensation is performed. The jitter trajectory is estimated based on OpenCV, and the inverse filter is applied to restore the clear image.

4. The method for inspection and testing of special equipment based on machine vision according to claim 1, wherein: The improved lightweight model is the improved YOLOv8n model; the backbone network of the improved YOLOv8n model is replaced with MobileNetV3-Small, the CBAM attention module is embedded in the Neck layer, and the detection head uses dynamic convolution. When deploying the improved YOLOv8n model, TensorRT is quantized to INT8, the model volume is compressed to 12MB, and the Hexagon DSP is used for parallel computing to accelerate inference.

5. The method for inspection and testing of special equipment based on machine vision according to claim 1, wherein: The part ID recognition stage further includes the following steps: For similar components, subclass labels are output through the fully connected layer of MobileNetV3, and a confidence threshold is set to review components with a confidence lower than the threshold.

6. The method for inspection and testing of special equipment based on machine vision according to claim 1, characterized in that: The AR hierarchical display includes displaying first-level information and second-level information. The first-level information is whether the component name and key indicators are qualified; the second-level information is to trigger a pop-up window through gestures to display the original clause and a line chart of historical data; the prompts for inspection items include voice prompts, which are dynamically broadcast using the Edge-TTS engine and the speech rate is adaptively adjusted according to environmental noise.

7. The method for inspection and testing of special equipment based on machine vision according to claim 1, wherein: The template engine is the Jinja2 template engine, and the preset format is the PDF format; the automatic generation stage of the inspection report further includes the following steps: signing using the SM2 national cryptography algorithm and writing the signature data into the inspection report in the preset format.

8. An inspection and testing system for special equipment based on machine vision, characterized in that: Used to implement the machine vision-based special equipment inspection and testing method according to any one of claims 1-7, including: The equipment variety determination module is used to automatically identify the type and basic information of the current inspection and testing equipment by obtaining equipment location information or inputting the equipment number and combining with the equipment database pre-stored in the cloud. The equipment image acquisition module is used to adopt different acquisition strategies according to different light intensities and acquire equipment images through multispectral fusion. The edge computing preprocessing module is used to preprocess the equipment image in combination with hardware acceleration to obtain preprocessed tensor data. The lightweight model inference module is used to process the preprocessed tensor data using an improved lightweight model to obtain component bounding boxes, component IDs, and component confidences. The component ID recognition module is used to query the component attribute database according to the component bounding box, component ID, and component confidence, and perform fine-grained classification to obtain the unique identifier of the component. The module for matching inspection items and clauses is used to obtain TSG clauses from the knowledge graph according to the unique identifier of the component, and convert the TSG clauses into executable logic through a rule engine to obtain a list of inspection items. The AR prompt module is used to perform different prompts for different inspection items using AR hierarchical display to complete the inspection and obtain the inspection result. The original data recording module is used to record the inspection result using a hash chain and perform abnormal data processing to obtain a structured inspection record. The blockchain evidence storage and submission module is used to upload the inspection record to the blockchain, automatically verify the validity of the signature through a smart contract, and finally automatically write the status receipt into the blockchain through RPA. The inspection report automatic generation module is used to read the inspection record, render an HTML intermediate file using a template engine, and then perform format conversion to obtain an inspection report in the preset format.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, the machine vision-based special equipment inspection and testing method according to any one of claims 1-7 is implemented.

Images