Electric power logistics warehouse classification warehousing system and method based on multi-modal image identification

Abstract

The invention relates to the technical field of intelligent warehousing and logistics, in particular to an electric power logistics warehouse classified warehousing system and method based on multi-modal image recognition. Comprising a visible light camera, an infrared thermal imager and a 3D structured light scanner of a multi-modal acquisition module, which are used for acquiring images of power equipment; collected electrical equipment image data are stored in a database, and the electrical equipment image data stored in the database are transmitted through a central control terminal by adopting an OPCUA protocol and are processed through a main control server by adopting a YOLOV7 target detection algorithm. And controlling an autonomous navigation AGV trolley and a six-degree-of-freedom cooperative mechanical arm of the execution unit module to carry out carrying. The electric power logistics warehouse classification warehousing system and method based on multi-modal image recognition are low in manual dependence degree, high in classification precision, high in environmental adaptability and efficient in searching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent warehousing and logistics, and discloses a power logistics warehouse classification and warehousing system and method based on multi-modal image recognition. BACKGROUND

[0002] The power industry logistics warehouse management has long been plagued by high dependence on manual work, insufficient classification accuracy, poor environmental adaptability, and fragmented processes. For example, a provincial warehouse of State Grid has more than 5000 types of power materials, among which key equipment such as insulators, transformers, and line fittings have complex specifications (e.g., insulators are divided into 10kV, 35kV, and 110kV according to voltage level, and are divided into porcelain, glass, and composite according to material) and similar shapes (e.g., the aperture difference between U-shaped rings and extension fittings is only 1-2mm), making traditional manual sorting inefficient and prone to errors.

[0003] According to the "Power Equipment Warehouse Management Technology White Paper (2022)", the average time spent on manual sorting of insulators is 8 minutes per piece, with an error rate of about 5%. However, the automation solution using traditional image recognition technology (such as SIFT feature matching) still has a misjudgment rate of up to 15% due to the inability to distinguish material differences (such as the similarity of porcelain and composite insulator surface texture). In addition, warehouse automation technologies such as RFID tag management require pre-pasting electronic tags, increasing the cost of each piece by 3-5 yuan, and cannot detect physical conditions (such as internal cracks in insulators). The industry urgently needs a solution that combines high-precision recognition, complex environment adaptation, and full-process automation. In terms of literature support, IEEE Transactions on Power Systems (2021) pointed out that existing deep learning-based power equipment recognition models (such as ResNet-50) have a classification accuracy of less than 90% under a single perspective, and do not consider multi-modal data fusion. The "Review of Intelligent Warehouse Robot Navigation Technology (2023)" mentioned that traditional two-dimensional code navigation AGVs have poor path flexibility and cannot adapt to dynamic warehouse environments. These studies provide a theoretical basis for the multi-modal fusion and adaptive navigation design of the present application.

[0004] Based on the above description, the power logistics warehouse management process often encounters the following problems: first, the problem of high-precision classification of power materials, especially the accurate differentiation of materials with similar forms and small size differences (such as different material insulators and the same series of hardware); second, the problem of recognition stability in complex storage environment (such as uneven lighting, stacking shielding, and multi-angle placement); third, the problem of collaborative control of image recognition and storage hardware (AGV, mechanical arm), realizing the whole process closed-loop management from recognition to warehousing / warehousing. For example, in a certain city power warehouse, the porcelain insulator and the composite insulator have similar surface texture, and the manual sorting error rate is 8%, while the traditional image recognition algorithm (such as HOG+SVM) ignores the internal structure difference (the composite insulator core rod fracture can be detected by infrared thermal imaging), and the misjudgment rate is still 12%; In the warehouse of a certain converter station of State Grid, the light intensity in the warehouse fluctuates in the range of 200-20000 Lux, resulting in more than 30% failure rate of traditional algorithm in shadow area insulator recognition; The stacked line hardware (such as shock absorber) is blocked by each other, and the manual unstacking inspection efficiency is less than 10 pieces / hour; A certain enterprise uses independent system to manage warehousing, warehousing and inventory, resulting in manual checking of position information when transformer is out of warehouse, with an average time consumption of 20 minutes, and the error rate caused by asynchronous information is about 3%. Based on the above problems, the present application provides a new scheme for power logistics warehouse classification and storage based on multi-modal image recognition, which solves the shortcomings and pain points of the prior art. SUMMARY

[0005] To solve the problems of the prior art, the following technical solutions are provided: The power logistics warehouse classification and storage system based on multi-modal image recognition comprises a multi-modal acquisition module, a control module and an execution unit module; the multi-modal acquisition module comprises a visible light camera, an infrared thermal imager and a 3D structured light scanner; the control module comprises a main control, a central control terminal and a database; the execution unit module comprises an autonomous navigation AGV trolley and a six-degree-of-freedom collaborative mechanical arm. The visible light camera, infrared thermal imager and 3D structured light scanner of the multi-modal acquisition module are used for power equipment image acquisition; the collected power equipment image data is stored in the database, the power equipment image data stored in the database is transmitted through the central control terminal using OPCUA protocol, and the data is processed through the main control server using YOLOV7 target detection algorithm, and the autonomous navigation AGV trolley and the six-degree-of-freedom collaborative mechanical arm of the execution unit module are controlled.

[0006] The visible light camera is an industrial camera with more than 20 million pixels, and the industrial camera is equipped with a fill light to collect power equipment surface images in an illumination range of 3000-6000Lux; the infrared thermal imager has a resolution of more than 640*512 thermal imagers, and detects the surface temperature of the material; the 3D structured light scanner obtains three-dimensional point cloud data of the material through laser projection and a camera; the autonomous navigation AGV car is configured with a DW3000 chip set, and the path planning adopts the ORB-SLAM3 open source algorithm of the SLAM map construction technology.

[0007] The use method of the power logistics warehouse classification storage system based on multi-modal image recognition includes the following steps: S1, multi-modal data acquisition Surface features are recognized by a visible light camera, temperature anomalies are detected by an infrared thermal imager, and three-dimensional dimensions are obtained by a 3D structured light scanner; multi-view point cloud registration is performed using an ICP algorithm; S2, target recognition and classification The multi-modal data acquisition data is output by a YOLOV7 model using a CBAM attention mechanism through a host server, and the material category and coordinates are recorded in a database; S3, storage and transportation execution The power equipment inventory data is read from the database, and transportation is performed by an autonomous navigation AGV car and a six-degree-of-freedom collaborative manipulator, and the inventory status is updated after transportation.

[0008] The infrared thermal imager and the 3D structured light scanner in S1 are calibrated by AprilTag markers for three-dimensional dimensions; when the infrared thermal imager detects a temperature anomaly, the 3D structured light scanner is triggered to perform high-precision scanning on the target area.

[0009] In the ORB-SLAM3 open source algorithm of the autonomous navigation AGV car in S3, a comprehensive cost function is inserted: J=0.6L+0.3Nα+0.1k(d+1 / t), Where L is the path length in meters; N is the number of turns; the turning energy consumption penalty coefficient α=2 meters / turn; k=0.85 is the calibration constant; d is the minimum safe distance from the obstacle in meters; t is the obstacle avoidance response time in seconds; 0.6, 0.3, and 0.1 are weighting coefficients; the formula is substituted with numerical values, not units.

[0010] The infrared thermal imager in S1 detects the temperature distribution of the material surface, and when a local temperature difference of more than 2°C is detected, or when a local temperature difference of more than 3 times the sensitivity of the sensor is detected, an alarm is triggered; the 3D structured light scanner measures the aperture size of the fitting, with a precision requirement of ±0.1mm, to distinguish the slight aperture difference between the U-shaped ring and the extended fitting.

[0011] The S2 adopts room temperature 20 DEG C ± 0.5 DEG C to collect data to establish a standard temperature source to establish temperature-deformation mapping relationship, and the sensitivity of change temperature ΔT / change size ΔD is less than or equal to 0.5 DEG C / 0.01 mm; the target area is scanned in real time at a frame rate of 30 Hz to perform dynamic detection, the calibration period in the dynamic detection stage is 30 seconds, and the sensor sensitivity is less than or equal to 0.3 DEG C, that is, the temperature measurement error of the full field of view is less than or equal to 0.3 DEG C.

[0012] The dynamic detection stage can detect a region of at least 20x20 pixels of the visible light camera; when the temperature difference of a local area is greater than or equal to 2 DEG C for 3 consecutive frames, and when the temperature difference of a local area exceeds 3 times the sensor sensitivity for 3 consecutive frames, an alarm is triggered.

[0013] In the S1, the AprilTag positioning mark is used to establish collaborative detection, and the mark is arranged at a density of 1 / 0.5 m2; when the infrared thermal imager detects temperature abnormalities, the 3D structured light scanner is preferentially triggered to perform high-precision three-dimensional scanning on the target area.

[0014] The application is significantly better than the prior art in classification accuracy, efficiency, environmental adaptability and safety. The measured data shows that the classification accuracy of insulators, fittings and other materials is greater than or equal to 89.5%, which is more than 10% higher than the traditional method of 65%-79%; the warehouse entry efficiency is improved by 3 times, and the single piece processing time is reduced from 8 minutes to 2 minutes; the AGV dynamic obstacle avoidance reduces the collision frequency by 95%, and supports stable operation in the environment of-20 DEG C to 50 DEG C. Taking a municipal power logistics center as an example, after using the system, the annual labor cost is reduced by 2 million yuan, the material mismatch rate is reduced from 5% to 0.3%, and the defect detection rate is increased from 80% to 96.8%. In addition, the multi-modal acquisition module, the control module and the execution unit module realize database label-free navigation, so that the warehouse reconstruction cost is reduced by 60%, the force feedback mechanical arm realizes safe grabbing of non-standard materials (such as special-shaped fittings), and the universality is improved by 90%. DETAILED DESCRIPTION

[0015] The power logistics warehouse classification and storage system based on multi-modal image recognition comprises a visible light camera, an infrared thermal imager, a 3D structured light scanner, a main control server, a self-navigation AGV trolley, a six-degree-of-freedom collaborative mechanical arm and a central control terminal; in combination with the numbering in this paper, the visible light camera is C1, the infrared thermal imager is C2, the 3D structured light scanner is C3, the main control server is S1, the self-navigation AGV trolley is AGV1, the six-axis mechanical arm is R1, and the central control terminal is CT1. The visible light camera adopts a 20 million pixel industrial camera, equipped with a ring light, and can collect high-definition images of the material surface in the 3000-6000 Lux illumination range, for identifying visible features such as the number of insulator sheds and the surface inscription of fittings; the infrared thermal imaging uses a thermal imager with a resolution of 640x512, detects the temperature distribution of the material surface, and when a local temperature difference of more than 2°C is detected (such as temperature anomalies caused by composite insulator core rod fracture), when the local temperature difference is more than 3 times the sensitivity of the sensor (i.e. 0.3°C x 3 = 0.9°C); the 3D structured light scanner obtains three-dimensional point cloud data of the material through laser projection and binocular cameras, measures the aperture size of the fittings, and the accuracy can reach ±0.1 mm, to distinguish the slight aperture difference between U-shaped rings and extended fittings; Further, the infrared thermal imager realizes material surface temperature distribution detection based on a 640x512 resolution uncooled focal plane detector (pixel size 14pm). When the system starts, it needs to perform blackbody radiation calibration, a 20°C±0.5°C standard temperature source is used to establish a temperature-gray scale mapping relationship, the calibration period is controlled within 30 seconds, and the temperature measurement error of the full field of view is ensured to be ≤0.3°C. In the dynamic detection stage, the target area is scanned in real time at a frame rate of 30Hz, a detection area ROI of at least 20x20 pixels (corresponding to an actual area of 5cm²) is delineated through visible light fusion technology, and a 5-frame sliding window moving average algorithm is used to suppress environmental noise. When the local area has a temperature difference of ≥2°C for 3 consecutive frames (such as abnormal temperature rise caused by composite insulator core rod fracture), the system triggers a multi-level alarm mechanism, synchronously records the temperature gradient distribution, outputs an alarm signal through the Modbus / TCP protocol, and the response delay is ≤200ms. The detector needs to meet the noise equivalent temperature difference NETD≤40mK@30°C, the working waveband covers 7.5-13.5pm, and multiple spectral filter wheels (3 wavebands switchable) are configured for different materials to enhance the differentiation ability of metals and composites. The switchable wavebands of the multi-spectral filter wheel are (7.5-8.5pm), (9-10pm), and (12-13pm), suitable for identifying different materials. In this paper, multiple spectral technology is used for better identification of different materials. The specific identification of different materials in the above technology is not described again.

[0016] Further, the 3D structured light scanner projects 1200 line pairs of coded fringes with 450 nm blue laser, and synchronously collects the deformed fringes with a binocular CMOS camera (resolution 2464x2056, global shutter, exposure time≤1 ms) to realize sub-millimeter three-dimensional reconstruction. In the system calibration stage, a NIST certified standard sphere is used for binocular parameter calibration to ensure a repeatability error≤15 μm. At the same time, γ correction (coefficient 0.95-1.05) is performed on the fringe projection system to eliminate non-linear response. The PMVS dense reconstruction algorithm is used for three-dimensional point cloud generation, with a point cloud density≥40 points / mm². Based on the RANSAC cylindrical fitting algorithm, aperture characteristics are extracted, and the measurement accuracy is improved to±0.1 mm by using cubic spline curve interpolation (step size 0.01 mm). For the small aperture difference between the U-shaped ring and the extended fitting, a built-in fitting feature database (containing 200+ standard part parameters) is used to realize a classification accuracy≥99.7% through the Mahalanobis distance classifier. The optical system is configured with a 3B class laser (output power 5 mW), with a depth of field range of 150-500 mm adjustable, supporting Φ8-Φ50 mm aperture detection, with a single measurement time≤3 seconds and a repeatability accuracy of±0.05 mm. The measurement results are reported in polar coordinate size output (θ=1° interval), and the quality judgment conclusion is automatically generated by comparing the preset tolerance threshold.

[0017] Further, the cooperative detection mechanism realizes the spatio-temporal registration of C2 and C3 through AprilTag positioning markers (layout density 1 / 0.5 m²), with a coordinate system conversion residual≤0.3 mm. When C2 detects temperature abnormalities, C3 is triggered to perform high-precision three-dimensional scanning of the target area, and the thermal coupling effect is analyzed through the temperature-deformation correlation (ΔT / ΔD sensitivity 0.5℃ / 0.01 mm). The system supports dynamic parameter optimization: the thermal imager gain is automatically adjusted according to the material emissivity (ε=0.85-0.95), and the laser power is adjusted based on the point cloud curvature feedback (step size 0.1 mW). This scheme maintains the 2℃ temperature difference detection sensitivity of C2 system, while improving the aperture differentiation ability of C3 to the Φ0.05 mm level, meeting the requirements of DL / T 1242-2023 standard for power fitting detection, and realizing the full-process automation of defect positioning, size measurement and classification judgment.

[0018] Further, the autonomous navigation AGV car realizes precise motion control through the fusion of ultra-wideband (UWB) ultra-wideband positioning and visual SLAM navigation: the UWB base station is deployed at the four corners of the warehouse, and the AGV1 coordinates are solved in real time based on the TOA (time of arrival) algorithm, with a positioning accuracy of ±5 cm (the deviation between the actual trajectory and the preset path is ≤3.8 cm); at the same time, the RGB-D camera (component number C4, resolution 1920x1080, depth accuracy ±2mm@2m) carried by the AGV1 constructs a three-dimensional semantic map of the warehouse through the ORB-SLAM3 algorithm, dynamically identifies temporary obstacles (such as goods boxes with a height >30 cm) and plans an obstacle avoidance path (the response time of re-planning is ≤0.5s). The navigation system is deeply coupled with the YOLOv7 target detection algorithm: after the host server (S1) identifies the category and location of devices such as insulators and transformers (such as "porcelain insulator-110kV@(x=3.2m, y=5.7m)"), the AGV1 subscribes to the target coordinates through the ROS topic, generates an optimal path combining the point cloud data and the semantic map, for example, after identifying the location of the transformer oil pillow, automatically adjusts the driving speed to 0.3m / s to prevent vibration impact.

[0019] The autonomous navigation AGV1 uses a hybrid navigation system that combines an industrial-grade UWB positioning module (DW3000 chipset) and a custom RGB-D camera (model C4, resolution 1920x1080). Four UWB anchor points are placed within a 30x40 meter warehouse. Based on the TOA algorithm, the coordinates are calculated, and the IMU data is fused through an extended Kalman filter. The actual positioning accuracy is ±3.8 centimeters (90% confidence interval). The ORB-SLAM3 algorithm processes the camera's depth information stream, generating 30 frames of three-dimensional grid maps with semantic labels per second. For example, when a wooden crate with dimensions 0.6x0.8x1.2 meters (lengthxwidthxheight) is detected, the system automatically identifies it as a temporary obstacle and marks it as an L2 risk level in the map. Within a 0.6-meter radius centered on the object, obstacle avoidance path optimization is activated. The optimized cost function is F(n) = g(n) + α×T(n) + β×O(n). g(n): the actual movement cost from the starting point to node n (including terrain friction coefficient correction). T(n) = N_T×P_T: turning penalty term (N_T is the number of turns, P_T=0.8 is the equivalent path growth coefficient). O(n) = e^(-d / s): obstacle risk term (d is the distance to the nearest obstacle, s=0.6 meters is the safety radius). The turning penalty coefficient P_T=0.8 is determined through energy consumption experiments: in a 10-meter straight-line travel, the energy consumption is 215Wh, and the energy consumption of the same distance path containing two 90° turns increases to 238Wh. Calculating the equivalent energy consumption of each turn as an additional 0.8 meters of travel (ΔE=(238-215) / 215x10≈0.8). This parameter is verified through 500 experimental samples using a Keysight 34461A digital multimeter, ensuring that the penalty coefficient has an engineering physical meaning. In a task of transporting porcelain insulators to the coordinates (3.2m, 5.7m), the YOLOv7 algorithm detects a temporarily placed epoxy resin crate (height 95 centimeters, exceeding the 30-centimeter threshold) in the gap between the shelves: Traditional path algorithm path: the planned path length is 8.4 meters, containing 2 turns (T=2x0.8=1.6); the distance to the nearest obstacle is 0.3 meters (O=e^(-0.3 / 0.6)=0.606); the comprehensive cost F=8.4+1.6+0.606x10(β=10)=18.66.

[0020] Optimized path of the present scheme: a detour path length of 9.1 meters is generated (an increase of 8.3%), with 0 turns; the safety distance is improved to 0.8 meters (O=e^(-0.8 / 0.6)=0.264); the comprehensive cost F=9.1+0+0.264x10=11.74 (a decrease of 37%).

[0021] The path automatically adjusts the travel speed to 0.3 m / s to reduce the impact of vibration, and sends speed instructions (data format: {linear.x: 0.3, angular.z: 0}) to the motor controller through the / cmd_vel topic of ROS. Actual operation tests show that AGV1 completes transportation with a maximum path offset of 2.1 cm (an average of 1.3 cm), while avoiding carbon brush wear caused by turning.

[0022] Further, the work flow of the six-axis robot arm forms a closed-loop control with the target detection results: the material type identified by YOLOv7 (such as "composite insulator - 220kV") is transmitted to the robot arm controller through TCP / IP protocol, triggering the preset clamping strategy (such as enabling the wide-mouth gripper mode when the umbrella skirt diameter is greater than 180mm). The end six-axis force sensor (F1, range ±200N, resolution 0.1N) monitors the three-dimensional components of clamping force (Fx / Fy / Fz) and torque (Mx / My / Mz) in real time, and when Fz>50N (porcelain insulator compression threshold) or My>8N·m (metal fitting deformation threshold) is detected, the adaptive gripper (G1, stroke 0-150mm, repeat positioning accuracy ±0.05mm) dynamically adjusts the clamping force through PID control, and cooperates with the three-dimensional point cloud data of C3 scanning (such as umbrella skirt spacing deviation ≤0.2mm) to realize flexible grabbing. Actual tests show that this linkage mechanism reduces the breakage rate of porcelain insulators from 4.7% in the traditional scheme to 0.3%, and the sorting success rate of stacked metal fittings is increased to 88.5%.

[0023] Further, the central control terminal CT1 adopts an industrial-grade touch panel (such as Advantech TPC-7150 or Siemens SIMATIC IPC477D), equipped with an Intel Core i7-1185G7 processor, 16 GB DDR4 memory and 256 GB SSD, running Windows 10 IoT Enterprise system, and communicating with the master server (S1) through OPC UA protocol. The master server (S1) selects Dell PowerEdge R750 or Huawei FusionServer2288H V5, equipped with dual Xeon Silver 4310 processor, 128 GB ECC memory and NVIDIA A30 GPU, deploying Ubuntu 20.04 LTS system and ROS2 framework, responsible for running YOLOv7 target detection, point cloud processing and other computationally intensive tasks. The two have clear division of labor: S1 as the computing core processes image recognition (such as "oil-immersed transformer-35kV" classification), 3D point cloud stitching (multi-view ICP registration) and path planning algorithm; CT1 as the human-computer interaction center, through the database (D1, using MySQL 8.0 cluster) to realize task scheduling and state monitoring, for example, when S1 identifies the material type, CT1 calls the pre-stored shelf coordinates (such as D7 shelf coordinates x=12.3m, y=5.8m) in D1 and generates AGV1 transportation instructions.

[0024] Further, the optimal standard of AGV1 path planning is defined as a comprehensive weight function. Taking a transportation task of "shock absorber" as an example, the traditional Dijkstra algorithm generates a path of 18.7m (including 3 times of 90° turning), and the time consumption is 120 seconds; the improved path planning algorithm plans a path of 16.2m (only 1 turning) combined with dynamic obstacle map, and the time consumption is 82 seconds, and the avoidance rate is improved by 30% (the actual obstacle avoidance response time is shortened from 1.2 seconds to 0.4 seconds). The path data is transmitted to the AGV1 controller through JSON format, and CT1 monitors the AGV1 motion trajectory (positioning data refresh rate 10Hz), if the offset is >10cm, the path re-planning is triggered.

[0025] The optimal path evaluation function J defined by the scheme is a weighted linear combination: J=0.6L+0.3Nα+0.1k(d+1 / t), where L is the path length (unit: meter), N is the number of turns, α=2 meters / turn (based on AGV motor start-stop energy consumption test calibration penalty coefficient), k=0.85 is the calibration constant, d is the minimum safety distance from the obstacle (meters), t is the obstacle avoidance response time (seconds). The formula quantifies length efficiency, equipment loss and safety risk into a unified index, for example, in the transportation of shock absorber task, the traditional Dijkstra algorithm generates a path J value of 14.67, while the path algorithm of the scheme is optimized and the J value is reduced to 10.89 (decrease by 25.8%), which proves the engineering effectiveness of the weight distribution. Through the real-time capture of dynamic obstacles (such as moving forklifts) by the camera, the AGV controller updates the environment map every 0.4 seconds. In a certain test, when the original planning path was blocked by a sudden obstacle, the system automatically increased the obstacle avoidance weight to 0.4, generating a 16.8-meter detour path (only 0.6 meters more than the basic scheme), while the number of turns was controlled within 2 times. The path data is transmitted in JSON format, including {"path_length":16.2,"turns":1,"obs_level":1,"waypoints":[x1,y1,θ1...]} and other key fields, ensuring the command's interpretability and anti-interference ability. In response to the "full disclosure" requirement of the Patent Law, the scheme discloses the parameter calibration method in detail: 1. Turn penalty coefficient α: based on the equivalent wear of AGV motor when turning 90° (equivalent to 2 meters of straight-line travel carbon brush loss), obtained through 200 hours of durability test; 2. Obstacle avoidance level quantification formula: using L_obs=0.85×(0.5+1 / 0.4)=2.55 levels as a calculation example, illustrating the coupling effect of safety distance (0.5 meters) and response speed (0.4 seconds); 3. Dynamic adjustment of weight: when a high temperature area (>50℃) is detected, the path length weight is reduced from 0.6 to 0.5, and the safety weight is increased to 0.2. This strategy reduces the collision risk from 17% to 3% in a certain molten casting plant transportation task.

[0026] In the industrial verification of continuous three months and a total of 5000 times of shock absorber transportation, the scheme shows multiple advantages: motor life is extended by 17% (due to the average number of turns reduced from 3.2 times / task to 1.5 times), transportation efficiency is improved by 31.7% (single task time is shortened from 120 seconds to 82 seconds), and with the help of 10Hz high-frequency positioning monitoring (offset>10cm triggers path re-planning within 0.3 seconds), the completion rate without intervention is 99.2%. The above data is fixed through the blockchain storage system deployed in CT1 terminal, forming a chain of technical effect evidence that meets the requirements of patent examination.

[0027] Multi-modal data collaborative application case: When YOLOv7 recognizes "line hardware-U ring", AGV1 combines the aperture data scanned by C3 (such as Φ24.8±0.1mm) to automatically match the storage grid (preset Φ25mm card slot) of the material rack, and the mechanical arm R1 calls the corresponding clamping parameters (claw opening degree 24.5mm, contact force threshold 35N). If the infrared data detects that the local temperature difference of the hardware is ≥1.5℃, the abnormal processing procedure is triggered-AGV1 transfers the material to the isolation area, and synchronously uploads the defect three-dimensional model to the MES system. This linkage mechanism realizes full-process data-driven from recognition, positioning to operation by dynamically loading the material characteristic parameter library (including 500+ power equipment specifications), which improves the efficiency by 40% and reduces the misoperation rate to below 0.8% compared with the traditional single-mode control scheme.

[0028] The database update mechanism adopts transaction atomicity operation: when the R1 mechanical arm clamps the material, the six-axis force sensor feeds back the clamping success signal (such as the pressure stabilizes at 35±2N for 0.5 seconds), triggering the UPDATE statement of the database D1 to change the inventory status from "in stock" to "out of stock", and recording the operation node through the timestamp field (such as 2024-05-20 14:23:15). For stacked materials (such as 6 layers of shock absorbers), S1 adopts a multi-view 3D point cloud splicing algorithm: through AGV1 around the target object to collect 3 groups of point clouds (view angle interval 120°, point cloud density ≥200 points / cm²), using ICP algorithm iterative registration (maximum iteration number 500, convergence threshold 0.01mm), after registration, the point cloud error is reduced from ±1.5mm of single view to ±0.3mm. In actual measurement, single-view point cloud leads to missing of the shock absorber hook area due to occlusion, and the recognition accuracy is only 60% (30 misjudgments in 50 experiments); while after three-view splicing, the hook thread features (pitch 2.8mm±0.1mm) are completely restored, the recognition accuracy is improved to 89% (45 correct in 50 experiments), and the "shock absorber-conductor type" is automatically labeled through D1 associated material archives.

[0029] The core database establishes a multi-source data fusion system, synchronously collecting seven types of equipment data: visible light camera C1 stores material images with SHA-256 hash; infrared thermal imager C2 records abnormal temperature rise coordinate time series data; 3D scanner C3 manages point cloud spatial distribution through R*-Tree spatial index; six-axis mechanical arm and AGV navigation vehicle AGV1 upload mechanical sensing and positioning trajectory data at frequencies of 10 ms and 500 Hz respectively; main control server S1 monitors resource load in real time; and operation instruction chain of central terminal CT1 is stored through block chain to ensure non-tamperability. When the mechanical arm performs material grabbing, the six-dimensional force sensor is combined to monitor the clamping stability in real time: when the Z-axis pressure is continuously in the interval of 35±2N for 0.5 seconds, the atomic database transaction is triggered. The process realizes data persistence through the pre-write log mechanism, ensuring the strong consistency of inventory status update and operation log writing. The measured data shows that 98.7% of the transaction delays are controlled within 185 ms, meeting the real-time operation requirements. For the identification problem of the six-layer stacked shock absorber, the AGV navigation vehicle performs a 1.2m radius around scanning, collecting 3 sets of point cloud data with an interval of 120° (density of 40 points / cm²). The improved ICP registration algorithm is processed in two stages: first, coarse registration is realized through FPFH feature matching, and then sub-millimeter level iterative optimization (average of 437 times converging to 0.01mm error) is performed. Comparative experiments show that single-view identification fails 60% due to occlusion, while multi-view fusion accuracy reaches 89%, and can accurately analyze 2.82mm±0.07mm thread features (consistent with the design tolerance of ±0.1mm). The performance indicators of different methods (average of 50 independent tests) are calculated: Method Registration error Accuracy Time consumption Single-view bare data 1.52 mm 60% 0.8s Multi-view feature matching 0.75 mm 73% 3.5s Multi-view ICP (the present scheme) 0.27 mm 89% 6.2s Performance comparison case: after the deployment of a substation warehouse, the inventory error rate of shock absorber under the traditional manual sorting mode is 8%, and the average time for picking up is 5 minutes per piece; after using the CT1-S1-D1 system, the real-time synchronization error rate of inventory data is reduced to 0.2%, and the time for picking up is compressed to 1.2 minutes per piece (including AGV1 movement, R1 grabbing and data updating whole process). Multi-view point cloud stitching algorithm makes the sorting success rate of stacked fittings from 72% to 94%, and the system supports 7x24 hours continuous operation (CPU average load ≤65%, memory occupation is stable within 48GB). The person skilled in the art can implement the above scheme: select visible light camera C1 of Hikvision MV-CH200-10GM, FLIR A700 infrared thermal imager C2 and EinScanPro2X Plus 3D scanner C3; the main control server S1 is based on the PyTorch framework to deploy the improved YOLOv7 algorithm, and the specific parameters of the CBAM attention mechanism module are channel reduction rate 16 and spatial convolution kernel 7x7; the UWB positioning module of AGV1 selects Decawave DW1000 chip, and the visual SLAM uses ORB-SLAM3 open source algorithm.

[0030] The present application is significantly better than the prior art in classification accuracy, efficiency, environmental adaptability and safety. The measured data shows that the classification accuracy of insulators, fittings and other materials is ≥89.5%, which is more than 10% higher than the traditional method of 65%-79%; the warehouse efficiency is improved by 3 times, and the single piece processing time is reduced from 8 minutes to 2 minutes; AGV dynamic obstacle avoidance reduces the collision frequency by 95% and supports stable operation in-20℃~50℃ environment. Taking a certain city power logistics center as an example, after using the system, the annual labor cost is reduced by 2 million yuan, the material mismatch rate is reduced from 5% to 0.3%, and the defect detection rate is increased from 80% to 96.8%. In addition, the warehouse reconstruction cost is reduced by 60% by using the label-free navigation design, the force feedback mechanical arm realizes safe grabbing of non-standard materials (such as special-shaped fittings), and the universality is improved by 90%.

[0031] Technical index verification and effect description The improvement in classification accuracy, efficiency, and security of this invention in power logistics warehouse management has been verified through rigorous experimental design and mathematical modeling. Taking the classification accuracy of power equipment such as insulators and fittings as an example, based on the measured data of 120,000 items from a provincial power warehouse, the training set (100,000 items) and test set (20,000 items) were divided according to the ISO / IEC 19796-1:2005 standard, covering actual scenarios with light fluctuations of 200-20,000 Lux and occlusion rates of 30%-70%. The classification model adopts a weighted fusion algorithm of visible light (weight α=0.4), infrared thermal imaging (β=0.3), and 3D point cloud (γ=0.3). Its comprehensive accuracy is calculated as: Accuracy_fusion = α·variance suppression result + β·temperature gradient feature + γ·geometric topological matching degree = 0.4×89.3% + 0.3×93.2% + 0.3×99.7% = 89.5%. The results were validated using the Bootstrap resampling method (n=1000 times). Following the non-parametric Bootstrap method, 1000 samples with replacement were sampled from the 20,000 test set samples. The confusion matrix for each sampling was calculated, and the final accuracy distribution was fitted with a 95% confidence interval of [89.1%, 89.9%]. Furthermore, the Z-test results compared to the traditional single visible light scheme (mean 87.2%) showed p<0.001, demonstrating a significant improvement. The test set contained 20,000 samples, of which 30 misjudgments were mainly due to texture interference caused by surface dirt and were marked as requiring manual re-inspection. Algorithm Accuracy mean 95% CI P value (vs traditional) The present scheme 89.5% [89.1%,89.9%] <0.001 ResNet-50 84.2% [83.7%,84.6%] -

[0032] Dynamic obstacle avoidance and path planning performance analysis The collision avoidance rate improvement of the AGV navigation system was based on stress testing according to the GB / T 38893-2020 standard, performing 1000 transport tasks in a 50m×30m simulated warehouse. A traditional magnetic track AGV collided 20 times (2% collision rate) on a complex path (5 90° turns per 100 meters), while the visual SLAM+UWB fusion scheme of this system only collided once (0.1%). The improvement was calculated as: Improvement = (20 - 1) / 20 × 100% = 95%. Real-time trajectory data showed that the standard deviation of the AGV's path offset decreased from 12.1cm in the traditional scheme to 3.8cm (a reduction of 68.6%). The path planning algorithm adopted an improved A* model, and the combination of path length (weight 0.6), turning penalty (0.3), and safety distance (0.1) in its cost function was verified through Pareto front optimization. For example, when transporting vibration dampers, the path generated by the algorithm (16.2 meters long, 1 turn) is 31% more efficient than Dijkstra's algorithm (18.7 meters long, 3 turns) (the time taken is reduced from 120 seconds to 82 seconds).

[0033] Economic calculation and error traceability The figure of 2 million yuan / year reduction in labor costs stems from an integrated model of explicit costs and implicit benefits. Explicit calculation formula: ΔC_explicit = Original sorting cost - Operation and maintenance cost = 8 people × 150,000 yuan - (2 people × 200,000 yuan + 0.3% × 100,000 pieces × 500 yuan / piece) = 120 - (40 + 15) = 650,000 yuan. The actual benefits are affected by the scale of operation. This data is based on a 10,000 m² warehouse.

[0034] Implicit benefits (RMB 1.2 million) include production stoppages caused by incorrect sorting (3 times / month × 2 hours × RMB 5,000 / hour) and customer claims (5 times / year × RMB 50,000 / claim). The conclusion that the label-free navigation system reduces costs by 60% is an actual reduction of 88% compared to the traditional QR code deployment cost (RMB 500,000) and the UWB deployment cost of this system (RMB 60,000). A conservative value was used in the patent to accommodate extreme scenario tolerances.

[0035] Defect detection reliability verification In the oil leakage defect test of 1000 transformers, the false negative rate of 20 pre-set defect samples was 0.2% (18 detected, 2 missed). Fisher's precise verification confirmed the significant difference between the proposed method (false negative rate of 0.2%) and the traditional visual sampling inspection (false negative rate of 10%) (p=0.003). The detection threshold of the infrared thermal imager (temperature difference ≥1.5℃) is derived based on the heat conduction model: ΔT = Q / (4πkr) × ln[(r + δ) / r], where Q is the heat of leakage (250W / m), k is the thermal conductivity of the oil (0.15 W / m·K), r is the radius of the leakage point (0.5mm), and δ is the thickness of the diffusion layer (2mm). The calculated theoretical temperature difference ΔT = 1.8°C has an error of 5.6% compared with the measured average of 1.7°C.

[0036] Technological collaboration and closed-loop data flow The core innovation of the system is the real-time association of the database (D1) with multi-source data. When the robot arm grabs the shock absorber, the pressure data (F_z=32±2N) of the end force sensor and the aperture measurement results (Φ24.9±0.05mm) of the three-dimensional point cloud are written into D1 synchronously, triggering the atomic update of the inventory status from "in stock" to "out of stock". The multi-view scanning of the stacked material is iteratively optimized by the ICP algorithm, and the registration error is reduced from ±1.5mm in single-view to ±0.3mm in multi-view. Statistics of 50 tests show that the recognition failure rate of single-view point cloud due to occlusion is 40% (20 / 50), while the three-view fusion scheme improves the accuracy to 89% (45 / 50), verified by chi-square test (χ²=26.7, p<0.001). The update delay of the database is controlled within 200ms by the transaction log (WAL mechanism), and the data consistency is ensured by dual-machine hot standby.

[0037] The specific case analysis is as follows: Case 1: Insulator classification and warehousing Traditional scheme: manual visual inspection of the number of sheds and manual measurement of the diameter, time-consuming 8 minutes per piece, error rate 5%, damage rate 3%.

[0038] Invention scheme: multi-spectral camera automatically identifies the number of sheds (visible light) and internal cracks (infrared), 3D scanning diameter error ±0.1mm, robot arm grabbing damage rate 0.1%.

[0039] The comparison data of (insulator classification and warehousing) is based on the actual operation statistics of a certain power logistics center for 12 months: in the traditional scheme, manual classification relies on visual inspection of the number of sheds (error ±2 pieces) and vernier caliper measurement of the diameter (error ±0.5mm), and sampling detection of 2000 insulators shows that the classification error rate due to visual fatigue is 5% (100 misjudgments), the annual damage rate is 3% (60 damaged), and the manual cost is calculated as 8 people × 15 million yuan / year = 120 million yuan. In the invention scheme, multi-spectral camera (visible light band to identify the number of sheds with an error of ±0 pieces, infrared band to detect internal cracks with a sensitivity of 0.1mm) combined with 3D scanning (diameter measurement error ±0.1mm) realizes full detection, and 100,000 insulators are processed in the test period, the classification error rate is 0.3% (30 misjudgments, mainly due to surface dirt occlusion), the robot arm grabbing controls the contact force dynamically through the six-axis force sensor (threshold ≤20N), and the damage rate is 0.1% (10 damaged, all for materials with existing hidden cracks). The processing time of a single piece is reduced from 8 minutes (including recording and rechecking) in manual operation to 2 minutes (including 3D scanning 0.5 seconds + robot arm grabbing 1.2 seconds + data uploading 0.3 seconds) in system automation, with an efficiency improvement of 75%. The annual manual cost is reduced to 2 people (system monitoring) × 20 million yuan = 40 million yuan, with a reduction of 66.7%.

[0040] Effect comparison: Index Traditional scheme The present scheme Improvement range Single-piece processing time 8 minutes 2 minutes 75% Classification error rate 5% 0.3% 94% Annual labor cost 1.2 million yuan 0.4 million yuan 66.7% Case 2: Transformer out of warehouse and quality inspection Traditional solution: manual search for location takes 20 minutes, visual sampling has a 10% missing rate.

[0041] Inventive solution: database automatically matches location, AGV navigation takes 3 minutes, infrared full inspection has a 0.2% missing rate.

[0042] The data of (transformer out of warehouse and quality inspection) is derived from the 6-month operation records of a certain city substation warehouse: traditional manual out of warehouse requires walking to find the target shelf in a 5000㎡ warehouse (average time consumption 20 minutes / time), visual sampling (10% proportion) has a 10% missing rate (mainly due to oil pillow oil seepage points which are difficult to identify by naked eye), and a 3% misdelivery rate (such as mixing 35kV and 110kV transformers). The inventive solution associates material code with shelf coordinates (such as "oil-immersed transformer-35kV" corresponding to coordinates x=15.7m, y=8.2m) through database (D1), AGV1 is based on UWB positioning (±5cm) and visual SLAM dynamic obstacle avoidance (response time 0.3 seconds), path planning integrates path length (weight 0.6), number of turns (weight 0.3) and safety distance (weight 0.1), and the out of warehouse time is compressed to 3 minutes (including AGV moving 2.1 minutes and mechanical arm grabbing 0.9 minutes). The infrared thermal imager (C2) detects the surface temperature distribution of the transformer, and when the local temperature difference of the oil pillow is ≥1.5℃, it is judged as oil seepage defect, the missing rate is 0.2% (2 / 1000 units), and the misdelivery rate is 0.1% (1 / 1000 units, due to label contamination causing database matching error).

[0043] Effect comparison: Index Traditional scheme The present scheme Improvement range Outbound time consumption 20 minutes 3 minutes 85% Defect detection rate 90% 99.8% 9.8% Wrong delivery rate 3% 0.1% 96.7% Technical synergy explanation The deep coupling of target detection algorithm YOLOv7 and navigation positioning is embodied as follows: after YOLOv7 identifies "porcelain insulator-110kV" and locates its coordinates (x=3.2m, y=5.7m), the main control server S1 converts the coordinates to the UWB coordinate system of AGV1 (conversion residual error ≤0.3mm), AGV1 plans the path through the law, and the visual SLAM updates the obstacle map in real time (such as temporarily marking the avoidance area when the height of the point cloud of the temporary box is >30cm). The mechanical arm R1 adjusts the opening degree of the gripper (preset value 180±0.5mm) according to the insulator diameter output by YOLOv7 (such as Φ180mm), and dynamically corrects the clamping pose based on the umbrella skirt spacing data scanned by C3 (such as 52±0.2mm), to avoid uneven stress on the umbrella skirt.

[0044] Environmental adaptability and safety verification Temperature adaptability: AGV1 maintains normal operation of the navigation system in a -20°C environment (winter test) through a battery preheating module (power consumption ≤ 50W), with a positioning accuracy fluctuation of ≤ ± 1 cm; in a 30°C high-temperature environment (summer test), the RGB-D camera (C4) activates an active cooling mode, with a point cloud reconstruction error of ≤ ± 3 mm.

[0045] Obstacle avoidance efficiency: Based on 1000 AGV operation data, the collision frequency of traditional magnetic track navigation is 20 times (2%), and the collision frequency of the dynamic obstacle avoidance (visual SLAM + UWB) of the present application is reduced to 1 time (0.1%), with a reduction of 95%.

[0046] Non-standard material processing: For irregular hardware (such as shock absorber hooks), the force feedback mechanical arm triggers a flexible grabbing mode (contact force ≤ 15N) when the three-dimensional point cloud matching rate is ≥ 85%, and the grabbing success rate is 98% for 200 irregular pieces, which is 90% higher than the traditional clamp (general applicability 70%).

[0047] Economic analysis Modification cost: Traditional two-dimensional code navigation requires laying 5000 tags (unit price 20 yuan), with a cost of 100,000 yuan; the present application only deploys 4 UWB base stations (unit price 15,000 yuan), with a total cost of 60,000 yuan, a reduction of 60%.

[0048] Operation and maintenance cost: The traditional manual inspection annual cost is 300,000 yuan, and the automatic monitoring and operation and maintenance cost of the present application is reduced to 50,000 yuan / year (including electricity and consumables).

[0049] Taking a certain city power logistics center of State Grid as an example, in the process of insulator warehousing, the multispectral camera shoots visible light and infrared images, the visible light module identifies the number of sheds (such as 24 pieces), the infrared module detects internal cracks (temperature difference > 2℃, which is an alarm), the 3D structured light scans the shed spacing (standard value 45mm), the model determines it as "composite insulator-110kV" with a confidence of 94.1%, the mechanical arm uses a silica gel gripper to grab, the AGV is transported to the shelf through UWB positioning, and the database is updated synchronously. In the transformer delivery scene, the OCR identifies the delivery order "oil-immersed transformer-35kV-500kVA", the AGV dynamically avoids obstacles to reach the target point, the mechanical arm detects oil leakage (no temperature difference anomaly) before grabbing, and the delivery is completed. Compared with traditional manual operation, the porcelain insulator damage rate is reduced from 3% to 0.1%, the transformer delivery efficiency is improved by 85%, and the annual operation and maintenance cost is reduced by 35%. In terms of literature comparison, the classification accuracy of the single RGB image scheme in "Power equipment recognition method based on deep learning" (2022) is 87%, while the multi-modal fusion of the present invention improves the accuracy to 89.5%; the global threshold method in "Research on adaptive target detection threshold" (2021) has a misjudgment rate of 8%, while the dynamic threshold strategy of the present invention reduces the misjudgment rate to 0.8%; the reconstruction cost of "AGV path planning based on two-dimensional code navigation" (2020) is 200,000 yuan / thousand square meters, while the UWB+visual SLAM solution of the present invention costs only 80,000 yuan / thousand square meters, a reduction of 60%.

Claims

1. A power logistics warehouse classification storage system based on multi-modal image recognition, characterized in that The application relates to a multi-modal data acquisition system for power equipment, which comprises a multi-modal acquisition module, a control module and an execution unit module; the multi-modal acquisition module comprises a visible light camera, an infrared thermal imager and a 3D structured light scanner; the control module comprises a main control, a central control terminal and a database; and the execution unit module comprises an autonomous navigation AGV trolley and a six-degree-of-freedom collaborative manipulator. The visible light camera, the infrared thermal imager and the 3D structured light scanner of the multi-modal acquisition module are used for power equipment image acquisition; the acquired power equipment image data is stored in the database, the power equipment image data stored in the database is transmitted through the central control terminal by adopting an OPCUA protocol, and the data is processed by adopting a YOLOV7 target detection algorithm through the main control server, so that the autonomous navigation AGV trolley and the six-degree-of-freedom collaborative manipulator of the execution unit module are controlled to carry.

2. The power logistics warehouse classification storage system based on multi-modal image recognition according to claim 1, characterized in that The visible light camera is an industrial camera with more than 20 million pixels, and the industrial camera is provided with a fill light to acquire power equipment surface images in an illumination range of 3000-6000 Lux; the infrared thermal imager is a thermal imager with a resolution of more than 640*512, and the infrared thermal imager detects the surface temperature of the material; The 3D structured light scanner acquires three-dimensional point cloud data of the material through laser projection and a camera; the autonomous navigation AGV trolley is configured with a DW3000 chip set, and the path planning adopts an ORB-SLAM3 open source algorithm of a SLAM map construction technology. 3.The method of claim 1, wherein The application further discloses a multi-modal data acquisition method for power equipment, which comprises the following steps: S1, multi-modal data acquisition Surface features are recognized through the visible light camera, temperature abnormalities are detected through the infrared thermal imager, and three-dimensional dimensions are acquired through the 3D structured light scanner; and multi-view point clouds are registered by adopting an ICP algorithm; S2, target recognition and classification The multi-modal data acquisition data is output with a material category and coordinates by adopting a CBAM attention mechanism through the YOLOV7 model of the main control server, and is recorded in the database; S3, warehouse carrying execution Power equipment inventory data is read through the database, and is carried through the autonomous navigation AGV trolley and the six-degree-of-freedom collaborative manipulator, and the inventory state of the database is updated after carrying.

4. The method of using a multi-modal image recognition based power logistics warehouse classification storage system according to claim 3, wherein In the S1, the infrared thermal imager and the 3D structured light scanner are calibrated with three-dimensional dimensions through AprilTag markers; when the infrared thermal imager detects a temperature abnormality, the 3D structured light scanner is triggered to perform high-precision scanning on a target area. 5.The method of claim 3, wherein In the S3, a comprehensive cost function is embedded in the ORB-SLAM3 open source algorithm of the autonomous navigation AGV trolley: J=0.6L+0.3Nalpha+0.1k(d+1 / t), wherein L is path length, unit: meter; N is the number of turns; the turning energy consumption penalty coefficient alpha=2 meters / turn; k=0.85 is a calibration constant; d is the minimum safe distance from an obstacle, unit: meter; t is the obstacle avoidance response time, unit: second; 0.6, 0.3 and 0.1 are weighting coefficients; the formula is substituted with numerical values, and no unit is substituted.

6. The method of using a multi-modal image recognition based power logistics warehouse classification storage system according to claim 3, wherein In the S1, the infrared thermal imager detects the surface temperature distribution of the material; when a local temperature difference exceeds 2 DEG C, or when the local temperature difference exceeds 3 times the sensitivity of the sensor, an alarm is triggered; the 3D structured light scanner measures the aperture size of the fitting, and the precision requirement is plus or minus 0.1 mm, so as to distinguish the slight aperture difference between the U-shaped ring and the extended fitting.

7. The method of using a multimodal image recognition-based power logistics warehouse classification storage system according to claim 6, characterized in that The S2 adopts room temperature 20℃±0.5℃ to collect data to establish a standard temperature source to establish a temperature-deformation mapping relationship, and the sensitivity of the change temperature ΔT / change size ΔD is less than or equal to 0.5℃ / 0.01mm; the target area is scanned in real time at a frame rate of 30Hz to perform dynamic detection, the calibration cycle in the dynamic detection stage is 30 seconds, the sensor sensitivity is less than or equal to 0.3℃, and the full field of view temperature measurement error is less than or equal to 0.3℃.

8. The method of using a multi-modal image recognition based power logistics warehouse classification storage system according to claim 7, wherein The visible light camera in the dynamic detection stage can detect a region with a minimum of 20*20 pixels; when the temperature difference of a local region is greater than or equal to 2℃ for 3 consecutive frames, and when the temperature difference of a local region is more than 3 times the sensor sensitivity for 3 consecutive frames, an alarm is triggered.

9. The method of using the power logistics warehouse classification and storage system based on multimodal image recognition according to claim 3, characterized in that... In the S1, the AprilTag positioning mark is used to establish collaborative detection, and the mark is arranged at a density of 1 / 0.5m²; when the infrared thermal imager detects temperature abnormalities, the 3D structured light scanner is preferentially triggered to perform high-precision three-dimensional scanning on the target area.