Inspection method of industrial inspection system based on multi-modal fusion

Abstract

The invention discloses an inspection method of an industrial inspection system based on multi-modal fusion, which belongs to the technical field of industrial inspection, and comprises the following steps: S1, obtaining positioning data of a node to be positioned by using an HDS-TWR distance measurement algorithm and a Taylor full centroid algorithm, and outputting a positioning coordinate; s2, using a sensor to measure the acceleration and angular velocity of the inspection system so as to align the motion and posture for evaluation; s3, improving a Kalman filtering algorithm to enable an output prediction result to be close to a true value; s4, infrared-vision dual-mode cooperative tracking is adopted, and meanwhile, a time synchronization displacement compensation mechanism is utilized to calculate a relative displacement difference between two tracking modes, and dynamic calibration is carried out; s5, a YOLOv8 model is adopted to process video frames or pictures collected in the inspection process, a target box and category probability are output, and finally an inspection result is obtained; according to the inspection method of the industrial inspection system based on multi-modal fusion, a high-precision and low-delay automatic inspection solution is provided for severe environments such as mines and nuclear power stations.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial inspection, and in particular to an inspection method for an industrial inspection system based on multi-modal fusion. Background Art

[0002] With the rapid development of industrial intelligence and the continuous improvement of the demand for safety monitoring, the applicability of traditional inspection methods in complex indoor environments and narrow and dangerous areas (such as mines, nuclear power plants, chemical plants, etc.) has been greatly challenged. Traditional inspection methods rely on manual operation, which not only has a high labor intensity and high cost, but also has a large safety risk in high-risk environments. Therefore, automatic inspection and positioning systems have gradually become a research hotspot to improve inspection efficiency, reduce costs, and enhance adaptability and safety in complex environments.

[0003] In recent years, multi-sensor fusion technology (Multi-Sensor Fusion, MSF) has been widely applied in fields such as intelligent robots, autonomous navigation, and environmental perception. This technology combines multiple sensors (such as ultra-wideband UWB, inertial measurement unit IMU, infrared sensors, visual sensors, etc.) to obtain environmental information to overcome the limitations of single sensors in measurement accuracy, environmental adaptability, etc. Among them, UWB positioning technology has become a high-precision positioning solution in complex environments due to its high precision, strong anti-interference ability, high stability, etc. However, UWB may experience multipath effects and signal attenuation in metal-dense environments or scenarios with more signal blockages, resulting in an increase in positioning errors. How to achieve high-precision and high-robustness autonomous navigation and environmental perception has become a key technical bottleneck in the development of industrial intelligence.

[0004] To solve the above problems, this method proposes an automatic inspection and positioning system based on multi-sensor fusion, which combines UWB high-precision positioning, visual environmental perception, and inertial measurement unit (IMU) dynamic compensation technology, aiming to solve the problems of autonomous navigation and task execution in complex closed environments. The system uses the HDS-TWR ranging algorithm and the combined centroid-Taylor solution to optimize the dynamic positioning robustness under multi-base station cooperation and reduce the errors caused by NLOS interference; in practical applications, the accuracy and real-time performance of the inspection system are crucial. To optimize the positioning performance, this method uses an improved Kalman filter algorithm to reduce UWB ranging errors, and uses a deep learning object detection model (YOLOv8) for visual inspection. In addition, this method uses the MQTT communication protocol for data transmission to ensure low power consumption and efficient communication. Summary of the Invention

[0005] The purpose of the present invention is to provide an inspection method for an industrial inspection system based on multi-modal fusion to solve the problems existing in the above background art.

[0006] To achieve the above object, the present invention provides an inspection method for an industrial inspection system based on multimodal fusion, including the following steps:

[0007] S1. After obtaining the ranging data from the UWB base station to the node to be located using the HDS-TWR ranging algorithm, use the Taylor centroid algorithm to calculate the positioning data of the node to be located and output the positioning coordinates;

[0008] S2. Use sensors to measure the acceleration and angular velocity of the inspection system, so as to evaluate the alignment of movement and posture;

[0009] S3. Improve the Kalman filter algorithm to process the data in steps S1 and S2, so that the output prediction result is close to the true value;

[0010] S4. Adopt infrared-vision dual-mode collaborative tracking, and at the same time use the time synchronization displacement compensation mechanism to calculate the relative displacement difference between the two tracking modes for dynamic calibration;

[0011] S5. Use the YOLOv8 model to process the video frames or pictures collected during the inspection process, output the target box and class probability, and finally obtain the inspection result.

[0012] Preferably, the HDS-TWR ranging algorithm in step S1 is obtained according to its ranging principle:

[0013]

[0014] After arrangement, we get:

[0015]

[0016] Where T round1A 、T round1B 、T round1C respectively represent the round-trip time for the tag to send the Poll message to base stations A, B, and C and receive the first response messages RespA, RespB, and RespC from the corresponding base stations; T round2A 、T round2B 、T round2C is the time interval from when the tag receives the first response message from the base station to when it sends the Final message; T reply1A 、T reply1B 、T reply1C is the time for base stations A, B, and C to send the RespA, RespB, and RespC response messages after receiving the Poll message; T reply2A 、T reply2B 、T reply2C is the time from when the tag sends the Final message to when base stations A, B, and C receive the Final message; T propA 、T propB 、TpropC They respectively represent the propagation time of the signal between the tag and base stations A, B, and C.

[0017] Preferably, in step S1, the Taylor full centroid algorithm is used to calculate the positioning coordinates specifically as follows:

[0018] Assume that the coordinates of base stations A, B, C and the tag are (x1, y1), (x2, y2), (x3, y3) and (x, y) respectively, and the ranging distances from each positioning base station to the node to be located are r1, r2, r3, and the following equations are obtained:

[0019]

[0020] Using the full centroid algorithm to calculate, it is rewritten in matrix form:

[0021]

[0022] Furthermore, we get:

[0023]

[0024] Combining formulas (5) and (6), define The following linear equations are obtained:

[0025] aθ = b (7)

[0026] Using the least squares method to calculate θ, we get:

[0027] θ = (a T a) -1 a T b (8)

[0028] Using the Taylor algorithm, first make a relatively accurate initial estimate of the coordinates of the tag to be measured, and then, under the premise of the least quadratic variance criterion, iterate continuously for multiple times until the preset threshold accuracy or the number of iterations is reached. Thus, the initial coordinates (x k , y k ) of the tag are obtained, and the actual coordinates satisfy:

[0029]

[0030] Among them, Δx and Δy are the differences between the estimated coordinates and the true coordinates;

[0031] As can be seen from formula (4), the distances measured by the base stations participating in the positioning ideally satisfy the formula:

[0032]

[0033] Among them, (x i , yi ) is the coordinate of the i-th base station, and r i is the distance from the i-th base station to the positioning node; the linear equations are solved by performing a first-order Taylor expansion on (10). For the binary Taylor expansion at (x k ,,y k ), we have:

[0034] r i = f(x, y) ≈ f(x, y) + (x - x k )f' x (x k ,y k ) + (y - y k )f' x (x k ,y k ) (11)

[0035] Substituting f(x, y) into formula (10), we get:

[0036]

[0037] Combined with formula (9), Δx and Δy are calculated by the least squares method, where:

[0038]

[0039] When |Δx| + |Δy| is less than the preset convergence threshold or the number of iterations exceeds the maximum allowable value, the algorithm terminates the iteration and outputs the positioning coordinates.

[0040] To improve the stability of the ranging data, the sliding window extreme value mean algorithm is used to process the distance observation values: a circular buffer queue with a length of 3 is established to store the latest three ranging results between the base station and the tag in real time. When the data is updated each time, extreme value detection is performed on the data within the window, and the maximum and minimum values are removed. Then, arithmetic mean operation is performed on the remaining valid observation values, and finally, the filtered ranging value is output as the input quantity for positioning calculation. This composite algorithm effectively suppresses the wild value interference through a dual fault tolerance mechanism, and improves the robustness of the positioning system while ensuring the calculation efficiency.

[0041] Preferably, in step S2, the MPU6050 sensor is used to measure data, and the DMP library automatically fuses the sensor data and outputs the quaternion for pose calculation. Specifically:

[0042] Assume that the quaternion outputs of multiple sensors are Q1, Q2, Q3...Q n , and the final quaternion is obtained using the weighted average method:

[0043]

[0044] Among them, qi is the weight coefficient, Q i is the quaternion output by the i-th sensor;

[0045] The data obtained by the MPU6050 sensor includes acceleration a = (a x , a y , a z ), angular velocity ω = (ω x , ω y , ω z ), pitch angle and roll angle θ, where:

[0046]

[0047] Among them, the pitch angle and roll angle are used to calculate and update the quaternion:

[0048]

[0049] Preferably, in step S3, the state value is split into x and y, and two Kalman filters are performed separately. The filtering method is divided into two steps: prediction and update:

[0050] Prediction:

[0051] X k = X k-1 (18)

[0052] P k = P k-1 + Q (19)

[0053] Update:

[0054]

[0055] X k = X k + K k (z k - X k ) (21)

[0056] P k = (I - K k )P k (22)

[0057] Among them, X k is the state value at time k; P k is the covariance matrix of the state estimate; Q is the process noise covariance matrix; K k is the Kalman gain calculated at time k; R is the measurement noise covariance matrix; z k is the observed value at time k; I is the identity matrix.

[0058] Preferably, the infrared-vision dual-mode collaborative path tracking in step S4 includes vision path tracking based on an OpenMV camera and an infrared reflection path tracking module. Among them, the vision path tracking identifies specific path markers through color threshold segmentation and multi-ROI detection, and sends the detection results to the main controller through a serial port. Specifically:

[0059] Definition of ROI area coordinates: Assuming the image resolution is W×H = 160×120, five ROI areas evenly distributed horizontally are defined. The coordinate formula for the i-th ROI is:

[0060] ROI i =(x i ,y,w,h) (23)

[0061] x i =x0 + i×(spacing + w), i∈(0,1,2,3,4) (24)

[0062] Among them, each value of ROI i corresponds to the x coordinate of the upper left corner point of the ROI selection position, the y coordinate of the upper left corner point of the ROI, the width of the ROI, and the height of the ROI;

[0063] Color threshold segmentation: Define the color threshold range as: Threshold = [L min ,L max ,A min ,A max ,B min ,B max , where each value corresponds to the minimum value of brightness, the maximum value of brightness, the minimum value of the A channel, the maximum value of the A channel, the minimum value of the B channel, and the maximum value of the B channel;

[0064] Detection result encoding: The detection result f i of each ROI is defined as a binary variable:

[0065] If there is a target color block in the ROI, f i =1, otherwise f i =0;

[0066] The five detection flags finally sent are f1, f2, f3, f4, f5;

[0067] Serial port data frame structure: The data frame format is 8-byte little-endian mode: Data = [0xA5, 0xA6, f1, f2, f3, f4, f5, 0x5B].

[0068] Preferably, the infrared reflection path tracking module specifically includes:

[0069] Infrared reflection intensity model: When an infrared emitter emits infrared light with an intensity of I0, after being reflected by the target surface, the intensity of the light detected by the receiving tube is I r It is expressed as:

[0070]

[0071] where K is the optical system efficiency; r is the target surface reflectivity; d is the distance between the sensor and the target; μ is the medium attenuation coefficient;

[0072] Signal threshold determination: The received signal voltage V and the light intensity I r are linearly related:

[0073] V = α × I r + β (26)

[0074] where α is the optoelectronic conversion coefficient and β is the environmental noise;

[0075] Set the threshold V th , and output a digital signal:

[0076] Preferably, the time synchronization displacement compensation mechanism is specifically as follows:

[0077] Assume that the camera and the tracking module detect a certain point P at times t and t + Δt respectively;

[0078] Establish the spatio-temporal observation equation of the camera and the tracking module. The observed position of the camera at time t is:

[0079] x c (t) = x0 + vt (27)

[0080] where x0 is the initial position;

[0081] The position of the monitoring point of the tracking module:

[0082] x t (t + Δt) = x0 + v(t + Δt) (28)

[0083] By differentiating the time difference Δt, calculate the relative displacement difference between the two monitoring points P:

[0084] Δx = x t (t + Δt) - x c (t) = vΔt (29)

[0085] Characterize the spatial offset caused by time asynchrony. Compare the relative displacement difference Δx with the preset physical distance L. If Δx > L, it indicates that the spatio-temporal consistency is damaged, and trigger the offset adjustment mechanism; where x t(t + Δt) is the position of point P monitored by the tracking module at time t + Δt; v is the speed of the vehicle; L is the distance between the camera and the tracking module.

[0086] Preferably, step S5 specifically includes:

[0087] The structure of the YOLOv8 model includes:

[0088] The CSPDarknet backbone network, the formula is:

[0089]

[0090] The SPPF module, serial pooling to fuse multi-scale features:

[0091]

[0092] Among them, X in is the input feature map; X out is the output feature map; n is the number of channels of the input feature map; Conv is the convolution operation; is the feature concatenation; MaxPool k is the max pooling operation with a window size of k;

[0093] The YOLOv8 model uses DEL to optimize the bounding box for prediction, and the DEL formula is:

[0094] Model the bounding box coordinates as a discrete probability distribution, and predict the probabilities P = [p0, p1,..., p n of n + 1 intervals, and calculate the coordinate values through expectation:

[0095]

[0096] The DFL loss encourages high probabilities in the intervals near the true coordinates:

[0097] DFL(p, t0) = -((t i+1 - t) log(p i ) + (t - t i ) log(p i+1 )) (33)

[0098] Among them, is the predicted coordinate value calculated through the probability distribution, y i is the central value of the i-th discrete interval, t0 is the true bounding box coordinate value, t i , t i+1 is the endpoint of the discrete interval where the true coordinate t0 is located, and satisfies t i ≤ t ≤ t i+1 ;

[0099] Improve the regression accuracy through CIOU Loss, and the formula is:

[0100]

[0101] where p(b, b gt ) is the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c is the diagonal length of the smallest enclosing rectangle of the predicted bounding box and the ground truth bounding box; w and h are the width and height of the predicted bounding box; w gt , h gt are the width and height of the ground truth bounding box; τ is the aspect ratio consistency penalty term; γ is the dynamic weight;

[0102] Select positive samples through the task alignment metric to improve the consistency between classification and regression:

[0103] align_metric = (p c ·IOU(b, b gt )) δ (35)

[0104] Select the top k anchors with the highest align_metric corresponding to each ground truth bounding box as positive samples;

[0105] where p c is the predicted class probability, IOU(, b gt ) is the intersection over union of the predicted bounding box and the ground truth bounding box, δ is a hyperparameter, and k is the number of positive samples selected for each ground truth bounding box.

[0106] Data transmission adopts the publish / subscribe communication model based on the MQTT protocol. The server and the terminal device act as publishers and subscribers to publish or subscribe to information to relevant topics, and use EMQX as the MQTT message middleware, that is, the MQTT Broker, which is responsible for message routing, session management, and QoS guarantee to realize the data transmission between the publisher and the subscriber. The MQTT protocol (Message Queuing Telemetry Transport) is a lightweight message middleware protocol designed for low-bandwidth, high-latency, and unreliable network environments. It defines three levels of service quality (QoS) levels: 0, 1, 2; the three levels respectively correspond to at most once, at least once, and exactly once in terms of transmission semantics, network overhead, and typical application scenarios; lowest, medium, and highest; non-critical data, sensor data that requires reliable transmission, payment instructions, and critical control signals.

[0107] Therefore, the inspection method of the industrial inspection system based on multi-modal fusion of the present invention is adopted. At the positioning layer, the non-line-of-sight error compensation technology of hybrid double-sided two-way ranging (HDS-TWR) and Taylor full centroid algorithm (Taylor-FCL) is adopted, which improves the positioning accuracy by 15%; through the state vector splitting and covariance scalarization optimization of Kalman filtering, the calculation time consumption and memory occupation are reduced, and the real-time performance of the embedded platform is improved; combined with the sliding window extreme value removal filtering, the ranging stability is significantly improved. In the tracking layer, by fusing the infrared sensor and the visual threshold segmentation technology, a time-synchronized displacement compensation mechanism is constructed, which greatly improves the vehicle body stability and the tracking robustness of the system. The visual detection module is based on the improved YOLOv8 model, and adopts the dynamic positive sample allocation strategy and the CIOU loss function, so that the mAP@0.5 of head and helmet detection reaches 93.5% and 94.7% respectively, and still maintains high robustness in complex occlusion scenarios.

[0108] The technical solution of the present invention will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 is a flowchart of the inspection method of the industrial inspection system based on multi-modal fusion of the present invention;

[0110] Figure 2 is a plan view of the experimental site layout of the embodiment of the present invention;

[0111] Figure 3 is a visual tracking test image of the embodiment of the present invention, where (a), (b), (c), and (d) are all images collected by the camera during the tracking process;

[0112] Figure 4 is a schematic diagram of the P-R curve of the embodiment of the present invention;

[0113] Figure 5 is a confusion matrix diagram of the model training results of the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0114] The following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0115] Please refer to Figure 1 , the inspection method of the industrial inspection system based on multi-modal fusion, and the specific experimental process is as follows:

[0116] In the UWB and IMU fusion positioning test, pure UWB positioning technology is used to fuse with IMU data for positioning, guiding the trolley to automatically cruise along the preset coordinate path. The path is configured by setting target coordinate points at interval distances. After the trolley reaches the target point, it automatically adjusts the head direction to align with the next target point and then continues to move. The Kalman filtering algorithm for UWB is optimized, and the optimized filtering algorithm is used to process the UWB positioning data to improve the accuracy and stability of positioning. The accuracy test of the tracking module is to evaluate the accuracy performance of the tracking module during the path tracking process and detect whether it can accurately identify and follow the path. The visual tracking accuracy test focuses on the performance of the visual system in the tracking task and analyzes its recognition accuracy and reliability of the path. For the visual tracking, the tracking module and the UWB fusion positioning tracking test, the visual tracking is fused with the tracking module for tracking, and the UWB is fused with the IMU for positioning to obtain the fused performance image.

[0117] Due to site limitations, a 3.5m×3.5m positioning experimental area is set up in the laboratory, as Figure 2 shown. The true coordinates of 3 UWB base stations are measured, which are A(0m,0m), B(0m,3.2m), and C(3.2m,3.2m) respectively. Then, 4 points are selected, and the coordinates are a(0.70m,0.90m), b(0.70m,2.40m), c(2.40m,2.40m), and d(2.40m,0.90m).

[0118] 1. Positioning and tracking test

[0119] During the experiment, an intelligent trolley equipped with an STM32F103 main board and 1 UWB positioning tag moves along the preset trajectory. The ranging distances between the trolley and the 3 base stations A, B, and C are read through the main base station A at the upper computer. The movement sequence is a→b→c→d→a. When the trolley reaches a target point, the system automatically triggers the adjustment mechanism, and the six-axis sensor data is used to monitor the head direction of the trolley in real time, and it continues to drive by automatically adjusting the head.

[0120] In this experiment, by adding a tracking module, a combination of multiple sensors such as an OpenMV camera, the trolley is enabled to move along the laid trajectory. The trajectory information of the trolley is obtained through the OpenMV camera and the TCRT5000 module, and the data is transmitted to the STM32F103 main board through the serial port. After the main board fuses and analyzes the data, it controls the trolley to execute the corresponding movement. When the camera and the TCRT5000 module detect a line deviation, the corresponding data flag bit changes. After the host analyzes the data, it controls the trolley to re-adjust the running state of the trolley. This makes the line tracking process smoother and more accurate than a single module.

[0121] During the process of tracking data acquisition, the accuracy of UWB and IMU positioning will directly affect the accuracy of the tracking data. Therefore, we conducted independent experiments on the automatic cruise algorithm for UWB positioning and used it as a reference for comparing subsequent experimental data to verify the effectiveness and perfection of the comprehensive experiment.

[0122] During the UWB automatic cruise experiment, since the trolley runs against a single target each time and the number of direction adjustments is small, the running trajectory of the trolley is relatively smooth, the heading angle fluctuates less and there are few sudden changes, and the data is relatively stable. However, the filtering algorithm was not optimized during positioning, resulting in large noise interference, large fluctuations between the positioned test path and the target path, and even large-scale runaway situations.

[0123] After optimizing the Kalman filter, the noise interference has been significantly improved. The distance data measured between each base station and the tag changes more stably, and the measured movement path of the trolley fits the target path better.

[0124] For the accuracy test of the tracking module, a single tracking module is used. The running state is changed through the path data obtained by the tracking module and it moves along the arranged line. During the experiment, when the single tracking module obtains data, the body jitter of the trolley will be affected by the performance differences of the motors and the distance differences between the modules. During the running of the trolley, the body jitter of the trolley is relatively large, the heading angle data changes greatly, and there are many data spikes.

[0125] The visual tracking accuracy test adopts the visual tracking method. Visual tracking is greatly affected by the light intensity. As Figure 3 shown, five region blocks A, B, C, D, and E are divided in the image captured by the camera. Among them, each region block has two states, 00 and 01. As shown in Table 1, state 00 means that no trajectory is detected in the block, and state 01 means that a trajectory is detected in the block. Figures (a), (b), (c), and (d) in the figure represent the images captured by the camera during the tracking process. When a trajectory is detected in one of the region blocks, the corresponding data bit in the table becomes 0x01.

[0126] Table 1 corresponds to the test data

[0127] Figure A B C D E (a) 00 00 01 00 00 (b) 00 00 01 00 00 (c) 00 00 01 00 00 (d) 00 01 00 00 00

[0128] One of the important factors affecting visual tracking is light. The infrared-visual dual-mode collaborative tracking method used in this experiment can better adapt to the ambient light and has a good processing effect on color misjudgment and light interference.

[0129] In this experiment, the IMU measures the heading angle data at a frequency of 20 times / s. At this time, the data fluctuates greatly and there are many data mutations. This is because if the change amplitude is large during a certain measurement process, but the number of measurements is small, it will lead to a large slope and form burrs.

[0130] In this experiment, it is impossible to make the car move to the designated location to perform other operations by only relying on visual tracking and other methods. Therefore, after integrating UWB technology, the movement route of the car can be set. After optimizing the Kalman filter technology, the measured position of the car is more in line with the target path. Although there are still cases of data deviation, the positioning data is more consistent and the car can expand more functions.

[0131] The accuracy of single visual tracking or tracking module in line patrol is poor. In this experiment, we integrated the OpenMV camera and the tracking module. The camera observed the front section, and the tracking module monitored the back section. After acquiring the data, the host analyzed it. When the data monitored by the camera changed, the data transmitted by the camera was temporarily stored in the register. After a certain delay, when the tracking module arrived at the position, the tracking status data was read. At this time, the host compared the two data. If the situation is the same, the corresponding movement is executed. If not, the original movement state is continued.

[0132] This method greatly increases the accuracy of the car's line patrol. The body vibration caused by changing direction during line patrol is greatly reduced, which improves the accuracy of line patrol.

[0133] In this experiment, the frequency of IMU measuring heading angle was increased to 50 times / s, which enables multiple data to be measured in a short time. The heading angle data graph is smoother and the data increase or decrease is smoother by reducing the difference between each adjacent data, thereby reducing the burrs.

[0134] We also optimized the parameters of the Kalman filter again, making the measured vehicle trajectory information closer to the target path and reducing noise interference.

[0135] 2 Data Transmission

[0136] In this experiment, the car collected a variety of key data, including temperature and humidity, light intensity, car heading angle, and x and y coordinates of the car's position. The specific data description is shown in Table 2:

[0137] Table 2 Description of collected data

[0138] Data name Data description Temp Collected temperature data Humi Collected humidity data Temp Threshold Set temperature threshold Illuminance Collected illuminance data Yaw Car heading angle data X Coordinate X coordinate of the car position Y Coordinate Y coordinate of the car position ID card Card number of the current card swiper Temp Warning Show whether the temperature exceeds the threshold Intnet Show whether the car is connected to the network Permission Show the permission of the current ID card

[0139] The trolley intuitively displays the collected data and relevant information on its own screen. Through the built-in WiFi module, the trolley uses the MQTT protocol and publishes some important data accurately to the EMQX broker according to the corresponding topics, realizing the initial transmission of data. This process not only ensures the real-time nature of the data, but also improves the efficiency and reliability of data transmission.

[0140] During the data transmission process, the backend server also uses the MQTT protocol to subscribe to relevant topics and efficiently obtain the data sent by the trolley from the EMQX broker. After obtaining the data, the server not only prints and displays it on the console for real-time monitoring, but also updates and stores the data in the database, providing a solid foundation for subsequent data analysis and applications.

[0141] In this experiment, Postman is used to simulate the front-end access to the server, and control instructions are sent to the server using Postman to control the running mode or specific movement of the trolley. After receiving the data sent by the front-end, the server quickly publishes these data to the EMQX broker with corresponding topics so that the trolley can receive and execute the corresponding actions.

[0142] Mode 1: Automatic cruise mode

[0143] The trolley automatically moves along the established trajectory, obtains sensor data during operation and uploads it to the server.

[0144] Table 3 Trolley tracking mode

[0145]

[0146] The mode of controlling the trolley is the automatic cruise mode. In the automatic cruise mode, the trolley can independently execute basic actions such as "forward", "backward", "left turn", "right turn", etc., without real-time manual intervention, greatly improving the operation efficiency and automation level. This mode is applicable to application scenarios with relatively stable environments and fixed paths, which can effectively reduce the manual operation cost and improve the work efficiency.

[0147] Mode 2: Administrator control mode

[0148] The administrator can manually control the movement of the trolley and obtain the trolley status through the backend server. The control mode for the trolley is the administrator control mode. The administrator control mode gives the administrator greater freedom of operation. In this mode, the administrator can obtain the current status of the trolley in real time through the backend server, including key information such as position, speed, and sensor data. At the same time, the administrator can manually control the movement of the trolley according to actual needs, and the operation interfaces for controlling the trolley to "move forward", "move backward", "turn left", and "turn right" are respectively shown. This mode enables the trolley to flexibly adjust its movement state according to the administrator's instructions in complex or special environments, enhancing the adaptability and controllability of the trolley. The administrator control mode is applicable to scenarios where the path needs to be frequently adjusted or emergencies need to be handled, and can ensure the stable operation of the trolley in complex environments.

[0149] 3. Visual Detection

[0150] The hardware environment of this experiment uses a 64-bit operating system of Windows 10, adopts the PyTorch 1.12.0 deep learning framework, the CUDA version is 12.3, and the graphics card used in the experimental model is NVIDIA GeForce RTX 3060 Laptop GPU. Before model training, the input image size of the model is 640×640, the initial learning rate is set to 0.01, the decay coefficient is set to 0.0005, the Batch size is 8, and a total of 100 epochs are trained. The detailed training parameter settings are shown in Table 4.

[0151] Table 4 Training Parameter Settings

[0152] Parameter Setting Parameter Setting Initial learning rate 0.001 Final learning rate 0.001 Number of training epochs 100 Number of warm-up epochs 10 Number of threads 1 Number of images per batch 8 Optimizer SGD + AdamW Weight decay 0.0005

[0153] The initial learning rate and the final learning rate are set to 0.001 respectively. Setting a smaller initial learning rate can help the model learn stably in the initial stage of training, avoid missing the optimal solution due to large update steps at the beginning of training, contribute to the model converging to a better local minimum, and avoid unstable training. The number of training epochs is 100, allowing the model to learn over a sufficient number of epochs to fully capture the characteristics of the data. The training combines the advantages of Stochastic Gradient Descent (SGD) and the AdamW optimizer. SGD helps the model escape from local minima, while AdamW provides an adaptive learning rate and weight decay. This combination can provide a faster convergence rate and better generalization ability. The weight decay is set to 0.0005 for regularization to prevent the model from overfitting. The number of warm-up epochs is set to 10. Gradually increasing the learning rate to the initial value in the warm-up stage helps the model start training stably in the early stage and avoid drastic fluctuations in the initial stage. The number of images per batch is 8. The model in this paper is trained using an NVIDIA GeForce RTX 3060 Laptop GPU. Considering the hardware resource limitations, setting it to 8 helps improve the model training efficiency. The number of threads is 1. In the case of limited resources, using a single thread can avoid resource competition and simplify the debugging process.

[0154] The following metrics are used in the research method of this study: Precision (P), Recall (R), F1-Score (F1), mean average precision (mAP), and Frames Per Second (FPS) to evaluate the detection accuracy, classification performance, and detection speed of the model.

[0155] In object detection, mAP is a commonly used evaluation metric to measure the detection performance of the model on different classes. mAP@0.5 represents the average precision of the class calculated when the Intersection over Union (IoU) threshold is 0.5. The calculation formula of mAP is as follows:

[0156]

[0157] AP=∫0 1 PdR;

[0158]

[0159] Among them, P refers to the proportion of real targets in the bounding boxes detected by the model; R refers to the proportion of the number of real targets successfully detected by the model to the total number of all real targets; True Positive (TP) is the number of bounding boxes containing real targets correctly predicted by the model; False Positive (FP) is the number of bounding boxes containing non-real targets wrongly predicted by the model; False Negative (FN) is the number of real targets not detected by the model; F1 score is the harmonic mean of precision and recall, which is used to comprehensively evaluate the performance of the model; AP is the area under the P-R curve, representing the average detection precision of the model for this category; mAP is a comprehensive evaluation index, which combines the precision and recall of the model for different categories and averages the precision for different categories; AP i is the average precision of the i-th category, and e represents the total number of detected categories.

[0160] 4. Results

[0161] (1) Training Results of YOLOv8n Model

[0162] In the task of performing head object detection, its average precision (AP) reached 93.5%. Specifically, for the "head" category, the average precision (mAP@0.5) of the model at a 0.5 IOU threshold is 0.935, which fully proves the reliability of the model in identifying head objects. Similarly, for the task of detecting whether a helmet is worn, the average precision of the model reached 94.7%, showing its high accuracy for the "helmet" category. At the same 0.5 IOU threshold, the average precision (mAP@0.5) for the helmet category is 0.947. In summary, the YOLOv8n model has demonstrated high-level performance in both head and helmet detection tasks, and its precision and reliability have outstanding advantages among similar models, as Figure 4 shown.

[0163] Figure 5 is the confusion matrix diagram of the training results of the YOLOv8n model, which is used to evaluate the performance of the classification model. It shows the prediction results of the classifier for four categories ("person", "head", "helmet" and "background"). The accuracy rate of "person" is 100%, "head" is 92% (0.92), "helmet" is 91% (0.91), and "background" is 96% (0.96).

[0164] (2) Video Detection

[0165] Video detection includes video file detection and real-time camera detection. Its core function is to achieve real-time detection of targets with the help of a camera. This processing process is carried out frame by frame, that is, each frame image in the video is carefully detected to ensure that no moment that may contain a target is missed, thus ensuring the comprehensiveness and accuracy of the detection. As shown in the figure below, the helmet recognition situation on the system interface will clearly mark the detected targets on the video frame through specific identifiers (such as bounding boxes, detection confidence, etc.), and continuously update with the frame-by-frame detection. At the same time, the detection results will be presented in the form of data in the lower left corner of the interface. In this way, users can intuitively see the detection results without having to interpret complex data.

[0166] (3) Image detection

[0167] In the image detection module, users can select image files by themselves. After the system performs image preprocessing and displays the image, model prediction is then applied. The interface will intuitively display the image detection results in a comparative form. The original image file selected by the user is on the left side of the interface, and the image file marked through model prediction is shown on the right side of the interface. At the same time, the statistical information of the detection results, such as the number of helmet targets detected, is presented in the form of data in the lower left corner of the interface, providing comprehensive analysis feedback for users.

[0168] Therefore, the present invention adopts the above-mentioned inspection method of the industrial inspection system based on multi-modal fusion. In the positioning layer, the non-line-of-sight error compensation technology of hybrid double-sided two-way ranging (HDS-TWR) and Taylor full centroid algorithm (Taylor-FCL) is adopted, which improves the positioning accuracy by 15%; through the state vector splitting and covariance scalarization optimization of Kalman filtering, the calculation time consumption and memory occupation are reduced, and the real-time performance of the embedded platform is improved; combined with the sliding window extreme value removal filter, the ranging stability is significantly improved. In the tracking layer, by fusing the infrared sensor and the visual threshold segmentation technology, a time-synchronized displacement compensation mechanism is constructed, which greatly improves the vehicle body stability and the tracking robustness of the system. The visual detection module is based on the improved YOLOv8 model, adopting the dynamic positive sample allocation strategy and the CIOU loss function, so that the mAP@0.5 of head and helmet detection reaches 93.5% and 94.7% respectively, and still maintains high robustness in complex occlusion scenarios.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. The inspection method of an industrial inspection system based on multimodal fusion, characterized in that, It includes the following steps: S1. After obtaining the ranging data from the UWB base station to the node to be located using the HDS-TWR ranging algorithm, use the Taylor full centroid algorithm to calculate the positioning data of the node to be located and output the positioning coordinates; S2. Use sensors to measure the acceleration and angular velocity of the inspection system to evaluate the alignment of movement and posture; S3. Improve the Kalman filter algorithm to process the data in steps S1 and S2, making the output prediction result close to the true value; S4. Adopt infrared-vision dual-mode collaborative tracking, and at the same time use the time synchronization displacement compensation mechanism to calculate the relative displacement difference between the two tracking modes for dynamic calibration; S5. Use the YOLOv8 model to process the video frames or pictures collected during the inspection, output the target box and class probability, and finally obtain the inspection result.

2. The inspection method of the industrial inspection system based on multimodal fusion according to claim 1, characterized in that, The HDS-TWR ranging algorithm in step S1 is as follows: Among them, T round1A , T round1B , T round1C respectively represent the round-trip time for the tag to send Poll messages to base stations A, B, and C until receiving the first response messages RespA, RespB, and RespC from the corresponding base stations; T round2A , T round2B , T round2C are the time intervals from the tag receiving the first response message from the base station to sending the Final message; T reply1A , T reply1B , T reply1C are the times for base stations A, B, and C to send response messages RespA, RespB, and RespC after receiving the Poll message; T reply2A , T reply2B , T reply2C are the times from the tag sending the Final message to base stations A, B, and C receiving the Final message; T propA , T propB , T propC respectively represent the propagation times of the signals between the tag and base stations A, B, and C.

3. The inspection method of the industrial inspection system based on multi-modal fusion according to claim 2, characterized in that, The specific process of using the Taylor full centroid algorithm to calculate the positioning coordinates in step S1 is as follows: Assume that the coordinates of base stations A, B, C and the tag are (x1, y1), (x2, y2), (x3, y3) and (x, y) respectively, and the ranging distances from each positioning base station to the node to be located are r1, r2, r3. The following equations can be obtained: Solve using the full centroid algorithm and rewrite it in matrix form: Furthermore, we get: Combining equations (5) and (6), we define to obtain the following system of linear equations: aθ = b (7) Solve θ using the least squares method to get: θ = (a T a) -1 a T b (8) Thus, the initial coordinate values (x k , y k ) of the object to be labeled are obtained, and the actual coordinates satisfy: Where, Δx and Δy are the differences between the estimated coordinates and the true coordinates; As can be seen from formula (4), the distances measured by the base stations participating in the positioning ideally satisfy the formula: where (x i , y i ) are the coordinates of the i-th base station, and r i is the distance from the i-th base station to the positioning node; by performing a first-order Taylor expansion on (10) to obtain a system of linear equations for solution, for the binary Taylor expansion at (x k ,, y k ), we have: r i = f(x,y) ≈ f(x,y) + (x - x k )f' x (x k ,y k ) + (y - y k )f' x (x k ,y k ) (11) Substitute f(x, y) into formula (10) to get: Combined with formula (9), calculate Δx and Δy through the least squares method, where: When |Δx| + |Δy| is less than the preset convergence threshold or the number of iterations exceeds the maximum allowable value, the algorithm terminates the iteration and outputs the positioning coordinates.

4. The inspection method of the industrial inspection system based on multimodal fusion according to claim 3, characterized in that, In step S2, the MPU6050 sensor is used to measure data, and the DMP library automatically fuses the sensor data and outputs the quaternion for pose calculation. Specifically: Suppose the quaternion outputs of multiple sensors are Q1, Q2, Q3... Q n , and the final quaternion is obtained using the weighted average method: where q i is the weight coefficient, and Q i is the quaternion output by the i-th sensor; The data obtained by the MPU6050 sensor includes acceleration a = (a x , a y , a z ), angular velocity ω = (ω x , ω y , ω z ), pitch angle and roll angle θ, where: Among them, the pitch angle and roll angle are used to calculate and update the quaternion:

5. The inspection method of the industrial inspection system based on multimodal fusion according to claim 4, characterized in that, In step S3, the state value is split into x and y, and two Kalman filters are performed separately. The filtering method is divided into two steps: prediction and update: Prediction: X k = X k-1 (18) P k = P k-1 + Q (19) Update: X k = X k + K k (z k - X k ) (21) P k = (I - K k )P k (22) Among them, X k is the state value at time k; P k is the covariance matrix of state estimation; Q is the process noise covariance matrix; K k is the Kalman gain calculated at time k; R is the measurement noise covariance matrix; z k is the observed value at time k; I is the identity matrix.

6. The inspection method of the industrial inspection system based on multi-modal fusion according to claim 5, characterized in that, The infrared-vision dual-mode collaborative tracking in step S4 includes vision tracking based on the OpenMV camera and an infrared reflection tracking module. Among them, vision tracking identifies specific path markers through color threshold segmentation and multi-ROI detection, and sends the detection results to the main controller through the serial port. Specifically: Definition of ROI region coordinates: Assume the image resolution is W×H = 160×120, and 5 ROI regions evenly distributed horizontally are defined. The coordinate formula of the i-th ROI is: ROI i =(x i , y, w, h) (23) x i = x0 + i×(spacing + w), i ∈ (0, 1, 2, 3, 4) (24) Among them, ROI i Each value corresponds to the x coordinate of the upper left corner point of the box selection position as the ROI, the y coordinate of the upper left corner point of the ROI, the width of the ROI, and the height of the ROI; Color threshold segmentation: Define the color threshold range as: Threshold = [L min , L max , A min , A max , B min , B max , where each value corresponds to the minimum and maximum values of luminance, the minimum and maximum values of the A channel, and the minimum and maximum values of the B channel; Detection result encoding: Each ROI detection result f i is defined as a binary variable: If there is a target color block f within the ROI i = 1, otherwise f i = 0; The final 5 detection flags sent are f1, f2, f3, f4, f5; Serial port data frame structure: The data frame format is 8-byte little-endian mode: Data = [0xA5, 0xA6, f1, f2, f3, f4, f5, 0x5B].

7. The inspection method of the industrial inspection system based on multi-modal fusion according to claim 6, characterized in that, The infrared reflection tracking module specifically includes: Infrared reflection intensity model: When an infrared light-emitting diode emits infrared light with an intensity of I0, after being reflected by the target surface, the intensity of the light detected by the receiving tube is I r It is expressed as: Among them, K is the optical system efficiency; r is the target surface reflectivity; d is the distance between the sensor and the target; μ is the medium attenuation coefficient; Signal threshold determination: received signal voltage V and light intensity I r are linearly related: V = α × I r + β (26) Among them, α is the optoelectronic conversion coefficient, and β is the environmental noise; Set the threshold value V th , and output a digital signal:

8. The inspection method of the industrial inspection system based on multimodal fusion according to claim 7, characterized in that, The time synchronization displacement compensation mechanism is specifically as follows: Assume that the camera and the tracking module detect a certain point P at times t and t+Δt respectively; Establish the spatio-temporal observation equation of the camera and the tracking module. The observation position of the camera at time t is: x c (t) = x0 + vt (27) Among them, x0 is the initial position; The position of the monitoring point of the tracking module: x t (t + Δt) = x0 + v(t + Δt) (28) By differentiating the time difference Δt, calculate the relative displacement difference between the two monitoring points P: Δx = x t (t + Δt) - x c (t) = vΔt (29) Characterize the spatial offset caused by time asynchrony, compare the relative displacement difference Δx with the preset physical distance L. If Δx > L, it indicates that the spatio-temporal consistency is violated and the offset adjustment mechanism is triggered; where x t (t + Δt) is the position of point P monitored by the tracking module at time t + Δt; v is the speed of the vehicle; L is the distance between the camera and the tracking module.

9. The inspection method of the industrial inspection system based on multimodal fusion according to claim 8, characterized in that, Step S5 specifically includes: The structure of the YOLOv8 model includes: The CSPDarknet backbone network, the formula is: The SPPF module, which serially pools and fuses multi-scale features: X out = MaxPool(X in ) ⊕ MaxPool 2 (X in ) ⊕ MaxPool 3 (X in ) ⊕ X in (31) Among them, X in is the input feature map; X out is the output feature map; n is the number of channels of the input feature map; Conv is the convolution operation; ⊕ is feature concatenation; MaxPool k is the max pooling operation with a window size of k; The YOLOv8 model uses DEL to optimize the bounding box for prediction. The DEL formula is: Model the bounding box coordinates as a discrete probability distribution, predicting the probabilities P = [p0, p1, ..., p n of n + 1 intervals, and calculate the coordinate values through expectation: The DFL loss encourages high probabilities in the interval near the true coordinates: DFL(p,t0) = -((t i+1 - t) log(p i ) + (t - t i ) log(p i+1 )) (33) Among them, is the predicted coordinate value calculated by the probability distribution, y i is the central value of the i-th discrete interval, t0 is the true bounding box coordinate value, t i , t i+1 are the endpoints of the discrete interval where the true coordinate t0 is located, and satisfy t i ≤ t ≤ t i+1 ; Improve the regression accuracy through CIOULoss, the formula is: Among them, p(b,b gt ) is the Euclidean distance between the center points of the predicted box and the ground-truth box; c is the diagonal length of the minimum bounding rectangle of the predicted box and the ground-truth box; w and h are the width and height of the predicted box; w gt , h gt are the width and height of the ground-truth box; τ is the aspect ratio consistency penalty term; γ is the dynamic weight; Select positive samples through the task alignment metric to improve the consistency between classification and regression: align_metric=(p c ·IOU(b,b gt )) δ (35) Select the top k anchors with the highest align_metric corresponding to each true box as positive samples; Among them, p c is the predicted class probability, IOU(b, b gt ) is the intersection over union of the predicted bounding box and the ground truth bounding box, δ is a hyperparameter, and k is the number of positive samples selected for each ground truth bounding box.

Images