Inspection method of industrial inspection system based on multi-modal fusion

Abstract

The application discloses a multi-modal fusion-based industrial inspection system inspection method, and belongs to the technical field of industrial inspection, comprising the following steps: S1, obtaining positioning data of a to-be-positioned node by using an HDS-TWR ranging algorithm and a Taylor total centroid algorithm, and outputting positioning coordinates; S2, measuring the acceleration and angular velocity of the inspection system by using a sensor, so as to evaluate the alignment movement and posture; S3, improving a Kalman filtering algorithm, so that the output prediction result is close to the true value; S4, adopting infrared-vision dual-mode cooperative tracking, and simultaneously using a time synchronization displacement compensation mechanism to calculate the relative displacement difference of the two tracking modes and perform dynamic calibration; and S5, processing video frames or pictures collected in the inspection process by using a YOLOv8 model, outputting a target frame and a category probability, and finally obtaining an inspection result; the multi-modal fusion-based industrial inspection system inspection method provided by the application provides a high-precision and low-delay automatic inspection solution for mines, nuclear power plants and other harsh environments.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial inspection, and in particular to an inspection method of an industrial inspection system based on multi-modal fusion. BACKGROUND

[0002] With the rapid development of industrial intelligence and the increasing demand for safety monitoring, the applicability of traditional inspection methods in complex indoor environments and narrow dangerous areas (such as mines, nuclear power plants, chemical plants, etc.) has been greatly challenged. Traditional inspection methods rely on manual operation, which not only has high labor intensity and high cost, but also has high safety risks in high-risk environments. Therefore, automatic inspection positioning systems have gradually become a research hotspot to improve inspection efficiency, reduce cost, and enhance adaptability and safety in complex environments.

[0003] In recent years, multi-sensor fusion technology (Multi-Sensor Fusion, MSF) has been widely applied in intelligent robots, autonomous navigation, and environmental perception. This technology combines multiple sensors (such as ultra-wideband UWB, inertial measurement unit IMU, infrared sensor, vision sensor, etc.) to obtain environmental information, in order to overcome the limitations of single sensors in terms of measurement accuracy, environmental adaptability, etc. Among them, UWB positioning technology has become a high-precision positioning solution in complex environments due to its high precision, strong anti-interference ability, and high stability. However, UWB may experience multipath effects and signal attenuation in metal-intensive environments or scenes with many signal blockages, leading to increased positioning errors. How to achieve high-precision and high-robustness autonomous navigation and environmental perception has become a key technical bottleneck in the development of industrial intelligence.

[0004] To solve the above problems, the present application proposes an automatic inspection positioning system based on multi-sensor fusion, which combines UWB high-precision positioning, visual environmental perception, and inertial measurement unit (IMU) dynamic compensation technology, aiming to solve the problem of autonomous navigation and task execution in complex closed environments. The system uses HDS-TWR ranging algorithm and full centroid-Taylor joint solution to optimize the dynamic positioning robustness under multi-base station cooperation and reduce errors caused by NLOS interference. In practical applications, the accuracy and real-time performance of the inspection system are crucial. To optimize positioning performance, the present application uses an improved Kalman filter algorithm to reduce UWB ranging errors, and uses a deep learning target detection model (YOLOv8) for visual inspection. In addition, the present application uses the MQTT communication protocol for data transmission to ensure low power consumption and efficient communication. SUMMARY

[0005] The present application aims to provide an inspection method of an industrial inspection system based on multi-modal fusion to solve the problems in the background art.

[0006] To achieve the above object, the application provides a multi-modal fusion-based industrial inspection system inspection method, comprising the following steps:

[0007] S1, after obtaining the ranging data from the UWB base station to the node to be positioned by using the HDS-TWR ranging algorithm, the Taylor total centroid algorithm is used to solve the positioning data of the node to be positioned, and the positioning coordinates are output;

[0008] S2, the acceleration and angular velocity of the inspection system are measured by using a sensor, so as to evaluate the motion and posture;

[0009] S3, the data in steps S1 and S2 are processed by improving the Kalman filtering algorithm, so that the output prediction result is close to the true value;

[0010] S4, infrared-vision dual-mode cooperative tracking is adopted, and a time synchronization displacement compensation mechanism is used to calculate the relative displacement difference of the two tracking modes for dynamic calibration;

[0011] S5, the YOLOv8 model is used to process the video frames or pictures collected in the inspection process, and the target frame and class probability are output, and finally the inspection result is obtained.

[0012] Preferably, the HDS-TWR ranging algorithm in step S1 is obtained according to its ranging principle:

[0013]

[0014] The arrangement is obtained:

[0015]

[0016] Wherein, T round1A , T round1B , T round1C respectively represent the round trip time from when the tag sends the Poll message to the base station A, B and C to when the first response message RespA, RespB and RespC is received; T round2A , T round2B , T round2C are the time intervals from when the first response message is received from the tag to when the Final message is sent; T reply1A , T reply1B , T reply1C are the times when the base stations A, B and C send the RespA, RespB and RespC response messages after receiving the Poll message; T reply2A , T reply2B , T reply2C are the times from when the tag sends the Final message to when the base stations A, B and C receive the Final message; T propA , T propB , TpropC Represent the signal propagation time between the tag and base stations A, B, and C respectively.

[0017] Preferably, in step S1, the Taylor full centroid algorithm is used to calculate the positioning coordinates as follows:

[0018] Assume that the coordinates of base stations A, B, C and the tag are (x1, y1), (x2, y2), (x3, y3) and (x, y) respectively, and the ranging distances from each positioning base station to the node to be located are r1, r2, and r3 respectively, and the following set of equations is obtained:

[0019]

[0020] Solve using the full centroid algorithm and rewrite it into matrix form:

[0021]

[0022] Further, we get:

[0023]

[0024] Combining formulas (5) and (6), we define The following linear equations are obtained:

[0025] aθ=b (7)

[0026] The least squares method is used to solve θ and we get:

[0027] θ=(a T a) -1 a T b (8)

[0028] Using the Taylor algorithm, we first make a relatively accurate initial estimate of the coordinates of the tag to be tested, and then iterate multiple times under the premise of the minimum quadratic error criterion until the preset threshold accuracy or number of iterations is reached. k ,y k ), the actual coordinates satisfy:

[0029]

[0030] Among them, Δx and Δy are the differences between the estimated coordinates and the true coordinates;

[0031] From formula (4), we can see that the distance between the base station and the tag measured by the positioning base station ideally satisfies the formula:

[0032]

[0033] Among them, (x i ,yi ) is the coordinate of the i-th base station, r i is the distance from the i-th base station to the positioning node; by making a Taylor one expansion to (10), a linear equation group is solved, for binary Taylor expansion at (x k ,,y k ), there is:

[0034] r i = f(x,y)≈f(x,y)+(x-x k )f' x (x k ,y k )+(y-y k )f' x (x k ,y k ) (11)

[0035] Taking f(x,y) into formula (10), we get:

[0036]

[0037] Combined with formula (9), Δx, Δy are calculated by least square method, wherein:

[0038]

[0039] When |Δx|+|Δy| is less than a preset convergence threshold or the number of iterations exceeds a maximum allowed value, the algorithm terminates iteration and outputs the positioning coordinates.

[0040] To improve the stability of ranging data, a sliding window extreme value mean algorithm is used to process the distance observation value: a ring buffer queue with a length of 3 is established to store the latest three ranging results of the base station and the tag in real time, extreme value detection is performed on the data in the window every time the data is updated, the maximum value and the minimum value are removed, and then the arithmetic mean operation is performed on the remaining valid observation values, and finally the filtered ranging value is output as the input of the positioning calculation. The composite algorithm effectively suppresses the interference of outliers through a double fault-tolerant mechanism, while ensuring the calculation efficiency and improving the robustness of the positioning system.

[0041] Preferably, step S2 uses MPU6050 sensor to measure data, DMP library automatically fuses sensor data, outputs quaternion for pose calculation, specifically:

[0042] Suppose the quaternion outputs of multiple sensors are Q1, Q2, Q3...Q n The final quaternion is obtained by weighted average method:

[0043]

[0044] Wherein, qi is the weight coefficient, Q i is the quaternion output by the i-th sensor;

[0045] The data obtained by the MPU6050 sensor includes acceleration a=(a x ,a y ,a z ), angular velocity ω=(ω x ,ω y ,ω z ), pitch angle and the roll angle θ, where:

[0046]

[0047] Among them, the pitch angle and roll angle are used to calculate the updated quaternion:

[0048]

[0049] Preferably, step S3 splits the state value into x and y, and performs two Kalman filters on each of them. The filtering method is divided into two steps: prediction and update:

[0050] predict:

[0051] X k =X k-1 (18)

[0052] P k =P k-1 +Q (19)

[0053] renew:

[0054]

[0055] X k =X k +K k (z k -X k ) (twenty one)

[0056] P k =(IK k )P k (twenty two)

[0057] Among them, X k is the state value at time k; P k is the covariance matrix of state estimation; Q is the covariance matrix of process noise; K k is the Kalman gain calculated at time k; R is the measurement noise covariance matrix; z k is the observation value at time k; I is the unit matrix.

[0058] Preferably, the infrared-visual dual-mode collaborative tracking in step S4 includes a visual tracking module based on the OpenMV camera and an infrared reflective tracking module, wherein the visual tracking identifies specific path markers through color threshold segmentation and multi-ROI detection, and sends the detection results to the main controller through the serial port, specifically:

[0059] ROI region coordinate definition: Assume that the image resolution is W×H=160×120, and define five ROI regions evenly distributed horizontally. The coordinate formula of the i-th ROI is:

[0060] ROI i =(x i ,y,w,h) (23)

[0061] x i =x0+i×(spacing+w),i∈(0,1,2,3,4) (24)

[0062] Among them, ROI i The selection position of each value corresponding to the box is the x-coordinate of the upper left corner of the ROI, the y-coordinate of the upper left corner of the ROI, the width of the ROI, and the height of the ROI;

[0063] Color threshold segmentation: Define the color threshold range as: Threshold = [L min ,L max ,A min ,A max ,B min ,B max ], where each value corresponds to the minimum and maximum brightness, the minimum and maximum values ​​of channel A, and the minimum and maximum values ​​of channel B;

[0064] Detection result encoding: Each ROI detection result f i Defined as a binary variable:

[0065] If there is a target color block f in the ROI i =1, otherwise f i =0;

[0066] The five detection flags finally sent are f1, f2, f3, f4, and f5;

[0067] Serial port data frame structure: The data frame format is 8-byte little-endian mode: Data = [0xA5, 0xA6, f1, f2, f3, f4, f5, 0x5B].

[0068] Preferably, the infrared reflective tracking module specifically includes:

[0069] Infrared reflection intensity model: when the infrared emission tube emits infrared light with intensity I0, after reflection on the target surface, the light intensity I detected by the receiving tube is r is expressed as:

[0070]

[0071] wherein K is the optical system efficiency; r is the target surface reflectivity; d is the distance between the sensor and the target; μ is the medium attenuation coefficient;

[0072] Signal threshold determination: the receiving signal voltage V and the light intensity I r are in linear relationship:

[0073] V = α × I r + β (26)

[0074] wherein α is the photoelectric conversion coefficient, and β is the environmental noise;

[0075] Set the threshold value V th , and output the digital signal:

[0076] Preferably, the time synchronization displacement compensation mechanism is specifically:

[0077] Suppose that the camera and the tracking module monitor a certain point P at times t and t+Δt, respectively;

[0078] Establish the space-time observation equation of the camera and the tracking module. The observation position of the camera at time t is:

[0079] x c (t) = x0 + vt (27)

[0080] wherein x0 is the initial position;

[0081] The position of the point monitored by the tracking module is:

[0082] x t (t+Δt) = x0 + v(t+Δt) (28)

[0083] By differentiating the time difference Δt, the relative displacement difference of the two monitoring points P is calculated:

[0084] Δx = x t (t+Δt) - x c (t) = vΔt (29)

[0085] Characterize the spatial offset caused by time asynchrony. Compare the relative displacement difference Δx with the preset physical interval L. If Δx>L, it indicates that the space-time consistency is destroyed, and the offset adjustment mechanism is triggered; wherein x t(t+Δt) is the position of point P monitored by the tracking module at time t+Δt; v is the speed of the vehicle; and L is the distance between the camera and the tracking module.

[0086] Preferably, step S5 specifically includes:

[0087] The structure of the YOLOv8 model includes:

[0088] CSPDarknet backbone network, the formula is:

[0089]

[0090] SPPF module, serial pooling fusion of multi-scale features:

[0091]

[0092] Among them, X in is the input feature map; X out is the output feature map; n is the number of channels of the input feature map; Conv is the convolution operation; For feature splicing; MaxPool k is the maximum pooling operation with a window size of k;

[0093] The YOLOv8 model uses DEL to optimize the bounding box for prediction. The DEL formula is:

[0094] Model the bounding box coordinates as a discrete probability distribution and predict the probability of n+1 intervals P=[p0,p1,...,p n ], calculate the coordinate values ​​by expectation:

[0095]

[0096] DFL loss encourages high probability of intervals near the true coordinates:

[0097] DFL(p,t0)=-((t i+1 -t)log(p i )+(tt i )log(p i+1 )) (33)

[0098] in, is the predicted coordinate value calculated by probability distribution, y i is the center value of the i-th discrete interval, t0 is the real bounding box coordinate value, t i ,t i+1 is the endpoint of the discrete interval where the real coordinate t0 is located, and satisfies t i ≤t≤t i+1 ;

[0099] The regression accuracy is improved by CIOU Loss, and the formula is:

[0100]

[0101] wherein, p(b,b gt ) is the Euclidean distance between the center points of the predicted box and the real box; c is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box; w, h are the width and height of the predicted box; w gt , h gt are the width and height of the real box; τ is the aspect ratio consistency penalty term; γ is the dynamic weight;

[0102] The positive samples are selected by task alignment indicators to improve the consistency of classification and regression:

[0103] align_metric=(p c ·IOU(b,b gt )) δ (35)

[0104] The top k anchors corresponding to the highest align_metric of each real box are selected as positive samples;

[0105] wherein, p c is the predicted class probability, IOU(,b gt ) is the intersection over union of the predicted box and the real box, δ is a hyperparameter, and k is the number of positive samples selected for each real box.

[0106] Data transmission adopts a publish / subscribe communication model based on the MQTT protocol. The server and terminal devices act as publishers and subscribers to publish or subscribe information to related topics. EMQX is used as the MQTT message middleware, i.e., MQTTBroker, which is responsible for message routing, session management, and QoS guarantee, and realizes data transmission between publishers and subscribers. MQTT protocol (Message Queuing Telemetry Transport) is a lightweight message middleware protocol designed for low-bandwidth, high-latency, and unreliable network environments. It defines three levels of service quality (QoS) levels: 0, 1, and 2. The three levels correspond to the following in terms of transmission semantics, network overhead, and typical application scenarios: at most once, at least once, and exactly once; minimum, medium, and maximum; non-critical data, sensor data that needs reliable transmission, and payment instructions, critical control signals.

[0107] Therefore, the application adopts the above-mentioned inspection method of the industrial inspection system based on multi-modal fusion. In the positioning layer, the hybrid double-sided two-way ranging (HDS-TWR) and Taylor full centroid algorithm (Taylor-FCL) non-line-of-sight error compensation technology is adopted to improve the positioning accuracy by 15%; through the state vector splitting and covariance scalar optimization of Kalman filtering, the calculation time and memory occupation are reduced, and the real-time performance of the embedded platform is improved; combined with the sliding window de-extreme value filtering, the ranging stability is significantly improved. Through the fusion of infrared sensors and visual threshold segmentation technology, the time-synchronized displacement compensation mechanism is constructed in the tracking layer, which greatly improves the body stability and improves the system tracking robustness. The visual detection module is based on the improved YOLOv8 model, adopts the dynamic positive sample allocation strategy and CIOU loss function, so that the head and helmet detection mAP@0.5 reaches 93.5% and 94.7% respectively, and still maintains high robustness in complex occlusion scenes.

[0108] The technical solutions of the application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0109] Figure 1 The flowchart of the inspection method of the industrial inspection system based on multi-modal fusion of the application;

[0110] Figure 2 The plan view of the experimental site of the embodiment of the application;

[0111] Figure 3 The visual tracking test image of the embodiment of the application, wherein (a), (b), (c), (d) are images collected by the camera during tracking;

[0112] Figure 4 The P-R curve diagram of the embodiment of the application;

[0113] Figure 5 The confusion matrix diagram of the model training result of the embodiment of the application. DETAILED DESCRIPTION

[0114] The following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the application.

[0115] Please refer to Figure 1 , the inspection method of the industrial inspection system based on multi-modal fusion, the specific experimental process is as follows:

[0116] In the UWB and IMU fusion positioning test, pure UWB positioning technology is used to fuse positioning with IMU data, guiding the vehicle to automatically cruise along a preset coordinate path. The path is configured by setting target coordinate points at intervals. After the vehicle reaches a target point, it automatically adjusts its head direction to align with the next target point and continues moving. The UWB Kalman filter algorithm is optimized and used to process UWB positioning data to improve positioning accuracy and stability. The tracking module accuracy test evaluates the tracking module's accuracy during path tracking, detecting its ability to accurately identify and follow the path. The visual tracking accuracy test focuses on the performance of the vision system in tracking tasks, analyzing its path recognition accuracy and reliability. The visual tracking, tracking module, and UWB fusion positioning tracking test integrates visual tracking with the tracking module and UWB with IMU positioning to produce a fused performance image.

[0117] Due to site limitations, a 3.5m×3.5m positioning experimental area was built in the laboratory, such as Figure 2 As shown in the figure, the real coordinates of the three UWB base stations are measured, which are A (0m, 0m), B (0m, 3.2m), and C (3.2m, 3.2m). Then, four points are selected, with coordinates of a (0.70m, 0.90m), b (0.70m, 2.40m), c (2.40m, 2.40m), and d (2.40m, 0.90m).

[0118] 1. Positioning and tracking test

[0119] The experiment used an intelligent car equipped with an STM32F103 motherboard and a UWB positioning tag to move along a pre-set trajectory. The host computer used Master Base Station A to read the distances between the car and three base stations, A, B, and C. The movement sequence was a → b → c → d → a. When the car reached a target point, the system automatically triggered an adjustment mechanism, using data from the six-axis sensor to monitor the car's heading in real time and automatically adjust the vehicle's direction to continue driving.

[0120] This experiment uses a combination of sensors, such as a tracking module and an OpenMV camera, to enable the robot to follow a pre-defined track. The OpenMV camera and TCRT5000 module capture the robot's trajectory and transmit the data via the serial port to the STM32F103 motherboard. The motherboard integrates and analyzes the data and controls the robot's movements accordingly. When the camera and TCRT5000 module detect line deviation, the corresponding data flag changes. The host computer analyzes the data and controls the robot to readjust its operating state. This makes the line tracking process smoother and more accurate than using a single module alone.

[0121] In the process of tracking data collection, the accuracy of UWB and IMU positioning will directly affect the accuracy of tracking data. Therefore, we independently experiment on the automatic cruise algorithm of UWB positioning, and use it as a reference to compare the subsequent experimental data and verify the effectiveness and perfection of the comprehensive experiment.

[0122] In the UWB automatic cruise experiment, because the car only runs against a single target each time, and the direction adjustment frequency is low, the car's running track is relatively smooth, the heading angle fluctuation is small, and the mutation is less, and the data is stable. However, the filtering algorithm is not optimized during positioning, resulting in large noise interference, resulting in large fluctuations between the positioning test path and the target path, and even a large amount of flying.

[0123] After optimizing the Kalman filter, the noise interference is obviously improved, and the distance data measured by each base station between the tag is more stable, and the measured car motion path is more consistent with the target path.

[0124] The tracking module precision test uses a single tracking module to obtain the path data, and changes the running state along the laid line. In the experiment, the single tracking module is affected by the motor performance difference and the distance difference between the modules when obtaining data. In the process of car running, the car body shakes greatly, the heading angle data changes greatly, and there are many data burrs.

[0125] The visual tracking precision test uses visual tracking method, which is greatly affected by light intensity. As shown in Figure 3 The five area blocks A, B, C, D, and E are divided in the image collected by the camera, and each area block has two states 00 and 01, as shown in Table 1. State 00 means that no track is monitored in the block, and state 01 means that a track is monitored in the block. (a), (b), (c), (d) in the figure represent the images collected by the camera in the tracking process. When a track is monitored in one of the area blocks, the corresponding data bit in the table changes to 0x01.

[0126] Table 1 corresponding test data

[0127] Figure A B C D E (a) 00 00 01 00 00 (b) 00 00 01 00 00 (c) 00 00 01 00 00 (d) 00 01 00 00 00

[0128] One of the important factors affecting visual tracking is light. The infrared-vision dual-mode collaborative tracking method used in this experiment can better adapt to environmental light, and has good processing effect on color misjudgment and light interference.

[0129] In this experiment, the frequency of IMU measuring heading angle data is 20 times / s, at this time the data fluctuation is large, and the data mutation is more, because in a certain measurement process, if the change amplitude is large, but the measurement times are less, it will lead to large slope, and form burr.

[0130] In this experiment, only visual tracking and other methods can make the car move, and it is impossible to make the car to the specified position to perform other operations, so after the fusion of UWB technology, the movement route of the car can be set. After optimizing the Kalman filter technology, the measured car position is more consistent with the target path. Although there are still data running off situations, the positioning data is more consistent, and the car can expand more functions.

[0131] Single visual tracking or tracking module for tracking operation, the accuracy is poor in the process of line tracking. In this experiment, we fuse the OpenMV camera and the tracking module, observe the front road segment through the camera, and monitor the rear road segment through the tracking module. After obtaining the data, the host analyzes the data, and when the camera detects changes in the data, the camera transmits the data to the register. After a certain delay, when the tracking module reaches the position, the tracking state data is read. At this time, the host compares the two data, if they are the same, the corresponding movement is executed, if they are not the same, the original movement state is continued.

[0132] This method greatly improves the accuracy of car line tracking. The body shaking caused by changing direction during line tracking is greatly reduced, which improves the accuracy of line tracking.

[0133] In this experiment, the frequency of IMU measuring heading angle is increased to 50 times / s, which allows multiple data to be measured in a short time. The heading angle data graph is smoother and more smooth. By reducing the difference between each adjacent two data, the data increment or decrement is more smooth, thereby achieving the effect of reducing burr.

[0134] And we optimize the parameters of Kalman filter again, so that the measured car trajectory information is closer to the target path. The noise interference is well reduced.

[0135] 2 Data transmission

[0136] In this experiment, the car collects a variety of key data, including temperature and humidity, light intensity, car heading angle, and car position x and y coordinates, etc. The specific data description is shown in Table 2:

[0137] Table 2 Description of collected data

[0138] Dataname DataDescription Temp Temperature data collected Humi Humidity data collected TempThreshold Temperature threshold set Illuminance Illuminance data collected Yaw Yaw angle data of the trolley XCoordinate X-coordinate of the trolley's position YCoordinate Y-coordinate of the trolley's position IDcard Card number of the current card swiping person TempWarning Display whether the temperature exceeds the threshold Intnet Display whether the trolley is connected to the network Permission Display the permissions of the current ID card

[0139] The data collected by the car and related information are displayed intuitively on the screen of the car. Through the WiFi module, the car publishes some important data to the EMQX broker according to the corresponding topic based on the MQTT protocol, realizing the preliminary transmission of data. This process not only ensures the real-time nature of the data, but also improves the efficiency and reliability of data transmission.

[0140] The backend server also uses the MQTT protocol in the data transmission process, subscribes to related topics, and efficiently obtains the data sent by the car from the EMQX broker. After obtaining the data, the server not only prints it on the console for real-time monitoring, but also updates and stores it in the database, providing a solid foundation for subsequent data analysis and application.

[0141] In this experiment, Postman is used to simulate front-end access to the server, and control instructions are sent to the server using Postman to control the running mode or specific movement of the car. After receiving the data sent by the front-end, the server quickly publishes these data to the EMQX broker with the corresponding topic, so that the car can receive and execute the corresponding actions.

[0142] Mode 1, automatic cruise mode

[0143] The car automatically moves along the established trajectory, and acquires sensor data during the running process and uploads it to the server.

[0144] Table 3 Car tracking mode

[0145]

[0146] The mode of controlling the car is the automatic cruise mode. In the automatic cruise mode, the car can autonomously perform basic actions such as "forward", "backward", "left turn", "right turn", etc., without the need for real-time human intervention, greatly improving the running efficiency and automation level. This mode is suitable for application scenarios with relatively stable environment and fixed path, which can effectively reduce the cost of manual operation and improve work efficiency.

[0147] Mode 2, administrator control mode

[0148] The administrator manually controls the movement of the car, and can obtain the car status through the back-end server. The mode for controlling the car is the administrator control mode. The administrator control mode gives the administrator greater operational freedom. In this mode, the administrator can obtain the current status of the car in real time through the back-end server, including key information such as position, speed, and sensor data. At the same time, the administrator can manually control the movement of the car according to actual needs, and the operation interface for controlling the car "forward", "backward", "turn left" and "turn right" is displayed respectively. This mode enables the car to flexibly adjust its movement state according to the administrator's instructions in complex or special environments, enhancing the adaptability and controllability of the car. The administrator control mode is suitable for scenarios where frequent path adjustments or responses to emergencies are required, and can ensure the stable operation of the car in complex environments.

[0149] 3. Visual inspection

[0150] This experiment used a 64-bit Windows 10 operating system, the PyTorch 1.12.0 deep learning framework, and CUDA version 12.3. The experimental model was powered by an NVIDIA GeForce RTX 3060 Laptop GPU. Before model training, the input image size was 640×640, the initial learning rate was set to 0.01, the decay coefficient was set to 0.0005, the batch size was 8, and training was performed for 100 epochs. Detailed training parameter settings are shown in Table 4.

[0151] Table 4 Training parameter settings

[0152] Parameter Setting Parameter Setting Initial learning rate 0.001 Final learning rate 0.001 Training rounds 100 Warm-up rounds 10 Number of threads 1 Number of images per batch 8 Optimizer SGD + AdamW Weight decay 0.0005

[0153] The initial learning rate and final learning rate are set to 0.001, setting a small initial learning rate can help the model stabilize learning in the early stage of training, avoid missing the optimal solution due to large update steps at the beginning of training, help the model converge to a better local minimum, and avoid unstable training. The number of training rounds is 100 rounds, allowing the model to learn for enough rounds to fully capture the characteristics of the data. Training combines the advantages of stochastic gradient descent (SGD) and AdamW optimizer, SGD helps the model escape local minimum, while AdamW provides adaptive learning rate and weight decay, this combination can provide faster convergence speed and better generalization ability. The weight decay is set to 0.0005 for regularization to prevent overfitting of the model. The warm-up round is set to 10 cycles. The warm-up phase gradually increases the learning rate to the initial value, which helps the model start training stably in the early stage and avoids sharp fluctuations in the initial stage. The number of images per batch is 8. The model training in this paper uses NVIDIA GeForce RTX 3060 Laptop GPU graphics card, considering the hardware resource limitation, setting to 8 helps to improve the efficiency of model training. The number of threads is 1, because in the case of limited resources, using single thread can avoid resource competition and simplify the debugging process.

[0154] The research method adopts the following indicators: precision (P), recall (R), F1 score (F1), mean average precision (mAP), and FPS (Frames Per Second) to evaluate the detection accuracy, classification performance, and detection speed of the model.

[0155] In target detection, mAP is a commonly used evaluation index, which is used to measure the detection performance of the model on different classes. mAP@0.5 represents the average precision of the class calculated when the IoU (Intersection over Union) threshold is 0.5. The calculation formula of mAP is as follows:

[0156]

[0157] AP = ∫0 1 PdR;

[0158]

[0159] Among them, P refers to the proportion of real targets in the bounding box detected by the model; R refers to the proportion of the number of real targets successfully detected by the model to the total number of real targets; the number of bounding boxes correctly predicted by the true positive model (TP) that contain real targets; the number of bounding boxes incorrectly predicted by the false positive model (FP) that contain non-real targets; the number of real targets not detected by the false negative model (FN); the F1 score is the harmonic mean of precision and recall, which is used to comprehensively evaluate the performance of the model; AP is the area under the PR curve, which indicates the average detection accuracy of the model in this category; mAP is a comprehensive evaluation indicator that combines the precision and recall of the model in different categories and averages the accuracy of different categories; AP i is the average precision of the i-th category, and e represents the total number of detection categories.

[0160] 4. Results

[0161] (1) YOLOv8n model training results

[0162] In the task of performing head target detection, its average precision (AP) reached 93.5%. Specifically, for the "head" category, the model's average precision (mAP@0.5) at a 0.5IOU threshold is 0.935. This data fully demonstrates the reliability of the model in identifying head targets. Similarly, for detecting whether a target is wearing a helmet, the model's average precision reached 94.7%, showing its high accuracy in the "helmet" category. At the same 0.5IOU threshold, the average precision (mAP@0.5) of the helmet category is 0.947. In summary, the YOLOv8n model has demonstrated a high level of performance in both head and helmet detection tasks, and its accuracy and reliability have outstanding advantages over similar models, such as Figure 4 shown.

[0163] Figure 5 This is a confusion matrix plot of the YOLOv8n model training results, used to evaluate the performance of the classification model. It shows the classifier's prediction results on four categories ("person", "head", "helmet", and "background"). The accuracy of "person" is 100%, "head" is 92% (0.92), "helmet" is 91% (0.91), and "background" is 96% (0.96).

[0164] (2) Video detection

[0165] Video detection includes video file detection and camera real-time detection. Its core function is to realize real-time detection of the target through the camera. This processing process is carried out frame by frame, that is, each frame of image in the video is detected in detail to ensure that no moment containing the target is missed, thereby ensuring the comprehensiveness and accuracy of the detection. As shown in the following figure, the system interface helmet recognition situation will clearly mark the detected target on the video frame through specific marks (such as frame selection, detection confidence, etc.), and continuously update with frame-by-frame detection, while the detection result is presented in the form of data in the lower left corner of the interface. In this way, the user can intuitively see the detection result without the need to interpret complex data.

[0166] (3) Picture detection

[0167] In the picture detection module, the user can select the image file by himself, the system displays the image after image preprocessing, and then uses the model to predict. The interface will intuitively display the image detection result in a contrast form. The left side of the interface is the original image file selected by the user, and the right side of the interface shows the image file labeled after model prediction. At the same time, the statistical information of the detection result is presented in the form of data in the lower left corner of the interface, such as the number of detected helmet targets, thereby providing comprehensive analysis feedback for the user.

[0168] Therefore, the above-mentioned industrial inspection system based on multi-modal fusion is adopted. In the positioning layer, the non-line-of-sight error compensation technology of hybrid double-sided two-way ranging (HDS-TWR) and Taylor-FCL is used, so that the positioning accuracy is improved by 15%; through the state vector splitting and covariance scalar optimization of Kalman filter, the calculation time and memory occupation are reduced, and the real-time performance of the embedded platform is improved; combined with the sliding window de-extreme value filtering, the ranging stability is significantly improved. Through the fusion of infrared sensors and visual threshold segmentation technology, a time-synchronized displacement compensation mechanism is constructed in the tracking layer, so that the body stability is greatly improved, and the system tracking robustness is improved. The visual detection module is based on the improved YOLOv8 model, adopts a dynamic positive sample allocation strategy and a CIOU loss function, so that the head and helmet detection mAP@0.5 reaches 93.5% and 94.7% respectively, and still maintains high robustness in complex occlusion scenes.

[0169] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application and not to limit them, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can still be modified or replaced by equivalents, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. The inspection method of the industrial inspection system based on multimodal fusion is characterized by: The following steps are involved: S1. After obtaining the ranging data from the UWB base station to the node to be located using the HDS-TWR ranging algorithm, the Taylor full centroid algorithm is used to solve the positioning data of the node to be located and output the positioning coordinates. S2, using sensors to measure the acceleration and angular velocity of the inspection system to evaluate the alignment motion and posture; S3, improve the Kalman filter algorithm, process the data in steps S1 and S2, so that the output prediction result is close to the true value; S4: Using infrared and visual dual-mode collaborative tracking, while using a time-synchronized displacement compensation mechanism to calculate the relative displacement difference between the two tracking modes for dynamic calibration; S5. Use the YOLOv8 model to process the video frames or pictures collected during the inspection process, output the target box and category probability, and finally obtain the inspection results; The HDS-TWR ranging algorithm in step S1 is: （1） （2） （3） in, 、 、 The tag sends to base stations A, B, and C respectively. Message to receive the first response message from the corresponding base station 、 、 Round trip time; 、 、 The time interval from when the tag receives the first response message from the base station to when it sends the Final message; 、 、 After receiving the Poll message, base stations A, B, and C send 、 、 Time to respond to the message; 、 、 The time from when the tag sends the Final message to when base stations A, B, and C receive the Final message; 、 、 represent the propagation time of the signal between the tag and base stations A, B, and C respectively; The Taylor full centroid algorithm is used in step S1 to calculate the positioning coordinates as follows: Assume that the coordinates of base stations A, B, C and tags are 、 、 as well as , the ranging distances from each positioning base station to the node to be positioned are 、 、 , we obtain the following system of equations: （4） Solve using the full centroid algorithm and rewrite it into matrix form: （5） Further, we get: （6） Combining formulas (5) and (6), we define , , , we get the following linear equations: （7） Least squares method get: （8） Thus, the initial coordinate value of the label is obtained , the actual coordinates satisfy: （9） in, 、 is the difference between the estimated coordinates and the true coordinates; From formula (4), we can see that the distance between the base station and the tag measured by the positioning base station ideally satisfies the formula: （10） in, For the The coordinates of the base stations, For the The distance from the base station to the positioning node; by doing a Taylor expansion of (10), the linear equations are solved. For the two-variable Taylor Expanded, there are: （11） Will Substituting into formula (10), we get: （12） （13） Combined with formula (9), the least squares method is used to calculate , ,in: ； when When the value is less than the preset convergence threshold or the number of iterations exceeds the maximum allowed value, the algorithm terminates the iteration and outputs the positioning coordinates.

2. The inspection method of the industrial inspection system based on multimodal fusion according to claim 1 is characterized in that: Step S2 uses the MPU6050 sensor to measure data. The DMP library automatically fuses the sensor data and outputs quaternions for pose calculation. Specifically: Assume that the quaternion outputs of multiple sensors are , the final quaternion is obtained using the weighted average method: （14） in, is the weight coefficient, For the The quaternion output by each sensor; The data obtained by the MPU6050 sensor includes acceleration , angular velocity , pitch angle and roll angle ,in: （15） （16） Among them, the pitch angle and roll angle are used to calculate the updated quaternion: （17）。 3. The inspection method of the industrial inspection system based on multimodal fusion according to claim 2 is characterized in that: Step S3 splits the state value into and , perform two Kalman filters respectively, and the filtering method is divided into two steps: prediction and update: predict: （18） （19） renew: （20） （21） （22） in, for The state value at the moment; is the covariance matrix of the state estimate; is the process noise covariance matrix; for Kalman gain calculated at each moment; is the measurement noise covariance matrix; for The observed value at the time; is the identity matrix.

4. The inspection method of the industrial inspection system based on multimodal fusion according to claim 3 is characterized in that: The infrared-visual dual-mode collaborative tracking in step S4 includes a visual tracking module based on the OpenMV camera and an infrared reflective tracking module. The visual tracking module identifies specific path markers through color threshold segmentation and multi-ROI detection, and sends the detection results to the main controller through the serial port. Specifically: ROI area coordinate definition: Assume the image resolution is , define 5 ROI areas evenly distributed laterally, The coordinate formula of a ROI is: （23） （24） in, Each value corresponds to the box selected at the upper left corner of the ROI coordinate, The upper left corner coordinate, Width, height; Color threshold segmentation: Define the color threshold range as: , where each value corresponds to the minimum and maximum brightness, the minimum and maximum values ​​of channel A, and the minimum and maximum values ​​of channel B; Test result code: Each Test results Defined as a binary variable: like Target color block exists in memory ,otherwise ; The final 5 detection flags sent are ; Serial port data frame structure: The data frame format is 8-byte little-endian mode: .

5. The inspection method of the industrial inspection system based on multimodal fusion according to claim 4 is characterized in that: The infrared reflective tracking module specifically includes: Infrared reflection intensity model: When the infrared emitting tube emits light with an intensity of The infrared light is reflected by the target surface, and the light intensity detected by the receiving tube is Expressed as: （25） in, is the optical system efficiency; is the target surface reflectivity; is the distance between the sensor and the target; is the medium attenuation coefficient; Signal threshold determination: received signal voltage and light intensity Into a linear relationship: （26） in, is the photoelectric conversion coefficient, is environmental noise; Setting thresholds , output digital signal: .

6. The inspection method of the industrial inspection system based on multimodal fusion according to claim 5 is characterized in that: The time synchronization displacement compensation mechanism is as follows: Assume that the camera and tracking module are at time and Monitor a certain point P; Establish the spatiotemporal observation equation of the camera and tracking module. The camera is in time The observation position is: （27） in, is the initial position; Location of the tracking module monitoring point: （28） By differentiating the time difference , calculate the relative displacement difference between the two monitoring points P: （29） Characterize the spatial offset caused by time asynchrony, and convert the relative displacement difference Preset physical distance Compare, if , indicating that the spatiotemporal consistency is destroyed, triggering the offset adjustment mechanism; among them, For time The position of point P detected by the tracking module; is the speed of the vehicle; is the distance between the camera and the tracking module.

7. The inspection method of the industrial inspection system based on multimodal fusion according to claim 6 is characterized in that: Step S5 specifically includes: The structure of the YOLOv8 model includes: CSPDarknet backbone network, the formula is: （30） SPPF module, serial pooling fusion of multi-scale features: （31） in, is the input feature map; is the output feature map; is the number of channels of the input feature map; is the convolution operation; For feature splicing; The window size is The maximum pooling operation; The YOLOv8 model uses DEL to optimize the bounding box for prediction. The DEL formula is: Model the bounding box coordinates as a discrete probability distribution, predict The probability of an interval , calculate the coordinate values ​​by expectation: （32） DFL loss encourages high probability of intervals near the true coordinates: （33） in, is the predicted coordinate value calculated by probability distribution, For the The center value of a discrete interval, is the real bounding box coordinate value, is the real coordinate The endpoints of the discrete interval where ; The regression accuracy is improved by CIOU Loss. The formula is: （34） in, , is the Euclidean distance between the center point of the predicted box and the center point of the real box; is the diagonal length of the minimum enclosing rectangle of the predicted box and the real box; is the width and height of the prediction box; , is the width and height of the real box; is the aspect ratio consistency penalty; is the dynamic weight; Select positive samples through task alignment indicators to improve the consistency of classification and regression: （35） Select each ground-truth box corresponding to The highest front Anchor points are used as positive samples; in, is the predicted category probability, is the intersection-over-union ratio of the predicted box and the true box, is a hyperparameter, The number of positive samples selected for each ground-truth box.

Images