Steel pipe whole-process intelligent tracking system based on multi-mode sensing fusion

Abstract

The invention provides a steel pipe whole-process intelligent tracking system based on multi-mode sensing fusion, and relates to the technical field of ferrous metallurgy intelligent manufacturing. The system comprises the following modules: a visual perception module, which is used for collecting the appearance characteristics, the spatial position and the motion state of a steel pipe in the whole process of steel pipe tracking; the one-by-one tracking and decision-making module is used for constructing a closed-loop work flow comprising data acquisition, fusion processing, decision-making output and feedback optimization by deeply fusing visual perception information and multi-source industrial signals; and the control interaction module is in two-way communication with the bottom PLC and an upper computer, receives equipment signals, feeds back the equipment signals to the tracking module, issues a tracking / alarming instruction to the PLC and synchronizes data to the MES. According to the method, the problems of blind areas, information chain scission and manual dependence of traditional tracking are solved, the whole process of the hot-rolled seamless steel pipes from a cooling bed area to material frame bundling through saw cutting segmentation is achieved, and one-by-one accurate tracking and quality control are achieved.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology in the iron and steel metallurgy industry, and in particular to an intelligent tracking system for the entire process of steel pipe based on multimodal perception fusion. Background Technology

[0002] Steel pipes are a fundamental building material in industry, but their production, distribution, and quality inspection processes suffer from difficulties in identification and tracking. Steel pipe tracking technology was developed to address this, assigning a unique identifier to each steel pipe and recording its entire process information through a system. However, existing steel pipe tracking technologies suffer from problems such as tracking chain interruptions, sensitivity to environmental interference, system isolation, high reliance on manual labor, and difficulties in achieving closed-loop defect tracing in the later stages of cooling.

[0003] Chinese patent CN117217098A discloses a method and device for predicting the mechanical properties of steel pipes based on digital twins and deep learning: A digital twin system for hot-rolled seamless steel pipes is established based on a hot-rolled seamless steel pipe production line to obtain real-time production parameters; a LUBE model for estimating upper and lower limit intervals based on the Sparrow Search Algorithm (SSA) is constructed; and based on the real-time production parameters and the SSA-optimized LUBE model, the predicted range of mechanical properties for each steel pipe in the hot-rolled seamless steel pipe production line is obtained.

[0004] The drawbacks of this technology are that the digital twin system needs to integrate real-time parameters from multiple sources (such as temperature and chemical composition), breaking down data silos relies on high-precision sensors, and noise from actual operating conditions (such as temperature measurement deviation) can cause input anomalies; the weight optimization of the LUBE model optimized by SSA is complex, the hyperparameter settings are highly subjective, and the PICP and PINAW indices conflict, making the prediction interval prone to being too wide (high coverage but low practicality) or too narrow (high false negative rate); preprocessing relies on the PauTa criterion, and valid samples are mistakenly discarded when the data distribution is skewed; the model generalization is limited by the training data, and retraining is required when new steel grades or process changes occur.

[0005] Chinese Patent Publication No. CN114472527A discloses a seamless steel pipe tracking production system, including a server, a switch, a code reading module, and a robot. The server has multiple logical workstations, which correspond to the actual physical workstations in production. The server controls the robot to track the identification of each pipe in the non-hot rolling zone using labeling and / or inkjet printing. In the hot rolling zone, the server receives the identification read by the code reading module, stores it in relation to the logical workstation, and receives signals from the PLC of the production equipment in the hot rolling zone according to the switching of physical workstations, and switches the logical workstations accordingly, thereby tracking the identification of each pipe. In the hot rolling zone, the pipes follow the principle of single-line transmission and first-in-first-out.

[0006] The disadvantages of this technology are that the hot rolling zone relies on single-line transmission and the first-in-first-out principle, and the logic station switching fails when the production line is abnormal (such as material jamming or accumulation); high-temperature zones (such as ring furnaces and cooling beds) cannot be labeled, and only PLC position signal tracking is used, and encoder reading deviations cause label misalignment; the robot labeling in non-hot rolling zones (such as inkjet printing on cooling beds) is affected by stains on the steel pipe surface or rolling friction, resulting in a decrease in recognition rate (requiring machine vision deep learning reinforcement); the temperature measurement accuracy in the roll reduction formula directly affects the calculation, and thermocouple errors can easily increase the scrap rate; the system has high complexity, and the failure rate of multi-module (code reader, PLC, server) collaboration is high, making maintenance difficult. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes an intelligent tracking method and system for the entire steel pipe process, integrating machine vision, multi-source signal fusion, and dynamic spatial modeling. This system is applicable to the downstream processes of hot-rolled seamless steel pipe production lines (tension reduction mill exit → cooling bed → roller cutting → track transmission → material frame warehousing). By constructing a collaborative tracking system of "visual perception - signal fusion - dynamic tracking - defect management - closed-loop control," it solves the problems of blind spots (cooling bed stacking area, track bifurcation points), information chain breaks (interruption of connection between sawn segments and the main pipe), and reliance on manual labor (manual pre-inspection, manual batching) in traditional tracking. This enables precise tracking and quality control of hot-rolled seamless steel pipes throughout the entire process, from the cooling bed area through sawing and segmentation to bundling in the material frame.

[0008] The specific implementation path includes: a visual perception module for high-precision acquisition of the appearance, position, and motion status of steel pipes; a multi-source signal fusion engine for integrating visual, industrial control, and sensor data to generate holographic status information; a pipe-by-pipe tracking and decision-making module for identity management, position tracking, and sawing decisions; and a control interaction module for linking underlying equipment and feeding back data to the MES to form a closed loop.

[0009] After the application of this invention, the automation rate increased from 70% to 95%, the traceability accuracy rate increased from 80% to 99.2%, and the per capita output value increased by 40%, providing support for intelligent manufacturing of steel.

[0010] The relevant terms in this invention are explained as follows:

[0011] (1) Machine vision inspection: Steel pipe images are collected by industrial cameras and combined with deep learning models to realize defect identification, posture judgment and number identification. It is the basic data source for intelligent tracking throughout the whole process.

[0012] (2) Multi-source signal fusion: Integrate industrial control signals (such as PLC, motor encoder), sensor data (such as temperature, pressure) and visual data, and provide a holographic description of the steel pipe status through time synchronization and feature-level fusion.

[0013] (3) Intelligent scheduling and task matching: Based on the current process of the steel pipe, available testing resources and system status, the testing tasks are dynamically scheduled to achieve multi-task parallelism and optimal resource allocation.

[0014] (4) Data closed-loop control: Summarize the quality inspection results of each process, issue early warning or rejection signals for abnormal steel pipes, and provide feedback for the optimization of subsequent processes to realize intelligent closed-loop quality control.

[0015] In a first aspect, the present invention provides a steel pipe end-to-end intelligent tracking system based on multimodal perception fusion, comprising the following modules:

[0016] M1, Visual Perception Module: Collects the appearance features, spatial position, and movement status of the steel pipe throughout the entire steel pipe tracking process, including the cooling bed inlet, sawing area, and material frame terminal; specifically includes the following modules:

[0017] M11, High-resolution video acquisition and environmental adaptation module;

[0018] M12, video storage and preprocessing module;

[0019] M13, Deep Learning-Based Intelligent Recognition and Feature Extraction Module: including deep learning-based target detection, target tracking, and feature extraction;

[0020] M2, Component Tracking and Decision Module: By deeply integrating visual perception information and multi-source industrial signals, a closed-loop workflow is established, including data acquisition, fusion processing, decision output, and feedback optimization. The processing of core process nodes includes:

[0021] Steel pipe number: A unique initial number is generated at the inlet of the cooling bed for this steel pipe;

[0022] Flow direction verification: The flow direction of the steel pipe is verified at the junction of the track bifurcation.

[0023] Cutting and segmentation: During the roller cutting process, the number of cutting segments is calculated, and the parent-child relationship of the steel pipe numbering is established;

[0024] Flow diversion verification: Verify the actual flow direction of the cut steel pipe segments;

[0025] Final confirmation: Confirm the identity and defect information of the steel pipes arriving at the material frame;

[0026] M3, Control Interaction Module: It communicates bidirectionally with the underlying PLC and the upper MES / ERP computer, receives equipment signal feedback to the tracking module, sends tracking / alarm commands to the PLC, and synchronizes data to the MES;

[0027] The modules are interconnected via high-speed industrial Ethernet, which aggregates the quality inspection results of each process, issues early warnings or rejection signals for abnormal steel pipes, and provides feedback for the optimization of subsequent processes, thus realizing an intelligent closed loop of quality control.

[0028] As a further improvement of the present invention, the module M11 performs video acquisition via an industrial camera, specifically:

[0029] Deploy industrial cameras at key nodes including the cooling bed inlet, track bifurcation point, roller cutter outlet, and material frame drop point;

[0030] The industrial camera integrates automatic exposure, 120dB wide dynamic range, autofocus, adaptive white balance, strong light suppression, and infrared night vision functions.

[0031] As a further improvement of the present invention, the specific working mode of module M12 includes:

[0032] The video stream is transmitted to the NVR via Gigabit Ethernet, using H.265 / H.265+ encoding;

[0033] Video preprocessing includes: Gaussian filtering to remove Gaussian noise and median filtering to remove salt-and-pepper noise, grayscale conversion, adaptive histogram equalization, and ROI extraction.

[0034] As a further improvement of the present invention, the specific method for target detection in module M13 is as follows: an end-to-end vision model suitable for real-time scenarios is used to perform target detection on the preprocessed image, and the bounding box coordinates, category and confidence of the steel pipe are output; the surface defects of the steel pipe are further output with the type, location and size of the defects.

[0035] As a further improvement of the present invention, the specific method for target tracking in module M13 is as follows:

[0036] A multi-target tracking method suitable for real-time scenarios is adopted. Based on the location information and appearance features of the target recognition results, a unique tracking identifier is assigned to each steel pipe. Continuous tracking is achieved through state prediction and appearance matching in cases of occlusion, intersection, entry and exit from the field of view, etc.

[0037] To address tracking instability issues such as steel pipe occlusion, intersection, and entry / exit from the camera's field of view, Kalman filtering is used to predict the steel pipe's position in the next frame, while simultaneously performing appearance feature similarity matching.

[0038] As a further improvement of the present invention, the specific method for feature extraction in module M13 is as follows: a feature extraction network based on deep learning is used to extract key visual features of the steel pipe. These key visual features include at least: shape contour features, surface texture features, a set of microscopic feature points on the end face, and an end face fingerprint code. Specifically:

[0039] External contour feature extraction: The outer circle contour and inner hole contour of the steel pipe are extracted by edge detection algorithm, and the diameter and roundness dimensional parameters of the steel pipe are calculated;

[0040] Surface texture feature extraction: The gray-level co-occurrence matrix algorithm is used to obtain unique texture features of the steel pipe surface, which serve as the texture fingerprint of the steel pipe;

[0041] End face micro-feature point set and end face fingerprint code extraction: More than 200 micro-feature points on the end face are extracted using a convolutional neural network (CNN) to generate a unique feature vector, which serves as the end face fingerprint code for redundant verification of the steel pipe's identity; the micro-feature points include scratch distribution, oxide scale spot location, and inner hole edge details.

[0042] As a further improvement of the present invention, for scenarios where the steel pipe number is ambiguous, obscured, or the signal is abnormal, the redundancy verification is performed, and the specific method includes:

[0043] At the inlet of the cooling bed and the node after cutting, a high-resolution area array camera is deployed with a coaxial light source so that the camera is facing the end face of the steel pipe to take clear pictures.

[0044] The basic features of the steel pipe are enhanced by using a feature encoding network based on deep learning, generating a unique end-face feature vector, which serves as the end-face fingerprint code of the steel pipe.

[0045] Redundancy verification and identity correction of facial features are achieved through similarity algorithms.

[0046] As a further improvement of the present invention, the uses of the redundancy verification include:

[0047] The feature vector of the steel pipe end face collected at the inlet of the cooling bed is compared with the feature vector of the end face of the cut steel pipe segment to help verify the correlation between the segment and the parent pipe.

[0048] When the steel pipe number is lost, misaligned, or visual tracking is temporarily interrupted due to obstruction, the historical end-face fingerprint code is retrieved and compared with the currently collected end-face feature vector to repair the tracking chain and avoid tracking chain breakage or misalignment caused by the failure of a single information source.

[0049] As a further improvement of the present invention, the method for deeply fusing visual perception information and multi-source industrial signals in module M2 includes a collaborative mechanism between visual signals and equipment signals, specifically including:

[0050] Time synchronization mechanism: Establish a unified time reference so that the visual recognition results correspond to the PLC signals in the time dimension;

[0051] Spatial mapping mechanism: Mapping the pixel coordinates of visual recognition to the physical coordinates of the actual production line to achieve seamless integration between virtual and reality;

[0052] Logical verification mechanism: Through multiple logical verifications, the consistency between visual recognition results and device signals is ensured.

[0053] As a further improvement of the present invention, the method for deeply fusing visual perception information and multi-source industrial signals in module M2 further includes a data fusion algorithm, specifically including:

[0054] Kalman filter algorithm: used for dynamic estimation and prediction of the position of steel pipe, maintaining the continuity of tracking in the case of temporary visual obstruction;

[0055] Bayesian inference algorithm: Based on prior knowledge and real-time observation data, it performs probabilistic inference on the identity and state of steel pipes;

[0056] Multimodal feature fusion algorithm: fuses visual features, geometric features, and motion features.

[0057] As a further improvement of the present invention, the processing of the core process nodes in module M2 specifically includes:

[0058] Steel pipe number: At the entrance of the cooling bed, the visual features extracted by the M1 module are associated and bound with the production data to generate a unique initial number for the steel pipe;

[0059] Flow direction verification: Flow direction verification is performed by combining the trigger signal of the photoelectric sensor at the track bifurcation point, the actual flow direction of the steel pipe captured by the camera, and the preset diversion command issued by the PLC.

[0060] Cutting and segmentation: Based on the original steel pipe length estimated by production data / visual estimation and the preset cutting length, the system intelligently calculates the number of cutting segments, automatically expands the original steel pipe number to a new number with a segment suffix, and establishes a parent-child relationship;

[0061] Diversion verification: At the track exit, cameras and photoelectric sensors are used to confirm the actual flow direction of the cut steel pipe segments and perform secondary verification with the PLC diversion command to dynamically update the queue number and position information of the steel pipe segments.

[0062] Final confirmation: Combining the material unloading completion signal sent by the PLC, the steel pipe entering the frame captured by the terminal camera and the numbering mark, the final position and identity of the steel pipe that has arrived at the material frame are confirmed, and the binding defect information of the steel pipe is pushed to the sawing operator interface.

[0063] As a further improvement of the present invention, the processing of the core process node also includes the prediction and calibration of the steel pipe position, the specific method of which is as follows:

[0064] Location prediction is performed using the following formula:

[0065] P t=P0+v×(t-t0);

[0066] Among them, P t Let t be the predicted position of the steel pipe at the current time t, t0 be the initial detection time, P0 be the initial position of the steel pipe at the initial detection time t0, v be the real-time transmission speed of the roller conveyor, and t-t0 be the time interval from the initial detection to the current time.

[0067] The predicted value is compared with the actual position detected by vision. If the deviation exceeds the threshold, speed calibration is triggered.

[0068] As a further improvement of the present invention, in the process of processing the steel pipe number,

[0069] The visual features include: the initial position of the steel pipe, the estimated length, and surface defects;

[0070] The production data includes: the actual length of the steel pipe, known defect information, production batch number, steel type, and specifications.

[0071] As a further improvement of the present invention, the processing of the steel pipe number also includes: real-time comparison of the steel pipe position / predicted position tracked by vision with the cooling bed exit arrival signal issued by the PLC; when it is confirmed that the steel pipe has reached the preset cooling bed exit position, the cooling bed queue count is reduced, the steel pipe number is officially activated, and the bound defect information is synchronized to the downstream process.

[0072] As a further improvement of the present invention, the specific process of the segmentation includes:

[0073] Based on the position of the steel pipe head tracked by vision, the linear speed of the roller conveyor provided by the PLC, and the trigger signal of the cutting machine, the time when the steel pipe head reaches the cutting point is calculated.

[0074] When calculating the number of cutting segments, the actual length measured by dynamic reference vision and the allowable error of the process are taken into account.

[0075] After the cutting was completed, the camera immediately verified the number of steel pipe segments.

[0076] As a further improvement of the present invention, the operation mode of module M3 also includes:

[0077] Industrial communication gateways are used;

[0078] Data standardization methods include: unifying the numbering system of different systems through an identifier mapping table;

[0079] Data synchronization mechanisms include real-time transmission, batch synchronization, or cached push, ensuring that data can still be delivered on time even in the event of network fluctuations.

[0080] As a further improvement of the present invention, module M3 also includes a real-time data stream processing mechanism, specifically including:

[0081] Streaming data processing: Apache Kafka streaming data processing technology is used for real-time data transmission and processing;

[0082] Data caching mechanism: Set up data caching at critical nodes to ensure that data is not lost when the network fluctuates or the system is busy;

[0083] Data compression and optimization: Intelligent compression of image and signal data.

[0084] As a further improvement of the present invention, module M3 also includes a data quality assurance mechanism, specifically including:

[0085] Data validation mechanism: Perform integrity and consistency checks on all input data;

[0086] Abnormal data handling: Automatically identify and process abnormal data through abnormal data detection and processing rules;

[0087] Data backup and recovery mechanism.

[0088] Secondly, the present invention provides a hardware deployment device for a steel pipe end-to-end intelligent tracking system integrating multimodal perception, to realize the functions of the system described in the first aspect, wherein the hardware deployment device includes:

[0089] Industrial cameras: used for image acquisition of steel pipes at the inlet of the cooling bed, providing basic data for high-precision visual inspection;

[0090] Coaxial lighting unit: Provides illumination compensation for key track nodes, eliminating shadows and reflections;

[0091] GPU server: Runs steel pipe detection and tracking algorithms in real time and processes multiple high-definition video streams;

[0092] PROFINET Gateway: Acquires and forwards PLC control signals, and calibrates synchronously with video recognition results.

[0093] As a further improvement, a layered hardware deployment is adopted, with sensing devices configured according to the steel pipe flow path to build a full-process intelligent sensing network. The specific deployment includes:

[0094] Cooling bed entrance monitoring point: Deploy a high-definition camera equipped with a protective cover and a dedicated light source at the location where the steel pipe enters the receiving trough to capture the first frame image of the steel pipe and perform initial identity binding;

[0095] Cooling bed exit monitoring point: Deploy monitoring equipment at the position where the steel pipe leaves the cooling bed and enters the longitudinal track to confirm the steel pipe's exit status from the cooling bed and path selection;

[0096] Longitudinal track bifurcation monitoring point: Deploy cameras at the bifurcation of the left and right tracks, and combine them with PLC signals to confirm the specific flow direction of the steel pipe and prevent path judgment errors.

[0097] Roller cutting monitoring points: Dedicated cameras are deployed at the roller cutting location to monitor the cutting process in real time and verify the number of steel pipe segments after cutting;

[0098] Material frame inlet monitoring point: Deploy monitoring equipment at the final confirmation point before the steel pipe enters the material frame to perform final identity verification and location confirmation.

[0099] As a further improvement of the present invention, the hardware deployment device further includes a distributed processing architecture, specifically including:

[0100] Edge computing devices are deployed at edge computing nodes to perform preliminary preprocessing and feature extraction on images from each monitoring point;

[0101] The central processing unit is equipped with a high-performance GPU, which is responsible for performing complex deep learning inference and data fusion tasks;

[0102] The cloud storage and analysis component is responsible for analyzing historical data and reports, providing data mining and trend analysis.

[0103] Thirdly, the present invention provides an operation and maintenance monitoring method for a steel pipe end-to-end intelligent tracking system that integrates multimodal perception, so as to realize operation and maintenance monitoring of the system described in the first aspect or the hardware deployment device described in the second aspect. The operation and maintenance monitoring method includes the following steps:

[0104] S1. Data Acquisition and Diagnosis: Collect data from the equipment layer and analyze it to obtain normal / abnormal health diagnosis results;

[0105] S2. Conduct operational analysis based on the health diagnosis results to obtain a solution; specifically including:

[0106] S21. Perform root cause analysis and fault classification on the diagnostic results of the anomaly, and derive a solution for hardware spare parts scheduling or software container reconstruction.

[0107] S22. For normal diagnostic results, perform performance optimization analysis and dynamically allocate resources accordingly to achieve GPU load balancing.

[0108] S3. Use the operations and maintenance dashboard to execute the processing plan and generate an automatic repair report.

[0109] As a further improvement of the present invention, the core evaluation indicators and corresponding evaluation methods used in step S1 include:

[0110] Tracking accuracy: By comparing the system output with the actual condition of the steel pipe, the proportion of correct matches is statistically analyzed to obtain the defect / number identification accuracy rate;

[0111] Operational stability: Record the system operation log for the whole year, count unplanned downtime events, and obtain the number of unplanned outages for the year;

[0112] Data integrity and traceability: Randomly sample historical records to verify whether the entire process information of steel pipes is complete and traceable, and obtain the record integrity rate of steel pipes from the cooling bed to the warehouse;

[0113] Anti-interference capability: The system is tested in simulated or actual harsh environments to detect the quality of image and signal acquisition and to obtain the stable operating rate under high dust and high temperature environments;

[0114] Multimodal reliability: In daily operation and special tests, we statistically analyze whether visual recognition and signal logic reach the same conclusion, and whether data from different sources can consistently identify the same steel pipe. We record inconsistencies and analyze the causes to obtain the consistency rate of multimodal fusion.

[0115] Compared with the prior art, the present invention achieves the following technical effects:

[0116] (1) Machine vision-based end-face feature fingerprint extraction technology (accurately identifies multi-dimensional visual features such as inner hole contour and surface texture), integrates position signals, and constructs a multi-modal identity recognition model. When steel pipes are stacked together, a 99.2% differentiation accuracy rate (actual data) is achieved, replacing the traditional coarse-grained tracking based solely on length / position signals, ensuring the uniqueness and continuity of steel pipe identity in the cooling bed area throughout the entire process, and completely avoiding mixed pipes and wrong pipes.

[0117] (2) Establish an intelligent cutting information inheritance mechanism, use the cutting trigger signal (multi-source signal fusion) to synchronously generate segment numbers (such as P-202405001→P-202405001-1), and use intelligent tracking algorithm to automatically and accurately (accuracy rate ≥98%) associate the mother tube defect information (location, type) with the sub-segment, realize seamless transmission and traceability of defect information throughout the process, and completely solve the problem of broken quality traceability chain after sawing.

[0118] (3) A multimodal dynamic trajectory fusion engine is constructed, which deeply integrates machine vision positioning (integrating anti-strong light / occlusion interference algorithm), real-time industrial control signals (such as encoder pulses) and spatial topology relationship algorithm to break through the bottleneck of tracking stability under complex working conditions. This multimodal fusion strategy ensures that the tracking accuracy of the steel pipe throughout the entire process reaches 95% under complex environments such as extreme temperature difference (-20℃~80℃) and high dust (PM10≤100mg / m³).

[0119] (4) An integrated pre-inspection guidance module, based on the defect coordinates identified by machine vision and the multi-source signal fusion positioning results, enables real-time push of defect information to the sawing terminal (response delay ≤ 500ms), reducing the cost of manual pre-inspection and improving the efficiency of defect handling. This intelligent tracking function guides operators to accurately avoid sawing in defective areas, reducing manual judgment time by 70%, and significantly reducing the rework rate caused by mixed steel from 8.5% to 0.9%.

[0120] (5) Construct standardized data interfaces (JSON / XML) and dynamic mapping tables, integrate multi-source data from intelligent tracking of the cooling bed, achieve millisecond-level efficient synchronization with upstream and downstream systems (MES / PLC), break down data silos between the cooling bed and downstream systems, and achieve full-process collaboration. This full-process data integration mechanism makes the error rate of production planning and scheduling close to 0, and improves efficiency by 40 times compared with traditional manual approval. Attached Figure Description

[0121] Figure 1 This is an architecture diagram of a full-process intelligent tracking system for steel pipes based on multimodal perception fusion disclosed in this invention.

[0122] Figure 2 A schematic diagram showing the deployment of video acquisition equipment for an intelligent tracking system.

[0123] Figure 3 This is a schematic diagram of the camera layout for key production stages during the process from the cooling bed to the material frame.

[0124] Figure 4 This is a diagram of the vision-signal collaboration and data fusion architecture.

[0125] Figure 5 This is a hardware deployment topology diagram of a steel pipe end-to-end intelligent tracking system based on multimodal perception fusion.

[0126] Figure 6 This diagram illustrates the deployment locations of industrial cameras and sensor devices.

[0127] Figure 7 This is a schematic diagram of an intelligent software network architecture used in this invention.

[0128] Figure 8 This is a structural diagram of the system operation and maintenance monitoring framework. Detailed Implementation

[0129] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Steps S1, S2… in the described embodiments of the present invention do not limit the unique execution steps of the present invention, and modules M1, M2… do not limit the unique construction method of the system of the present invention; the various models, simulation environments, and software described in the present invention are not considered as the only limiting methods of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0130] In this invention, computer device / equipment / system refers to a related entity applied to a computer, such as hardware, a combination of hardware and software, software, or software in execution. More specifically, for example, software includes, but is not limited to, a process running on a processor, a processor, an object, executable software, an execution thread, a program, and / or a computer. Furthermore, an application program or script running on a server, and the server itself, can also be software. One or more software programs may be in an execution process and / or thread, and the software may be localized on one computer and / or distributed across two or more computers, and may be run on various computer-readable media.

[0131] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0132] In a first aspect, the present invention provides an embodiment of a steel pipe end-to-end intelligent tracking system based on multimodal perception fusion, such as... Figure 1 As shown, the core modules of the system include: a visual perception module, a branch-by-branch tracking and decision-making module, and a control interaction module. The modules are interconnected through a high-speed industrial Ethernet (PROFINET protocol, 1Gbps, latency ≤50ms) and a real-time communication link (TCP / IP protocol, packet loss rate ≤0.1%), forming a low-latency, high-precision, and robust multimodal intelligent tracking closed-loop system.

[0133] Intelligent tracking systems can be built through the following steps:

[0134] S1. Construct a visual perception module (video system): as the "eyes", it collects the appearance features, spatial position, and motion status of the steel pipe;

[0135] The hardware includes an industrial camera (40-megapixel area array in the cold bed area and 2048-pixel line array in the track area), an industrial light source, an image acquisition card, and an NVR; the software includes image preprocessing, target detection, and feature extraction algorithms, with an MTBF ≥ 10,000 hours.

[0136] This module serves as the foundational data source for multimodal perception fusion. Its functionality is deeply integrated into the entire steel pipe tracking process (from initial identification at the cooling bed entrance, to cutting positioning in the sawing area, and finally to warehousing confirmation at the material frame terminal), providing core visual information support for the pipe-by-pipe tracking and decision-making module. Specifically, it is constructed through the following three steps:

[0137] S11, High-resolution video acquisition and environmental adaptation;

[0138] like Figure 2 As shown, industrial cameras are deployed at key nodes such as the cooling bed inlet, track bifurcation point, roller cutter outlet, and material frame drop point: Cooling bed inlet (40-megapixel area array camera, coaxial light source, vertically shooting the end face); Track bifurcation point (20-megapixel area array camera, strip light source, tilted 30° to cover the dual tracks); Roller cutter outlet (2048-pixel line array camera, line light source, vertical roller conveyor); Material frame drop point (5-megapixel area array camera, ring light source, vertical shooting).

[0139] All cameras integrate automatic exposure, wide dynamic range (120dB), autofocus, adaptive white balance, strong light suppression, and infrared night vision functions, adapting to day and night (0~15000 lux), smoke and dust (PM10≤100mg / m³), strong reflection (reflectivity ≥80%), rain and fog (humidity ≤95%) environments, ensuring images without ghosting (≤1 pixel), high contrast (≥30:1), and clear details (resolving 0.1mm scratches).

[0140] S12. Video storage and preprocessing;

[0141] The video stream is transmitted to the NVR (supporting 64-channel storage) via Gigabit Ethernet, using H.265 / H.265+ encoding, saving 50% of storage and 40% of bandwidth.

[0142] Video preprocessing includes four steps: Gaussian filtering (to remove Gaussian noise) + median filtering (to remove salt-and-pepper noise); grayscale conversion (weighted averaging to reduce data volume); adaptive histogram equalization (to increase contrast by ≥50%); and ROI extraction (to remove background and reduce data volume by ≥60%). Preprocessing latency is ≤50ms.

[0143] S13. Intelligent recognition and feature extraction based on deep learning;

[0144] The pre-processed video stream (with frame rates of 30fps, 60fps, and 90fps depending on scene requirements) is input in real-time to a dedicated image recognition processing unit (deployed on a GPU-accelerated server, configured with dual NVIDIA RTX 4090 graphics cards, each with 24GB of VRAM). This unit runs deep learning-based object detection, tracking, and feature extraction algorithms to achieve precise positioning, continuous tracking, and real-time extraction of key visual features for each steel pipe. The specific algorithm flow and functions are as follows:

[0145] S131. Target Detection Algorithm: An end-to-end vision model suitable for real-time scenarios is adopted (this model is trained on 100,000 images of steel pipe production scenes, including 10 types of targets such as steel pipes, defects, and steel pipe surface features, with 1000 training iterations and an mAP (mean accuracy) of 98.5%). Target detection is performed on the preprocessed images, outputting the bounding box coordinates of the steel pipe (accuracy ±1 pixel), category (e.g., "steel pipe" or "defect"), and confidence score (the confidence threshold is set to 0.8; detection results below the threshold are considered false detections). For steel pipe surface defects (e.g., cracks, dents, excessive oxide scale), the model can further output the defect type (12 common defects), location (axial coordinates with the steel pipe head as the origin, accuracy ±1 mm), and dimensions (length, width, and depth, accuracy ±0.1 mm), providing data support for subsequent defect binding and sawing guidance.

[0146] S132. Target Tracking Algorithm: A real-time multi-target tracking algorithm is adopted. Based on the steel pipe bounding boxes and features output by the real-time visual model, a unique tracking ID is assigned to each steel pipe (the tracking ID is generated after the steel pipe enters the tracking area and is cancelled after leaving the tracking area), realizing continuous tracking of multiple steel pipes. This algorithm predicts the position of the steel pipe in the next frame through Kalman filtering and combines appearance feature similarity matching (using cosine similarity calculation, with a similarity threshold set to 0.7) to solve tracking stability problems in scenarios such as steel pipe occlusion (normal tracking is possible when the occlusion area is ≤40%), intersection (accurate differentiation is possible when steel pipes cross each other during transmission), and entering / leaving the field of view (seamless tracking is possible when steel pipes enter / leave the camera's field of view). The tracking success rate is ≥99%, and the tracking frame rate is ≥30fps, meeting the real-time tracking requirements of high-speed production lines.

[0147] S133. Feature Extraction Algorithm: A Transformer-based feature extraction network is used to extract key visual features of the steel pipe, including: ① Outer contour features: The outer circle contour and inner hole contour of the steel pipe are extracted using an edge detection algorithm (Canny algorithm, edge detection accuracy ±0.5 pixels), and the dimensional parameters such as the diameter (accuracy ±0.1mm) and roundness (roundness error ≤0.05) are calculated; ② Surface texture features: The gray-level co-occurrence matrix algorithm is used to extract the texture features of the steel pipe surface (such as energy, entropy, contrast, etc., 16-dimensional features). These features are unique and can be used as the "texture fingerprint" of the steel pipe; ③ End face features: For scenarios where the end face needs to be collected, such as the cooling bed inlet, a convolutional neural network (CNN) is used to extract 200+ microscopic feature points of the end face (such as scratch distribution, oxide scale spot position, inner hole edge details), generating a unique "end face fingerprint code" (feature vector dimension of 512 dimensions) for redundant verification of the steel pipe's identity.

[0148] The extracted visual features (such as position, speed, shape, and defect information) are transmitted in real time to the tracking and decision-making module via high-speed industrial Ethernet. The transmission frequency is consistent with the camera acquisition frequency (30fps / 60fps / 90fps). At the same time, it is compared and fused in real time with PLC signals (such as roller speed and switch status) and sensor signals (such as photoelectric switches and encoders) (fusion delay ≤10ms) to jointly drive the tracking decision and ensure the accuracy and reliability of the tracking results.

[0149] To further enhance the robustness and fault tolerance of the tracking system, end-face feature recognition technology is specifically introduced as a key redundant verification mode to address extreme scenarios such as ambiguous pipe numbers, obstructions, or abnormal signals. The specific implementation logic is as follows:

[0150] First, high-resolution area array cameras, coupled with coaxial light sources, are strategically deployed at key points such as the cooling bed inlet and after cutting, ensuring the cameras are directly facing the steel pipe end face. The directional illumination from the coaxial light source significantly enhances the contrast of microscopic features such as the circular outline, hole structure, oxide scale texture, and scratches on the end face, avoiding feature blurring caused by ambient light interference and providing a clear image foundation for subsequent feature extraction.

[0151] Secondly, a multimodal feature fusion algorithm combining traditional image processing and deep learning is used to construct a unique identifier for the steel pipe. First, basic features are selected using traditional image processing techniques (such as contour extraction algorithms to obtain the inner and outer circular contours of the end face and texture analysis algorithms to uncover surface texture patterns). Then, a deep learning feature encoding network is used to deeply optimize and enhance the dimensionality of these basic features, ultimately generating a high-dimensional, robust, and unique end face feature vector. This vector serves as the "end face fingerprint" of the steel pipe, becoming a core identity credential in addition to its serial number.

[0152] Finally, through S=(f a ⋅f b ) / (‖f a ‖⋅‖f b The similarity algorithm (similarity = f(feature vector_1, feature vector_2), with a value range of 0~1, the closer to 1 the higher the similarity) is used to achieve redundant verification and identity correction of end face features. It is mainly applied in two scenarios: First, cross-node identity association verification, for example, comparing the end face feature vector of the steel pipe collected at the entrance of the cooling bed with the end face feature vector of the cut steel pipe segment to help verify the association between the segment and the parent pipe and supplement the reliability of the number association; Second, identity repair in abnormal scenarios, when the steel pipe number is lost, misaligned, or visual tracking is temporarily interrupted due to occlusion, the historical end face fingerprint code is retrieved and compared with the currently collected end face feature vector to quickly repair the tracking chain, effectively avoiding tracking chain breakage or misalignment caused by the failure of a single information source, and fully demonstrating the technical advantages of multimodal and multi-source fusion.

[0153] S2. Construct a pipe-by-pipe tracking and decision-making module (steel pipe-by-pipe tracking system): As the "brain", it integrates visual data, PLC signals, encoder signals, and sensor data to achieve position determination (accuracy ±2mm), full life cycle management of numbering, defect binding, cutting calculation, and flow direction decision-making;

[0154] Deployed on an industrial server (Intel Xeon Gold 6330 CPU, 128GB memory, dual GPUs), it supports 128-way parallel tracing with a response time of ≤100ms.

[0155] This module is the core engine of the end-to-end intelligent tracking. By deeply integrating visual perception information with multi-source industrial signals (PLC control signals, sensor signals, production scheduling information), it establishes a closed-loop workflow of "data acquisition - fusion processing - decision output - feedback optimization" to realize the management of each steel pipe from entering the cooling bed to the feeding frame, and the entire life cycle.

[0156] like Figure 4 As shown, the methods for coordinating visual signals and device signals include:

[0157] (1) Time synchronization mechanism: Establish a unified time reference to ensure that the visual recognition results and PLC signals correspond accurately in the time dimension.

[0158] (2) Spatial mapping mechanism: The pixel coordinates of visual recognition are accurately mapped to the physical coordinates of the actual production line to achieve seamless integration between virtual and reality.

[0159] (3) Logic verification mechanism: Through multiple logic verifications, the consistency between the visual recognition results and the device signals is ensured, thereby improving the reliability of tracking.

[0160] Data fusion algorithms include:

[0161] (1) Kalman filter algorithm: used for dynamic estimation and prediction of the position of steel pipe, and can maintain the continuity of tracking even when the vision is temporarily blocked.

[0162] (2) Bayesian inference algorithm: Based on prior knowledge and real-time observation data, probabilistic inference is performed on the identity and state of the steel pipe.

[0163] (3) Multimodal feature fusion algorithm: It integrates multiple features such as visual features, geometric features, and motion features to improve the accuracy of recognition.

[0164] Based on the key nodes in the steel pipe production process, the functions of this module are divided into the following core links according to the process nodes:

[0165] S21, Cold Bed Entry: Identity Binding and Initial Positioning (Integration: Visual + Production Data)

[0166] Figure 3 The diagram shows the connection relationships and data flow between the camera, encoder, and MES system. The instant the steel pipe enters the cooling bed receiving trough from the tensioning mill outlet (triggered by a photoelectric switch, trigger response time ≤10ms), the high-resolution industrial camera (40-megapixel area array camera) deployed at the cooling bed inlet immediately triggers to capture the first frame image (trigger delay ≤1ms). The image is transmitted to the vision recognition processing unit via gigabit Ethernet (transmission delay ≤20ms). The vision recognition unit first extracts the initial position coordinates of the steel pipe using an edge detection algorithm (Canny algorithm) (with the cooling bed inlet as the origin, X-axis along the roller conveyor direction, Y-axis perpendicular to the roller conveyor direction, position accuracy ±1mm), estimates the length of the steel pipe using a shape contour analysis algorithm (estimation error ≤5mm), and preliminarily identifies obvious defects on the steel pipe surface (such as cracks ≥10mm in length) using a surface defect detection algorithm, extracting the location and type information of the defects.

[0167] Meanwhile, the individual tracking and decision-making module obtains the precise production data of the steel pipe from the production scheduling system (MES / Level 2) through a standardized data interface (OPC UA protocol), including: the actual length of the steel pipe (measured by a laser length measuring instrument with an accuracy of ±1mm), known defect information (such as internal defects detected during hot rolling), production batch number (such as "HR-20240827-01"), steel grade (such as 20# steel), and specifications (such as Φ159mm×10mm). The system automatically associates and binds the visual features of the steel pipe (initial position, estimated length, surface defects) extracted by the visual perception module with the precise production data of the MES system. It generates a unique initial number for the steel pipe using the rule of "production line number - date - serial number" (e.g., "P-20240827-0001", where "P" represents the tracking system after cooling, "20240827" is the date, and "0001" is the serial number for that day). This number will serve as the core identification for tracking the steel pipe throughout the entire process and will be used in all subsequent processes.

[0168] To improve the predictive capability of subsequent tracking, based on multimodal position data (including the initial position recognized by vision and the roller conveyor speed acquired by PLC), the short-term position of the steel pipe is predicted using the following formula. The prediction result will serve as the predictive basis for subsequent node tracking, facilitating the advance adjustment of camera acquisition parameters and tracking algorithm parameters:

[0169] P t =P0+v×(t-t0)

[0170] Where: P t P0 is the predicted position of the steel pipe at the current time t (unit: m), t0 is the initial detection time, P0 is the initial position of the steel pipe at the initial detection time t0 (unit: m), v is the real-time transmission speed of the roller conveyor (collected by encoder, unit: m / s, accuracy ±0.01m / s), and t-t0 is the time interval from the initial detection to the current time (unit: s).

[0171] For example, if the initial position of the steel pipe is P0 = 5m, the initial detection time is t0 = 10:00:00, and the roller speed is v = 1.2m / s, then at t = 10:00:05, the predicted position of the steel pipe is P... t =5 + 1.2×(5 - 0)=11m. This predicted value will be compared with the actual position of the subsequent visual inspection. If the deviation exceeds 50mm, speed calibration will be triggered to ensure the accuracy of position prediction.

[0172] S22. Image Preprocessing and Feature Refinement (Enhanced Visual Processing)

[0173] Advanced preprocessing (denoising, enhancement, distortion correction) and feature refinement (precise diameter and angle measurement and end-face feature extraction based on machine learning) are performed on images of key nodes such as the cooling bed inlet to ensure highly accurate and reliable visual information provided for tracking decisions. In determining the exit position from the cooling bed, the system integrates visual and PLC positioning signals: real-time comparison is made between the visually tracked / predicted position of the steel pipe and the positioning signal from the PLC at the cooling bed exit. When the visual system confirms that the steel pipe has reached the preset exit position from the cooling bed and receives the positioning signal from the PLC, the system precisely triggers, reduces the cooling bed queue count, officially activates the steel pipe number, and synchronizes the associated defect information to downstream processes.

[0174] The track diversion confirmation integrates vision, sensors, and PLC commands: after exiting the cooling bed, the steel pipe enters the left and right track bifurcation points. The system combines the trigger signals from photoelectric sensors at the track bifurcation points, the actual flow direction of the steel pipe captured by the camera, and the preset diversion commands issued by the PLC (based on the steel pipe properties) to verify the flow direction. If the actual flow direction (vision + sensor) matches the preset command (PLC), the steel pipe position information is updated; if they do not match, an alarm is triggered and an anomaly is marked to prevent misflow.

[0175] During the roller cutting process, precise triggering, segmented calculation, and numbering expansion are deeply integrated with vision, cutting signals, and PLC speed: the system integrates the position of the steel pipe head tracked by vision, the precise linear speed of the roller conveyor provided by the PLC, and the trigger signal from the cutting machine to accurately calculate the moment when the steel pipe head reaches the cutting point. Based on the original steel pipe length estimated by production data / vision and the preset cutting length, the system intelligently calculates the number of cutting segments n=L. original / L preset ±δ (Number of cutting segments = round(Original steel pipe length / Preset cutting length + Allowable error correction coefficient)); where, n, number of cutting segments; L original Original steel pipe length; L preset δ is the preset cutting length; δ is the allowable error correction coefficient. This calculation dynamically references the actual length measured visually and the allowable process error. At the moment cutting is triggered, the system automatically expands the original steel pipe number to a new number with a segment suffix (e.g., "P-202405001-1"), establishing a parent-child relationship. After cutting is completed, the camera immediately verifies the number of segmented steel pipes to ensure that the actual number matches the calculated number of segments; any discrepancies are automatically marked "pending re-inspection".

[0176] The post-cutting diversion and queue update system integrates vision, PLC commands, and sensors: the cut steel pipe segments are diverted according to preset rules (length, quality grade). At the track exit, the system uses cameras and photoelectric sensors to confirm the actual flow direction of each steel pipe segment, and performs secondary verification with the PLC's diversion commands, dynamically updating the queue number and position information of the steel pipe segments to ensure accurate and traceable flow direction.

[0177] At the material frame terminal, final confirmation and information push integrate visual and PLC position signals. When the steel pipe (segment) reaches the material frame's dropping position, the system combines the dropping completion signal from the PLC, the image of the steel pipe entering the frame captured by the terminal camera, and the serial number (if visible) to perform final position and identity confirmation. The system proactively pushes the binding defect information of the steel pipe (segment) to the sawing operator's interface, intelligently guiding the cutting position to avoid the defective area and improving product quality.

[0178] S3. Construct a control interaction module (control system interaction module): As the "nerve center", it communicates bidirectionally with the underlying PLC (Siemens S7-1500, etc.) and the host computer (MES / ERP). It receives equipment signal feedback to the tracking module, sends tracking / alarm commands to the PLC, and synchronizes data to the MES. It adopts an industrial communication gateway with a transmission reliability of ≥99.99%.

[0179] This module supports the following functions:

[0180] (1) Real-time signal exchange: The PLC collects signals such as roller speed and position switch, and compares them with the video recognition results to correct the position judgment.

[0181] (2) Data standardization: unify the numbering system of different systems through the identification code mapping table.

[0182] (3) Data synchronization mechanism: Real-time transmission, batch synchronization or cache push can be selected to ensure that data can still be transmitted on time in the event of network fluctuations.

[0183] (4) System integration: Supports integration with MES and ERP systems to achieve closed-loop management of production data, quality inspection results and label printing.

[0184] This module also includes an efficient data flow management mechanism to ensure smooth data flow between system components and to perform real-time data flow processing and data quality assurance.

[0185] S31. Real-time data stream processing:

[0186] (1) Streaming data processing: Advanced streaming data processing technologies such as Apache Kafka are used to realize real-time data transmission and processing.

[0187] (2) Data caching mechanism: Set up data caching at key nodes to ensure that data is not lost when the network fluctuates or the system is busy.

[0188] (3) Data compression and optimization: Intelligent compression of image and signal data to reduce storage and transmission overhead while ensuring quality.

[0189] S32. Data Quality Assurance:

[0190] (1) Data verification mechanism: All input data are verified for integrity and consistency to ensure data quality.

[0191] (2) Abnormal data processing: Establish an abnormal data detection and processing mechanism to automatically identify and process abnormal data.

[0192] (3) Data backup and recovery: Establish a sound data backup and recovery mechanism to ensure data security.

[0193] Secondly, based on the first aspect, the present invention further provides a hardware deployment embodiment of a steel pipe end-to-end intelligent tracking system based on multimodal perception fusion, such as... Figure 5 As shown in Table 1, the hardware device configuration and deployment list is as follows.

[0194] Table 1:

[0195]

[0196] like Figure 6 As shown, this invention employs a layered hardware deployment, configuring sensing devices according to the steel pipe flow path to construct a full-process intelligent sensing network. For the steel pipe end-face feature recognition function, end-face imaging devices (coaxial light / ring backlight) and sensor devices are deployed at key nodes to capture microscopic features and generate a "physical fingerprint," achieving multimodal identity recognition. Specifically:

[0197] Cooling bed inlet monitoring point: A high-definition camera is deployed at the location where the steel pipe enters the receiving trough to capture the first frame image of the steel pipe and perform initial identification binding. The camera at this location is equipped with a protective cover and a dedicated light source to ensure stable operation in high-temperature and dusty environments.

[0198] Cooling bed exit monitoring point: A monitoring device is installed at the position where the steel pipe leaves the cooling bed and enters the longitudinal track to confirm the steel pipe's exit status from the cooling bed and the selected path.

[0199] Longitudinal track bifurcation monitoring point: Deploy cameras at the bifurcation points of the left and right tracks, and combine them with PLC signals to confirm the specific flow direction of the steel pipe and prevent path judgment errors.

[0200] Roller cutting monitoring point: A dedicated camera is set at the roller cutting position to monitor the cutting process in real time and verify the number of steel pipe segments after cutting.

[0201] Material box inlet monitoring point: such as Figure 3 As shown, a monitoring device is set up at the final confirmation point before the steel pipe enters the material frame to perform final identity verification and location confirmation.

[0202] In terms of network architecture, such as Figure 7As shown, this invention adopts a modular and scalable software architecture design, which enables the system to operate efficiently and be easily maintained.

[0203] This invention employs a distributed processing architecture, deploying edge computing devices at edge computing nodes to perform preliminary preprocessing and feature extraction on images from each monitoring point, effectively reducing the network transmission burden. The central processing unit, equipped with a high-performance GPU, is responsible for executing complex deep learning inference and data fusion tasks. The cloud storage and analysis stage analyzes historical data and reports, providing powerful data mining and trend analysis capabilities.

[0204] Thirdly, this invention provides a system operation and maintenance monitoring framework to monitor the normal operation of a steel pipe end-to-end intelligent tracking system based on multimodal perception fusion. The monitoring framework structure is as follows: Figure 8 As shown.

[0205] Use the core evaluation metrics listed in Table 2 for operation and maintenance monitoring.

[0206] Table 2:

[0207]

[0208] Compared with traditional technical solutions, the present invention has made improvements in several aspects, resulting in significant technical effects. Specific comparisons are shown in Table 3.

[0209] Table 3:

[0210]

[0211] The economic benefits of using this system are shown in Table 4.

[0212] Table 4:

[0213]

[0214] The investment recovery period for this system is approximately 0.14 years, with an investment of 3.2 million yuan and an annual return of 23.5 million yuan.

Claims

1. A full-process intelligent tracking system for steel pipes based on multimodal perception fusion, characterized in that, Includes the following modules: M1, Visual Perception Module: Collects the appearance features, spatial position, and movement status of the steel pipe throughout the entire steel pipe tracking process, including the cooling bed inlet, sawing area, and material frame terminal; specifically includes the following modules: M11, High-resolution video acquisition and environmental adaptation module; M12, video storage and preprocessing module; M13, Deep Learning-Based Intelligent Recognition and Feature Extraction Module: including deep learning-based target detection, target tracking, and feature extraction; M2, Component Tracking and Decision Module: By deeply integrating visual perception information and multi-source industrial signals, a closed-loop workflow is established, including data acquisition, fusion processing, decision output, and feedback optimization. The processing of core process nodes includes: Steel pipe number: A unique initial number is generated at the inlet of the cooling bed for this steel pipe; Flow direction verification: The flow direction of the steel pipe is verified at the junction of the track bifurcation. Cutting and segmentation: During the roller cutting process, the number of cutting segments is calculated, and the parent-child relationship of the steel pipe numbering is established; Flow diversion verification: Verify the actual flow direction of the cut steel pipe segments; Final confirmation: Confirm the identity and defect information of the steel pipes arriving at the material frame; M3, Control Interaction Module: It communicates bidirectionally with the underlying PLC and the upper MES / ERP computer, receives equipment signal feedback to the tracking module, sends tracking / alarm commands to the PLC, and synchronizes data to the MES; The modules are interconnected via high-speed industrial Ethernet, which aggregates the quality inspection results of each process, issues early warnings or rejection signals for abnormal steel pipes, and provides feedback for the optimization of subsequent processes, thus realizing an intelligent closed loop of quality control.

2. The system according to claim 1, characterized in that, The M11 module acquires video via an industrial camera, specifically: Deploy industrial cameras at key nodes including the cooling bed inlet, track bifurcation point, roller cutter outlet, and material frame drop point; The industrial camera integrates automatic exposure, 120dB wide dynamic range, autofocus, adaptive white balance, strong light suppression, and infrared night vision functions.

3. The system according to claim 1, characterized in that, The specific working mode of module M12 includes: The video stream is transmitted to the NVR via Gigabit Ethernet, using H.265 / H.265+ encoding; Video preprocessing includes: Gaussian filtering to remove Gaussian noise and median filtering to remove salt-and-pepper noise, grayscale conversion, adaptive histogram equalization, and ROI extraction.

4. The system according to claim 1, characterized in that, In module M13, the specific method for target detection is as follows: an end-to-end vision model suitable for real-time scenarios is used to perform target detection on the preprocessed image, and the bounding box coordinates, category, and confidence level of the steel pipe are output; the surface defects of the steel pipe are further output with the type, location, and size of the defects.

5. The system according to claim 1, characterized in that, The specific method for target tracking in module M13 is as follows: A multi-target tracking method suitable for real-time scenarios is adopted. Based on the location information and appearance features of the target recognition results, a unique tracking identifier is assigned to each steel pipe. Continuous tracking is achieved through state prediction and appearance matching in the case of occlusion, intersection, and entry / exit from the field of view. To address tracking instability issues such as steel pipe occlusion, intersection, and entry / exit from the camera's field of view, Kalman filtering is used to predict the steel pipe's position in the next frame, while simultaneously performing appearance feature similarity matching.

6. The system according to claim 1, characterized in that, In module M13, the specific method for feature extraction is as follows: a deep learning-based feature extraction network is used to extract key visual features of the steel pipe. These key visual features include at least: shape contour features, surface texture features, a set of microscopic feature points on the end face, and an end face fingerprint code. Specifically: External contour feature extraction: The outer circle contour and inner hole contour of the steel pipe are extracted by edge detection algorithm, and the diameter and roundness dimensional parameters of the steel pipe are calculated; Surface texture feature extraction: The gray-level co-occurrence matrix algorithm is used to obtain unique texture features of the steel pipe surface, which serve as the texture fingerprint of the steel pipe; End face micro-feature point set and end face fingerprint code extraction: More than 200 micro-feature points on the end face are extracted using a convolutional neural network (CNN) to generate a unique feature vector, which serves as the end face fingerprint code for redundant verification of the steel pipe's identity; the micro-feature points include scratch distribution, oxide scale spot location, and inner hole edge details.

7. The system according to claim 6, characterized in that, For scenarios involving unclear, obscured, or abnormal steel pipe numbers, the aforementioned redundancy verification is performed, specifically through the following methods: At the inlet of the cooling bed and the node after cutting, a high-resolution area array camera is deployed with a coaxial light source so that the camera is facing the end face of the steel pipe to take clear pictures. The basic features of the steel pipe are enhanced by using a feature encoding network based on deep learning, generating a unique end-face feature vector, which serves as the end-face fingerprint code of the steel pipe. Redundancy verification and identity correction of facial features are achieved through similarity algorithms.

8. The system according to claim 7, characterized in that, The uses of the redundancy verification include: The feature vector of the steel pipe end face collected at the inlet of the cooling bed is compared with the feature vector of the end face of the cut steel pipe segment to help verify the correlation between the segment and the parent pipe. When the steel pipe number is lost, misaligned, or visual tracking is temporarily interrupted due to obstruction, the historical end-face fingerprint code is retrieved and compared with the currently collected end-face feature vector to repair the tracking chain and avoid tracking chain breakage or misalignment caused by the failure of a single information source.

9. The system according to claim 1, characterized in that, In module M2, the method for deeply fusing visual perception information with multi-source industrial signals includes a collaborative mechanism between visual signals and equipment signals, specifically including: Time synchronization mechanism: Establish a unified time reference so that the visual recognition results correspond to the PLC signals in the time dimension; Spatial mapping mechanism: Mapping the pixel coordinates of visual recognition to the physical coordinates of the actual production line to achieve seamless integration between virtual and reality; Logical verification mechanism: Through multiple logical verifications, the consistency between visual recognition results and device signals is ensured.

10. The system according to claim 1, characterized in that, In module M2, the method for deeply fusing visual perception information and multi-source industrial signals also includes a data fusion algorithm, specifically comprising: Kalman filter algorithm: used for dynamic estimation and prediction of the position of steel pipe, maintaining the continuity of tracking in the case of temporary visual obstruction; Bayesian inference algorithm: Based on prior knowledge and real-time observation data, it performs probabilistic inference on the identity and state of steel pipes; Multimodal feature fusion algorithm: fuses visual features, geometric features, and motion features.

11. The system according to claim 1, characterized in that, In module M2, the processing of core process nodes specifically includes: Steel pipe number: At the entrance of the cooling bed, the visual features extracted by the M1 module are associated and bound with the production data to generate a unique initial number for the steel pipe; Flow direction verification: Flow direction verification is performed by combining the trigger signal of the photoelectric sensor at the track bifurcation point, the actual flow direction of the steel pipe captured by the camera, and the preset diversion command issued by the PLC. Cutting and segmentation: Based on the original steel pipe length estimated by production data / visual estimation and the preset cutting length, the system intelligently calculates the number of cutting segments, automatically expands the original steel pipe number to a new number with a segment suffix, and establishes a parent-child relationship; Diversion verification: At the track exit, cameras and photoelectric sensors are used to confirm the actual flow direction of the cut steel pipe segments and perform secondary verification with the PLC diversion command to dynamically update the queue number and position information of the steel pipe segments. Final confirmation: Combining the material unloading completion signal sent by the PLC, the steel pipe entering the frame captured by the terminal camera and the numbering mark, the final position and identity of the steel pipe that has arrived at the material frame are confirmed, and the binding defect information of the steel pipe is pushed to the sawing operator interface.

12. The system according to claim 11, characterized in that, The processing of the core process nodes also includes the prediction and calibration of the steel pipe position, specifically using the following method: Location prediction is performed using the following formula: P t =P0+v×(t-t0); Among them, P t Let t be the predicted position of the steel pipe at the current time t, t0 be the initial detection time, P0 be the initial position of the steel pipe at the initial detection time t0, v be the real-time transmission speed of the roller conveyor, and t-t0 be the time interval from the initial detection to the current time. The predicted value is compared with the actual position detected by vision. If the deviation exceeds the threshold, speed calibration is triggered.

13. The system according to claim 11, characterized in that, During the processing of the steel pipe number, The visual features include: the initial position of the steel pipe, the estimated length, and surface defects; The production data includes: the actual length of the steel pipe, known defect information, production batch number, steel type, and specifications.

14. The system according to claim 11, characterized in that, The processing of the steel pipe number also includes: real-time comparison of the steel pipe position / predicted position tracked by vision with the cooling bed exit arrival signal issued by the PLC; when it is confirmed that the steel pipe has reached the preset cooling bed exit position, the cooling bed queue count is reduced, the steel pipe number is officially activated, and the bound defect information is synchronized to the downstream process.

15. The system according to claim 11, characterized in that, The specific process of segmenting and cutting includes: Based on the position of the steel pipe head tracked by vision, the linear speed of the roller conveyor provided by the PLC, and the trigger signal of the cutting machine, the time when the steel pipe head reaches the cutting point is calculated. When calculating the number of cutting segments, the actual length measured by dynamic reference vision and the allowable error of the process are taken into account. After the cutting was completed, the camera immediately verified the number of steel pipe segments.

16. The system according to claim 1, characterized in that, The operating mode of module M3 also includes: Industrial communication gateways are used; Data standardization methods include: unifying the numbering system of different systems through an identifier mapping table; Data synchronization mechanisms include real-time transmission, batch synchronization, or cached push, ensuring that data can still be delivered on time even in the event of network fluctuations.

17. The system according to claim 1, characterized in that, The module M3 also includes a real-time data stream processing mechanism, specifically including: Streaming data processing: Apache Kafka streaming data processing technology is used for real-time data transmission and processing; Data caching mechanism: Set up data caching at critical nodes to ensure that data is not lost when the network fluctuates or the system is busy; Data compression and optimization: Intelligent compression of image and signal data.

18. The system according to claim 1, characterized in that, The module M3 also includes a data quality assurance mechanism, specifically including: Data validation mechanism: Perform integrity and consistency checks on all input data; Abnormal data handling: Automatically identify and process abnormal data through abnormal data detection and processing rules; Data backup and recovery mechanism.

19. A hardware deployment device for a multimodal sensing-integrated intelligent tracking system for the entire steel pipe process, to achieve the functions of the system described in any one of claims 1-18, comprising: Industrial cameras: used for image acquisition of steel pipes at the inlet of the cooling bed, providing basic data for high-precision visual inspection; Coaxial lighting unit: Provides illumination compensation for key track nodes, eliminating shadows and reflections; GPU server: Runs steel pipe detection and tracking algorithms in real time and processes multiple high-definition video streams; PROFINET Gateway: Acquires and forwards PLC control signals, and calibrates synchronously with video recognition results.

20. The hardware deployment apparatus according to claim 19, characterized in that, A layered hardware deployment is adopted, with sensing devices configured according to the steel pipe flow path to build a full-process intelligent sensing network, including: Cooling bed entrance monitoring point: Deploy a high-definition camera equipped with a protective cover and a dedicated light source at the location where the steel pipe enters the receiving trough to capture the first frame image of the steel pipe and perform initial identity binding; Cooling bed exit monitoring point: Deploy monitoring equipment at the position where the steel pipe leaves the cooling bed and enters the longitudinal track to confirm the steel pipe's exit status from the cooling bed and path selection; Longitudinal track bifurcation monitoring point: Deploy cameras at the bifurcation of the left and right tracks, and combine them with PLC signals to confirm the specific flow direction of the steel pipe and prevent path judgment errors. Roller cutting monitoring points: Dedicated cameras are deployed at the roller cutting location to monitor the cutting process in real time and verify the number of steel pipe segments after cutting; Material frame inlet monitoring point: Deploy monitoring equipment at the final confirmation point before the steel pipe enters the material frame to perform final identity verification and location confirmation.

21. The hardware deployment apparatus according to claim 19, characterized in that, It also includes distributed processing architecture, specifically including: Edge computing devices are deployed at edge computing nodes to perform preliminary preprocessing and feature extraction on images from each monitoring point; The central processing unit is equipped with a high-performance GPU, which is responsible for performing complex deep learning inference and data fusion tasks; The cloud storage and analysis component is responsible for analyzing historical data and reports, providing data mining and trend analysis.

22. A method for operation and maintenance monitoring of a steel pipe end-to-end intelligent tracking system integrating multimodal perception, to achieve operation and maintenance monitoring of the system described in any one of claims 1-18 or the hardware deployment device described in claim 20 or 21, the method comprising the following steps: S1. Data Acquisition and Diagnosis: Collect data from the equipment layer and analyze it to obtain normal / abnormal health diagnosis results; S2. Conduct operational analysis based on the health diagnosis results to obtain a solution; Specifically, it includes: S21. Perform root cause analysis and fault classification on the diagnostic results of the anomaly, and derive a solution for hardware spare parts scheduling or software container reconstruction. S22. For normal diagnostic results, perform performance optimization analysis and dynamically allocate resources accordingly to achieve GPU load balancing. S3. Use the operations and maintenance dashboard to execute the processing plan and generate an automatic repair report.

23. The operation and maintenance monitoring method according to claim 22, characterized in that, The core evaluation indicators and corresponding evaluation methods used in step S1 include: Tracking accuracy: By comparing the system output with the actual condition of the steel pipe, the proportion of correct matches is statistically analyzed to obtain the defect / number identification accuracy rate; Operational stability: Record the system operation log for the whole year, count unplanned downtime events, and obtain the number of unplanned outages for the year; Data integrity and traceability: Randomly sample historical records to verify whether the entire process information of steel pipes is complete and traceable, and obtain the record integrity rate of steel pipes from the cooling bed to the warehouse; Anti-interference capability: The system is tested in simulated or actual harsh environments to detect the quality of image and signal acquisition and to obtain the stable operating rate under high dust and high temperature environments; Multimodal reliability: In daily operation and special tests, we statistically analyze whether visual recognition and signal logic reach the same conclusion, and whether data from different sources can consistently identify the same steel pipe. We record inconsistencies and analyze the causes to obtain the consistency rate of multimodal fusion.

Images