A large model target detection method for a belt foreign object sorting robot
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG JINING CANAL COAL MINE
- Filing Date
- 2026-04-30
- Publication Date
- 2026-08-04
AI Technical Summary
[0002]在煤炭、矿石、固废等物料输送过程中，原物料中常混有锚杆、道木、铁件、大块矸石、设备脱落件等异物；这些异物若不及时检出并剔除，可能造成皮带撕裂、转载点堵塞、破碎机损坏等重大设备事故，甚至引发安全隐患，导致非计划停机，严重影响生产效率和作业安全
[0012]由于采用如上所述的技术方案，本发明具有如下优越性：
Smart Images
Figure CN122500688A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot target detection technology, and in particular to a method for detecting large-scale targets in a belt-mounted foreign object sorting robot. Background Technology
[0002] During the transportation of materials such as coal, ore, and solid waste, foreign objects such as anchor bolts, timber, iron parts, large pieces of gangue, and detached parts from equipment are often mixed in with the raw materials. If these foreign objects are not detected and removed in time, they may cause major equipment accidents such as belt tearing, blockage at transfer points, and damage to crushers, and may even cause safety hazards, leading to unplanned shutdowns and seriously affecting production efficiency and operational safety.
[0003] Currently, belt conveyor foreign object detection and sorting includes traditional machine vision and sensor detection methods and target detection methods based on small-scale deep learning. Traditional machine vision and sensor detection uses industrial cameras combined with traditional image processing algorithms such as edge detection, shape matching, and threshold segmentation, or is supplemented by metal detectors and ultrasonic sensors. These methods are effective for detecting metal foreign objects of specific shapes under simple, stable lighting and background conditions, but they are not robust enough and have high false alarm and false negative rates when faced with complex working conditions such as large variations in particle size of conveyed materials, severe stacking and overlap, dust and water mist interference, and uneven lighting. They are difficult to adapt to the needs of detecting various types and shapes of foreign objects.
[0004] In recent years, research on target detection methods based on small-scale deep learning has employed convolutional neural networks such as Faster R-CNN, YOLO, and SSD for belt conveyor foreign object identification. Compared with traditional methods, the detection accuracy has been improved. However, all of these methods require a large number of labeled foreign object samples for training, while in actual production lines, samples of some dangerous foreign objects, such as small-sized iron parts and flexible strips, are sparse, resulting in insufficient generalization ability of the model. Furthermore, a single model has limited ability to represent foreign objects of multiple scales and shapes, and it is difficult to integrate contextual information such as material flow state and equipment operating parameters.
[0005] Recently, multimodal large-scale models have been increasingly developed and are gradually shifting towards image and video object detection. Multimodal large-scale models based on the Transformer architecture, such as vision-language large-scale models and general segmentation large-scale models, have demonstrated powerful capabilities in general object detection, zero-shot learning, and cross-modal understanding. These large-scale models can obtain rich visual semantic priors through massive data pre-training, and achieve high-precision detection by combining few-shot or zero-shot prompts, while also possessing stronger anti-interference capabilities and contextual reasoning capabilities. However, in the case of high-speed conveyor belt operation, how to efficiently and in real-time apply large-scale models to foreign object detection and link them in a closed loop with robot sorting actions remains an unsolved technical challenge in the field of industrial vision inspection.
[0006] The invention patent CN119469150B, which describes a target localization method, device, electronic device, and storage medium based on a large model, proposes to determine the target object from the candidate objects based on the object image, the position information of the candidate object in the target image, and the information of the first object. This invention involves multiple rounds of detection and preset targets of the large model, and does not make full use of the model's understanding and target recognition capabilities. It does not fully utilize the model's understanding and target recognition capabilities, but rather fine-tunes the model to understand and select the target in one go. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a large-scale target detection method for belt-mounted foreign object sorting robots, which can effectively improve the efficiency of target detection and reduce the number of detection rounds.
[0008] The technical solution adopted in this invention is: A method for large-scale target detection in a conveyor belt foreign object sorting robot, comprising the following steps: S1: Data acquisition and construction for diverse and complex working conditions; For specific industrial scenarios and edge operating conditions, collect raw images containing various typical foreign objects; S2: Semi-automatic data annotation with human-machine collaboration; An open-source multimodal large model is introduced, and the original images collected by S1 are automatically pre-annotated in combination with the set system prompt words. The output format meets the data annotation format and forms pseudo labels. The background script converts the pseudo labels into the XML / TXT general annotation format with one click and imports them into the annotation software. Manuals make small drag-and-drop corrections to data with loose bounding boxes or false alarms, and quickly form a high-quality fine-tuning dataset. S3: Data fine-tuning training based on joint loss function; The high-quality fine-tuned dataset generated by S2 is input into a multimodal large model, and supervised fine-tuning is performed using the low-rank adaptive fine-tuning technique LoRA for large models; specifically, the pre-trained backbone weight matrix is completely frozen. This means freezing the original parameters; two trainable minimal rank matrices are introduced as a bypass in the multi-head attention mechanism layer of the Transformer network. (d×r dimensionality reduction) and (r×k dimensional increase); The main and bypass outputs are superimposed at the additive fusion node: the forward calculation formula is: for the input activation value LoRA adapted layer output ;in, This represents a low-rank dimension, typically a power of 2; The scaling factor controls the strength of the impact of LoRA updates on the original output; the number of training parameters is reduced from O(dk) to O(r(d+k)); where d and k are the dimensions of the pre-trained weight matrix of the large model, d is the number of rows in the weight matrix, corresponding to the output feature dimension; and k is the number of columns in the weight matrix, corresponding to the input feature dimension. Introducing predictions that include the corresponding foreign object category Predicted coordinates of the corresponding bounding box Joint loss function: By aligning the generalization capability of the multimodal large model with the task of directly outputting detection boxes, a multimodal large model can be obtained that can output the foreign object category and accurate coordinates end-to-end in a single forward propagation. S4: Pruning and quantization reconstruction for edge nodes; S41: Structured pruning: The importance of each channel in the network is evaluated by using the first-order Taylor expansion method as a metric, i.e., a global pruning threshold is set. Or adaptive threshold by layer Importance score or Redundant neuron branches and their corresponding weights and activation maps are completely pruned; after pruning, a short-term fine-tuning recovery is performed to finally obtain the structured pruning model; S42: Post-training quantization: We collected real, unlabeled images of belts as a calibration set. Based on the structured pruning model, we further adopted the INT8 symmetric quantization strategy to reduce the model storage and computational overhead, and obtained a quantized multimodal large model. S5: Real-time deployment at the edge and single forward inference; The quantized multimodal large model is deployed to the edge computing control unit of the belt sorting robot. When the system is running at high speed, the industrial vision acquisition unit, which is vertically mounted above the belt conveyor and includes a high frame rate RGB industrial camera and a 3D depth camera, acquires images in real time through the field of view and inputs them into the multimodal large model frame by frame. The multimodal large model performs a single forward inference and directly outputs the category of all foreign objects in the picture and the coordinates of the two-dimensional pixel bounding box. S6: Dynamic calculation of three-dimensional coordinates and collaborative sorting with robots; The edge computing control unit combines the depth information Z from the 3D depth camera to extract the center pixel coordinates (u,v) of the detection box. Through the pre-calibrated camera intrinsic parameter matrix and the hand-eye extrinsic parameter matrix of the hand-eye calibration model, it performs 2D to 3D coordinate transformation, converting the two-dimensional pixel coordinates output by the multimodal large model into three-dimensional static physical coordinates in the reference coordinate system of the robotic arm. Simultaneously, the system reads the pulse data from the belt speed encoder in real time to obtain the linear speed of the conveyor belt. The total communication time for combining system image processing and inference. Perform dynamic feedforward compensation and calculate the forward compensation amount in the longitudinal direction of belt movement. ; The end effector of the robotic arm receives the final corrected coordinates (X, Y+ΔY, Z), and the kinematic solver at the bottom of the robotic arm pre-plans the trajectory. The following downward gripper smoothly and reliably removes the high-speed moving foreign object from the belt conveyor. S7: Intelligent generation of operation and maintenance reports and loop retrieval for difficult samples; By making full use of the natural language generation and logical analysis capabilities of the multimodal large model, after the end of a single shift of production, the structured data of the detection logs recorded in the background is input into the multimodal large model, which automatically compiles and generates natural language analysis reports and pushes them to the central control room. The system automatically captures difficult samples with low confidence during edge inference and uses them periodically for incremental fine-tuning of multimodal large models to achieve self-iteration and complete the production management closed loop.
[0009] Specifically, in step S1, the original image also includes an image of a similar object to the foreign object, and the similar object image is mixed with the acquired image of the foreign object to prevent overfitting of the model and increase the model's recognition ability.
[0010] Specifically, in step S4, The significance score for the i-th output channel of the first layer of the convolutional layer is calculated as follows: Based on first-order Taylor expansion: ;in, For the joint loss function, For the first The activation output of the i-th channel of the layer.
[0011] Specifically, in step S4, the zero points of symmetric quantization remain consistent before and after quantization, i.e., the zero points correspond. ; ;in: : No. The absolute maximum value of the layer activation value; : Scaling factor; 127: The maximum positive value of an INT8 signed integer, with -128 reserved as an overflow buffer.
[0012] Due to the adoption of the technical solution described above, the present invention has the following advantages: 1. Completely break through the bottleneck of multi-round verification and achieve ultra-fast end-to-end sorting: This invention, through LoRA customized instruction fine-tuning, gives large models the ability to output accurate bounding boxes in a single forward propagation, reducing inference latency to the millisecond level, and truly meeting the industrial-grade real-time sorting needs when the belt is running at high speed.
[0013] 2. Extremely strong scene generalization, subverting the traditional annotation and training mode: Relying on the powerful zero-shot perception capability of large models, it pioneers a cost-reducing mode of automated pre-annotation of large models + manual review and fine-tuning, reducing the time cost of manual drawing frames by more than 80%; Only one fine-tuning is needed to be naturally immune to interference from different lighting, dust occlusion and unknown variants and foreign objects, and say goodbye to the predicament of traditional small models such as YOLO needing to be retrained when changing scenes.
[0014] 3. Successfully achieved ultra-low-cost edge deployment of large models: The innovative approach of combining fine-tuning, pruning, and INT8 quantization to reduce the size of multimodal large models that often require tens of gigabytes of computing memory is compressed into lightweight models and deployed to the local controller of ordinary robots. This completely eliminates the pain points of relying on high-end cloud server computing power and network latency, and ensures data privacy and offline operation capability in industrial settings.
[0015] 4. Forming a deep physical execution closed loop: Deeply integrating cutting-edge AI vision base with industrial robotic arm kinematics, it completes the time difference compensation and three-dimensional coordinate mapping algorithm based on encoder speed feedforward, and opens up the complete physical sorting lifecycle from recognition to grasping and intelligent report feedback. Attached Figure Description
[0016] Figure 1 This is the overall system operation flowchart of the present invention. Detailed Implementation
[0017] The present invention will be further explained and described below with reference to the accompanying drawings and embodiments. However, this should not be construed as limiting the scope of protection of the present invention. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention.
[0018] Combined with appendix Figure 1 The method for large-scale target detection in a conveyor belt foreign object sorting robot, as shown, includes the following steps: S1: Data acquisition and construction for diverse and complex working conditions; For industrial scenarios such as coal washing, ore crushing, or solid waste recycling, original images containing various typical foreign objects such as large pieces of gangue, anchor bolts, waste wire, wood blocks, and detonator wires are collected. To enhance the long-tail recognition capability of the multimodal large model, images of edge conditions such as uneven lighting, high dust concentration, and overlapping foreign objects are introduced for data background. The original images also include images of similar objects to the foreign objects, which are obfuscated with the collected images of foreign objects to prevent overfitting of the model and increase the model's recognition capability.
[0019] S2: Semi-automatic data annotation with human-machine collaboration; By introducing open-source multimodal large models, such as Qwen-VL, and combining them with set system prompts, the original images collected by S1 are automatically pre-annotated. The output format meets the data annotation format and forms pseudo-labels. The background script converts the pseudo-labels into the XML / TXT universal annotation format with one click and imports them into the annotation software. Manually drag and drop minor corrections are made to data with loose bounding boxes or false alarms to quickly form a high-quality fine-tuning dataset.
[0020] S3: Data fine-tuning training based on joint loss function; The high-quality fine-tuned dataset generated by S2 is input into a multimodal large model, and supervised fine-tuning is performed using the low-rank adaptive fine-tuning technique LoRA for large models; specifically, the pre-trained backbone weight matrix is completely frozen. This means freezing most of the original parameters; only two trainable minimal rank matrices are introduced as bypasses in the multi-head attention mechanism layer of the Transformer network. (d×r dimensionality reduction) and (r×k dimensional increase); The main and bypass outputs are superimposed at the additive fusion node: the forward calculation formula is: for the input activation value LoRA adapted layer output ;in, This represents a low-rank dimension, typically a power of 2; The scaling factor controls the strength of the impact of LoRA updates on the original output; the number of training parameters is reduced from O(dk) to O(r(d+k)); where d and k are the dimensions of the pre-trained weight matrix of the large model, d is the number of rows in the weight matrix, corresponding to the output feature dimension; and k is the number of columns in the weight matrix, corresponding to the input feature dimension. To enable large models to have accurate pixel-level regression capabilities, a method is introduced that includes predictions of the corresponding foreign object category. Predicted coordinates of the corresponding bounding box Joint loss function: Aligning the generalization ability of the multimodal large model with the task of directly outputting detection boxes makes the model's inference ability more accurate. Thus, after fine-tuning, a multimodal large model can output the foreign object category and accurate coordinates in a single forward propagation.
[0021] S4: Pruning and quantization reconstruction for edge nodes; S41: Structured pruning: The importance of each channel in the network is evaluated by using the first-order Taylor expansion method as a metric, i.e., a global pruning threshold is set. Or adaptive threshold by layer Importance score or Redundant neuron branches and their corresponding weights and activation maps are completely pruned, rather than being randomly sparsely connected. This maintains the regularity of the network structure, which is beneficial for subsequent parallel computing optimization by hardware accelerators. After pruning, a short-term fine-tuning recovery is performed for 1-3 epochs to repair the accuracy decay caused by channel removal, ultimately resulting in a structured pruned model. The significance score for the i-th output channel of the first layer of the convolutional layer is calculated as follows: based on first-order Taylor expansion: ;in, For the joint loss function, For the first The activation output of the i-th channel of the layer.
[0022] S42: Post-training quantization: We collected real, unlabeled images of belts as a calibration set. Based on the structured pruning model, we further adopted the INT8 symmetric quantization strategy to reduce the model storage and computational overhead, and obtained a quantized multimodal large model. Symmetric quantization ensures that the zeros remain consistent before and after quantization, meaning the zeros correspond. ; ; in: : No. The absolute maximum value of the layer activation value; : Scaling factor; 127: The maximum positive value of an INT8 signed integer, with -128 reserved as an overflow buffer.
[0023] To adapt to the limited GPU memory resources and real-time inference requirements of edge computing control units, this invention adopts a two-stage model compression strategy that combines structured pruning and post-training quantization, achieving lightweight deployment of the model without significantly sacrificing detection accuracy.
[0024] S5: Real-time deployment at the edge and single forward inference; The quantized multimodal large model is deployed to the edge computing control unit of the belt sorting robot. When the system is running at high speed, the industrial vision acquisition unit, which includes a high frame rate RGB industrial camera and a 3D depth camera, is vertically mounted above the belt conveyor to acquire images in real time through the field of view and input them into the multimodal large model frame by frame. The multimodal large model performs a single forward inference and outputs the category of all foreign objects in the picture and the coordinates of the two-dimensional pixel bounding boxes without any delay.
[0025] S6: Dynamic calculation of three-dimensional coordinates and collaborative sorting with robots; The edge computing control unit combines the depth information Z from the 3D depth camera to extract the center pixel coordinates (u,v) of the detection box. Through the pre-calibrated camera intrinsic parameter matrix and the "eye-to-hand" extrinsic parameter matrix of the hand-eye calibration model, it performs 2D to 3D coordinate transformation, converting the two-dimensional pixel coordinates output by the multimodal large model into three-dimensional static physical coordinates in the reference coordinate system of the robotic arm. Simultaneously, the system reads the pulse data from the belt speed encoder in real time to obtain the linear speed of the conveyor belt. The total communication time for combining system image processing and inference. Perform dynamic feedforward compensation and calculate the forward compensation amount in the longitudinal direction of belt movement. ; The end effector of a Delta parallel robot or a six-axis industrial robotic arm receives the final corrected coordinates (X, Y+ΔY,Z). The kinematic solver at the bottom of the robotic arm pre-plans the trajectory, and the following downward gripper smoothly and reliably removes the high-speed moving foreign object from the belt conveyor.
[0026] S7: Intelligent generation of operation and maintenance reports and loop retrieval for difficult samples; By making full use of the natural language generation and logical analysis capabilities of the multimodal big model, after the end of a single shift of production, the structured data of the detection logs recorded in the background, such as the type of foreign object, the frequency of grabbing, and the time distribution, are input into the multimodal big model, which automatically compiles and generates natural language analysis reports, such as equipment wear suggestions and high frequency warnings, and pushes them to the central control room. The system automatically captures difficult samples with low confidence during edge inference and uses them periodically for incremental fine-tuning of multimodal large models to achieve self-iteration and complete the production management closed loop.
[0027] The parts of this invention not described in detail are prior art.
[0028] The embodiments selected herein for the purpose of disclosing the inventive objectives are currently considered suitable; however, it should be understood that the invention is intended to include all variations and modifications of the embodiments that fall within the scope of this concept and invention.
Claims
1. A method for detecting large-scale targets in a conveyor belt foreign object sorting robot, characterized in that: The specific steps are as follows: S1: Data acquisition and construction for diverse and complex working conditions; For specific industrial scenarios and edge operating conditions, collect raw images containing various typical foreign objects; S2: Semi-automatic data annotation with human-machine collaboration; An open-source multimodal large model is introduced, and the raw images collected by S1 are automatically pre-annotated in combination with the set system prompt words. The output format meets the data annotation format and forms pseudo-labels. The background script converts pseudo-tags into the XML / TXT universal annotation format with one click and imports them into the annotation software; Manually drag and drop minor corrections to data with loose borders or false alarms to quickly generate a high-quality fine-tuning dataset; S3: Data fine-tuning training based on joint loss function; The high-quality fine-tuned dataset generated by S2 is input into a multimodal large model, and supervised fine-tuning is performed using the low-rank adaptive fine-tuning technique LoRA for large models; specifically, the pre-trained backbone weight matrix is completely frozen. This means freezing the original parameters; two trainable minimal rank matrices are introduced as a bypass in the multi-head attention mechanism layer of the Transformer network. (d×r dimensionality reduction) and (r×k dimensional increase); The main and bypass outputs are superimposed at the additive fusion node: the forward calculation formula is: for the input activation value LoRA adapted layer output ;in, This represents a low-rank dimension, typically a power of 2; The scaling factor controls the strength of the impact of LoRA updates on the original output; the number of training parameters is reduced from O(dk) to O(r(d+k)); where d and k are the dimensions of the pre-trained weight matrix of the large model, d is the number of rows in the weight matrix, corresponding to the output feature dimension; k is the number of columns in the weight matrix, corresponding to the input feature dimension; Introducing predictions that include the corresponding foreign object category Predicted coordinates of the corresponding bounding box Joint loss function: By aligning the generalization capability of the multimodal large model with the task of directly outputting detection boxes, a multimodal large model can output the foreign object category and precise coordinates in a single forward propagation. S4: Pruning and quantization reconstruction for edge nodes; S41: Structured pruning: The importance of each channel in the network is evaluated by using the first-order Taylor expansion method as a metric, i.e., a global pruning threshold is set. Or adaptive threshold by layer Importance score or Redundant neuron branches and their corresponding weights and activation maps are completely pruned; after pruning, a short-term fine-tuning recovery is performed to finally obtain the structured pruning model; S42: Post-training quantization: We collected real, unlabeled images of belts as a calibration set. Based on the structured pruning model, we further adopted the INT8 symmetric quantization strategy to reduce the model storage and computational overhead, and obtained a quantized multimodal large model. S5: Real-time deployment at the edge and single forward inference; The quantized multimodal large model is deployed to the edge computing control unit of the belt sorting robot. When the system is running at high speed, the industrial vision acquisition unit, which is vertically mounted above the belt conveyor and includes a high frame rate RGB industrial camera and a 3D depth camera, acquires images in real time through the field of view and inputs them into the multimodal large model frame by frame. The multimodal large model performs a single forward inference and directly outputs the category of all foreign objects in the picture and the coordinates of the two-dimensional pixel bounding box. S6: Dynamic calculation of three-dimensional coordinates and collaborative sorting with robots; The edge computing control unit combines the depth information Z from the 3D depth camera to extract the center pixel coordinates (u,v) of the detection box. Through the pre-calibrated camera intrinsic parameter matrix and the hand-eye extrinsic parameter matrix of the hand-eye calibration model, it performs 2D to 3D coordinate transformation, converting the two-dimensional pixel coordinates output by the multimodal large model into three-dimensional static physical coordinates in the reference coordinate system of the robotic arm. Simultaneously, the system reads the pulse data from the belt speed encoder in real time to obtain the linear speed of the conveyor belt. The total communication time for combining system image processing and inference. Perform dynamic feedforward compensation and calculate the forward compensation amount in the longitudinal direction of belt movement. ; The end effector of the robotic arm receives the final corrected coordinates (X, Y+ΔY, Z), and the kinematic solver at the bottom of the robotic arm pre-plans the trajectory. The following downward gripper smoothly and reliably removes the high-speed moving foreign object from the belt conveyor. S7: Intelligent generation of operation and maintenance reports and loop retrieval for difficult samples; By making full use of the natural language generation and logical analysis capabilities of the multimodal large model, after the end of a single shift of production, the structured data of the detection logs recorded in the background is input into the multimodal large model, which automatically compiles and generates natural language analysis reports and pushes them to the central control room. The system automatically captures difficult samples with low confidence during edge inference and uses them periodically for incremental fine-tuning of multimodal large models to achieve self-iteration and complete the production management closed loop.
2. The method for detecting large-scale targets in a conveyor belt foreign object sorting robot according to claim 1, characterized in that: In step S1, the original image also includes an image of a similar object to the foreign object, and the similar object image is obfuscated with the acquired image of the foreign object.
3. The method for detecting large-scale targets in a conveyor belt foreign object sorting robot according to claim 1, characterized in that: In step S4, The significance score for the i-th output channel of the first layer of the convolutional layer is calculated as follows: Based on first-order Taylor expansion: ;in, For the joint loss function, For the first The activation output of the i-th channel of the layer.
4. The method for detecting large-scale targets in a conveyor belt foreign object sorting robot according to claim 1, characterized in that: In step S4, the symmetric quantization ensures that the zero points remain consistent before and after quantization, i.e., the zero points correspond. ; ; in: : No. The absolute maximum value of the layer activation value; : Scaling factor; 127: The maximum positive value of an INT8 signed integer, with -128 reserved as an overflow buffer.