A vision-based intelligent sorting method and system for building materials

Abstract

This invention relates to a vision-based intelligent sorting method and system for building materials. The vision-based intelligent sorting system for building materials includes a multimodal vision module, an intelligent sorting execution module, a deep learning processing module, a rapid calibration module, and a mobile carrier module. The multimodal vision module is connected to the deep learning processing module, and the intelligent sorting execution module is connected to the mobile carrier module. The deep learning processing module is used for data communication with the intelligent sorting execution module and the mobile carrier module. The rapid calibration module is used for data communication with the deep learning processing module and the intelligent sorting execution module. The communication interface of the mobile carrier module is used for data communication with the intelligent sorting execution module and the deep learning processing module. This invention achieves automated grasping and handling of target building materials.

Description

Technical Field

[0001] This invention relates to the technical field of building material sorting, and in particular to a vision-based intelligent sorting method and system for building materials. Background Technology

[0002] With the rapid development of the construction industry, higher demands are being placed on the efficiency of building material sorting and handling. Traditional building material sorting systems mainly rely on manual operation or simple mechanical equipment to complete sorting and handling work. Manual sorting usually relies on the operator's vision and experience to identify and grab materials. This method is inefficient and prone to errors when dealing with large quantities, complex or mixed building materials.

[0003] In automated sorting systems, sorting and classification are commonly performed using robotic arms, conveyor belts, or simple sensors. However, these systems typically rely on simple classification based on characteristics such as color, size, or weight, making it difficult to accurately sort building materials that look similar but have different properties. Summary of the Invention

[0004] To achieve automated grasping and handling of target building materials, this application provides a vision-based intelligent sorting method and system for building materials.

[0005] The above-mentioned objective of this application is achieved through the following technical solution:

[0006] A vision-based intelligent building materials sorting system includes a multimodal vision module, an intelligent sorting execution module, a deep learning processing module, a rapid calibration module, and a mobile carrier module, wherein:

[0007] The multimodal vision module is used to communicate with the deep learning processing module, and the multimodal vision module is connected to the deep learning processing module.

[0008] The intelligent sorting execution module is used for data communication with the deep learning processing module, and the intelligent sorting execution module is connected to the mobile carrier module;

[0009] The deep learning processing module is used to communicate with the multimodal vision module, and the deep learning processing module is used to communicate with the intelligent sorting execution module and the mobile carrier module.

[0010] The rapid calibration module is used to communicate data with the deep learning processing module and the intelligent sorting execution module.

[0011] The communication interface of the mobile carrier module is used for data communication with the intelligent sorting execution module and the deep learning processing module.

[0012] By adopting the above technical solutions, the vision-based intelligent building materials sorting system achieves full automation from image acquisition to grasping and handling through the collaborative work of a multimodal vision module, a deep learning processing module, an intelligent sorting execution module, a rapid calibration module, and a mobile carrier module. The multimodal vision module uses a binocular stereo camera and a ToF depth sensor to jointly acquire images and depth information, generating 3D material distribution data and transmitting it to the deep learning processing module. This module uses the YOLO algorithm, CBAM optimization mechanism, and data augmentation technology to identify and locate building materials. After correction by the rapid calibration module, the identification results are provided to the intelligent sorting execution module, which controls the six-axis robotic arm, vacuum suction cup, and adaptive gripper to perform grasping operations. The mobile carrier module achieves precise movement based on Mecanum wheels and laser SLAM navigation, and dynamically adjusts based on path planning and positioning information. The various modules of the system efficiently transmit data and control commands through communication interfaces, while simultaneously using a remote monitoring system for result evaluation and feedback optimization, continuously adjusting the recognition algorithm and control strategy to improve recognition accuracy, handling efficiency, and environmental adaptability.

[0013] In a preferred embodiment, this application can be further configured such that: the multimodal vision module includes a binocular stereo camera, a ToF depth sensor, a ring light, and an anti-glare curtain assembly; the binocular stereo camera works in conjunction with the ToF depth sensor; and the multimodal vision module is connected to the deep learning processing module.

[0014] By adopting the above technical solution, the multimodal vision module integrates a binocular stereo camera and a ToF depth sensor, achieving collaborative acquisition of image information and depth data, effectively generating high-precision 3D material distribution data. The binocular camera acquires spatial structure based on the parallax principle, while the ToF sensor supplements depth information by measuring time-of-flight; the two complement each other, making the data more comprehensive and three-dimensional. A ring-shaped fill light provides uniform illumination, reduces shadow interference, and improves image clarity; the anti-glare curtain assembly effectively filters strong reflected light, improving imaging quality in complex or strong light environments.

[0015] In a preferred embodiment, the present application may be further configured such that: the intelligent sorting execution module includes a six-axis collaborative robotic arm, a vacuum suction cup, and an adaptive gripper, the adaptive gripper including a pressure sensor and a silicone damping layer; the intelligent sorting execution module is connected to the deep learning processing module; and the intelligent sorting execution module is connected to the mobile carrier module.

[0016] By adopting the above technical solution, the intelligent sorting execution module consists of a six-axis collaborative robotic arm, a vacuum suction cup, and an adaptive gripper, used to achieve precise gripping and handling of different building materials. The robotic arm controls the precise movement of each joint based on position and posture information provided by the deep learning processing module; the vacuum suction cup is suitable for building materials with flat surfaces, ensuring stable adsorption by adjusting the suction force; the adaptive gripper is suitable for irregularly shaped building materials, and its built-in pressure sensor and silicone damping layer automatically adjust the gripping force and angle to prevent slippage and damage. The intelligent sorting execution module is connected to the mobile carrier module to achieve synchronous transmission of information after gripping and planning of the handling path, improving the overall accuracy and efficiency of the operation.

[0017] In a preferred embodiment, this application can be further configured such that: the deep learning processing module includes a YOLO algorithm processing unit, a CBAM module, and a data augmentation unit; the deep learning processing module is connected to the multimodal vision module; the deep learning processing module is connected to the intelligent sorting execution module; and the deep learning processing module is connected to the mobile carrier module.

[0018] By adopting the above technical solution, the deep learning processing module consists of a YOLO algorithm processing unit, a CBAM module, and a data augmentation unit, working together to achieve the identification and optimization of building materials. The YOLO algorithm rapidly detects and classifies 3D material distribution data, the CBAM module utilizes an attention mechanism to optimize the identification results, improving accuracy and positioning precision, and the data augmentation unit enhances the model's generalization ability through diversified training. This module connects with the multimodal vision module, the intelligent sorting execution module, and the mobile carrier module, providing stable data support to ensure the accurate execution of sorting and handling operations.

[0019] In a preferred embodiment, the present application may be further configured such that the fast calibration module includes a nine-point nonlinear calibration unit and a calibration data output interface, wherein the calibration data output interface is connected to the deep learning processing module and the intelligent sorting execution module, respectively.

[0020] By adopting the above technical solution, the rapid calibration module generates calibration data through a nine-point nonlinear calibration unit and transmits the data to the deep learning processing module and the intelligent sorting execution module through the calibration data output interface. The nine-point nonlinear calibration unit generates calibration data for correcting errors and optimizing positioning accuracy by identifying and calculating preset calibration points. The connection of the calibration data output interface ensures that the deep learning processing module and the intelligent sorting execution module can perform real-time adjustments and optimizations based on the calibration data.

[0021] In a preferred embodiment, the present application may be further configured such that: the mobile carrier module includes a Mecanum wheel module, a laser SLAM navigation module, and a communication interface; the Mecanum wheel module works in conjunction with the laser SLAM navigation module to control the movement and positioning of the mobile carrier module; and the communication interface is connected to the deep learning processing module and the intelligent sorting execution module, respectively.

[0022] By adopting the above technical solution, the mobile carrier module achieves omnidirectional movement and precise positioning through the collaborative work of the Mecanum wheel module and the laser SLAM navigation module. The Mecanum wheel module provides flexible multi-directional movement capabilities, including forward, backward, lateral, and rotational movement. The laser SLAM navigation module, through a combination of lidar and inertial measurement unit, achieves real-time scanning and positioning of the environment, thereby generating accurate position information and path planning. Data is processed through a communication interface connected to the deep learning processing module and the intelligent sorting execution module.

[0023] A vision-based intelligent sorting method for building materials, applied to a vision-based intelligent sorting system for building materials, comprising:

[0024] The first image data output by the binocular stereo camera and the second depth data output by the ToF depth sensor are acquired. The first image data and the second depth data are input into the YOLO algorithm processing unit for preliminary identification and positioning to obtain three-dimensional material distribution data.

[0025] The calibration data output by the nine-point nonlinear calibration unit is obtained, and the coordinate correction and error correction are performed on the three-dimensional material distribution data based on the calibration data to obtain the building material positioning data.

[0026] The data augmentation unit is used to perform dataset augmentation training on the building material location data to obtain an augmented material distribution dataset.

[0027] The enhanced material distribution dataset is identified and classified by the YOLO algorithm processing unit to obtain material identification results.

[0028] The material identification result is input into the CBAM module for optimization processing, and the optimized material identification result is output.

[0029] The optimized material identification results are input into the intelligent sorting execution module, which uses the six-axis collaborative robotic arm, the vacuum suction cup, and the adaptive clamp to grasp and transport the target building materials.

[0030] Acquire the motion data output by the Mecanum wheel module and the global positioning data output by the laser SLAM navigation module;

[0031] The motion data and the global positioning data are input into the deep learning processing module to obtain path planning and positioning control information.

[0032] The path planning and positioning control information is input into the intelligent sorting execution module to control the movement and positioning of the mobile carrier module.

[0033] By employing the above technical solutions, and through the collaborative acquisition of images and depth information by a binocular stereo camera and a ToF depth sensor, high-precision 3D material distribution data can be simultaneously acquired for subsequent identification and positioning. A nine-point nonlinear calibration unit provides calibration data to correct identification errors in real time, improving positioning accuracy. The deep learning processing module, composed of the YOLO algorithm, CBAM module, and data augmentation unit, performs rapid identification, feature optimization, and diversified training, enhancing the model's robustness and generalization ability, ensuring consistent identification accuracy in complex environments. The intelligent sorting execution module, through the cooperation of a six-axis robotic arm, vacuum suction cups, and adaptive grippers, achieves stable gripping and handling of different building materials. Mecanum wheels and a laser SLAM navigation module work together to support omnidirectional movement and precise positioning, improving adaptability in confined spaces. All modules interact efficiently through a communication interface. The deep learning processing module and the intelligent sorting execution module collaboratively optimize path and control parameters, while the remote monitoring system continuously provides feedback and trains the model, ensuring the accuracy, stability, and efficiency of the sorting process.

[0034] In a preferred embodiment, this application can be further configured as follows: the data augmentation unit is used to perform dataset augmentation training on the building material positioning data to obtain an augmented material distribution dataset, including:

[0035] The data augmentation unit performs dataset augmentation training on the building material location data based on the Mosaic data augmentation method to obtain the augmented material distribution dataset.

[0036] By adopting the above technical solution, the data augmentation unit performs dataset augmentation training on building material positioning data based on the Mosaic data augmentation method, which can effectively improve the generalization ability and robustness of the recognition model.

[0037] In a preferred embodiment, this application can be further configured as follows: the step of inputting the material identification result into the CBAM module for optimization processing and outputting the optimized material identification result includes:

[0038] The CBAM module optimizes the material recognition results based on channel attention and spatial attention mechanisms, and outputs the optimized material recognition results.

[0039] By adopting the above technical solutions, the CBAM module optimizes the material recognition results through channel attention and spatial attention mechanisms, improving recognition accuracy and positioning precision. The channel attention mechanism extracts the importance of each channel in the feature map through global pooling, weighting key channels and weakening irrelevant information; the spatial attention mechanism further identifies key regions in the image and weights positional features. The combination of these two mechanisms enables joint optimization of the feature map in both channel and spatial dimensions, improving the model's ability to recognize target building materials and its feature extraction performance. The optimized recognition results are clearer and more stable, providing accurate support for subsequent classification and positioning.

[0040] In a preferred embodiment, this application can be further configured such that: the vision-based intelligent sorting method for building materials further includes;

[0041] The remote monitoring system evaluates and provides feedback on the optimized material identification results and the path planning and positioning control information to obtain evaluation feedback results;

[0042] Based on the evaluation feedback results, optimization information is generated, and the optimization information is input into the deep learning processing module and the intelligent sorting execution module for optimization training to obtain the optimized training model and execution control strategy.

[0043] The optimized training model is input into the YOLO algorithm processing unit and the CBAM module to update the recognition algorithm and optimization strategy, thereby obtaining the updated material recognition result. The updated material recognition result is used to improve the recognition accuracy and target positioning accuracy of the YOLO algorithm processing unit and the CBAM module.

[0044] The optimized execution control strategy is input into the intelligent sorting execution module to update the execution control parameters of sorting and handling operations, thereby obtaining updated path control information. The updated path control information is used to optimize the motion path and grasping accuracy of the intelligent sorting execution module.

[0045] By adopting the above technical solution, the remote monitoring system can evaluate and provide feedback on the optimized material identification results and path planning information in real time. It comprehensively analyzes identification accuracy, path deviation, and control effectiveness, generating optimized information which is then input into the deep learning processing module and the intelligent sorting execution module for training and updates, continuously optimizing the identification model and control strategy. The optimized model is loaded into the YOLO algorithm processing unit and the CBAM module to improve identification and positioning accuracy. Simultaneously, the updated control strategy is used to adjust the execution parameters of sorting and handling, improving path planning and grasping accuracy. Through this dynamic evaluation and optimization mechanism, the system achieves high-precision identification and stable control in complex environments.

[0046] In summary, this application includes at least one of the following beneficial technical effects:

[0047] 1. The vision-based intelligent building materials sorting system achieves full automation from image acquisition to grasping and handling through the collaborative work of a multimodal vision module, a deep learning processing module, an intelligent sorting execution module, a rapid calibration module, and a mobile carrier module. The multimodal vision module uses a binocular stereo camera and a ToF depth sensor to jointly acquire images and depth information, generating 3D material distribution data and transmitting it to the deep learning processing module. This module uses the YOLO algorithm, CBAM optimization mechanism, and data augmentation technology to identify and locate building materials. After correction by the rapid calibration module, the identification results are provided to the intelligent sorting execution module, which controls a six-axis robotic arm, vacuum suction cups, and adaptive grippers to perform grasping operations. The mobile carrier module achieves precise movement based on Mecanum wheels and laser SLAM navigation, and dynamically adjusts based on path planning and positioning information. The various modules of the system efficiently transmit data and control commands through communication interfaces, while simultaneously using a remote monitoring system for result evaluation and feedback optimization, continuously adjusting the recognition algorithm and control strategy to improve recognition accuracy, handling efficiency, and environmental adaptability.

[0048] 2. Through the collaborative acquisition of binocular stereo cameras and ToF depth sensors, image and depth information can be acquired simultaneously, generating high-precision 3D material distribution data for subsequent identification and positioning. A nine-point nonlinear calibration unit provides calibration data to correct identification errors in real time, improving positioning accuracy. The deep learning processing module, composed of the YOLO algorithm, CBAM module, and data augmentation unit, performs rapid identification, feature optimization, and diversified training, enhancing the model's robustness and generalization ability, ensuring consistent identification accuracy in complex environments. The intelligent sorting execution module, through a six-axis robotic arm, vacuum suction cups, and adaptive grippers, achieves stable gripping and handling of different building materials. Mecanum wheels and a laser SLAM navigation module work together to support omnidirectional movement and precise positioning, improving adaptability in confined spaces. All modules interact efficiently via communication interfaces. The deep learning processing module and the intelligent sorting execution module collaboratively optimize path and control parameters, while the remote monitoring system continuously provides feedback and trains the model, ensuring the accuracy, stability, and efficiency of the sorting process.

[0049] 3. The remote monitoring system can provide real-time evaluation and feedback on the optimized material identification results and path planning information. It comprehensively analyzes identification accuracy, path deviation, and control effectiveness, generating optimized information which is then input into the deep learning processing module and the intelligent sorting execution module for training and updates, continuously optimizing the identification model and control strategy. The optimized model is loaded into the YOLO algorithm processing unit and the CBAM module to improve identification and positioning accuracy. Simultaneously, the updated control strategy is used to adjust the execution parameters of sorting and handling, improving path planning and grasping accuracy. Through this dynamic evaluation and optimization mechanism, the system achieves high-precision identification and stable control in complex environments. Attached Figure Description

[0050] Figure 1 This is a flowchart of a vision-based intelligent building materials sorting system according to one embodiment of this application.

[0051] Figure 2 This is a flowchart of a vision-based intelligent sorting method for building materials in one embodiment of this application;

[0052] Figure 3 This is a flowchart illustrating the implementation of step S30 in a vision-based intelligent building materials sorting method according to an embodiment of this application.

[0053] Figure 4 This is a flowchart illustrating the implementation of step S50 in a vision-based intelligent building materials sorting method according to an embodiment of this application.

[0054] Figure 5 This is a flowchart illustrating the implementation of step S90 in a vision-based intelligent building materials sorting method according to an embodiment of this application.

[0055] Explanation of reference numerals in the attached figures:

[0056] 1. Multimodal vision module; 2. Intelligent sorting execution module; 3. Deep learning processing module; 4. Rapid calibration module; 5. Mobile carrier module. Detailed Implementation

[0057] The present application will be further described in detail below with reference to the accompanying drawings.

[0058] In one embodiment, such as Figure 1 As shown, this application discloses a vision-based intelligent building materials sorting system. The system includes a multimodal vision module 1, an intelligent sorting execution module 2, a deep learning processing module 3, a rapid calibration module 4, and a mobile carrying module 5, wherein:

[0059] The multimodal vision module 1 is used to communicate with the deep learning processing module 3, and the multimodal vision module 1 is connected to the deep learning processing module 3;

[0060] The intelligent sorting execution module 2 is used to communicate with the deep learning processing module 3, and the intelligent sorting execution module 2 is connected to the mobile carrier module 5;

[0061] The deep learning processing module 3 is used to communicate with the multimodal vision module 1, and the deep learning processing module 3 is used to communicate with the intelligent sorting execution module 2 and the mobile carrier module 5.

[0062] The rapid calibration module 4 is used to communicate data with the deep learning processing module 3 and the intelligent sorting execution module 2.

[0063] The communication interface of the mobile carrier module 5 is used to communicate with the intelligent sorting execution module 2 and the deep learning processing module 3.

[0064] Specifically, the multimodal vision module 1 is used for data communication with the deep learning processing module 3, and is connected to the deep learning processing module 3. The intelligent sorting execution module 2 is used for data communication with the deep learning processing module 3, and is connected to the mobile carrier module 5. The deep learning processing module 3 is used for data communication with the multimodal vision module 1, and also for data communication between the intelligent sorting execution module 2 and the mobile carrier module 5. The rapid calibration module 4 is used for data communication between the deep learning processing module 3 and the intelligent sorting execution module 2. The communication interface of the mobile carrier module 5 is used for communication with the intelligent sorting execution module 5. The picking execution module 2 communicates with the deep learning processing module 3. Specifically, the multimodal vision module 1 uses a binocular stereo camera and a ToF depth sensor to simultaneously acquire image and depth data within the target area. The image data acquired from different perspectives by the binocular stereo camera is fused with the depth data acquired by the ToF depth sensor. A 3D point cloud model is constructed to reconstruct and represent the spatial position and shape information of the target building material. During the reconstruction process, calibration data generated by the nine-point nonlinear calibration unit 41 is used to perform coordinate correction and error correction on the 3D point cloud data, thereby generating building material positioning data. The generated building material positioning data... The data is input to the deep learning processing module 3 via a data interface. The deep learning processing module 3 includes a YOLO algorithm processing unit, a CBAM module, and a data augmentation unit. The YOLO algorithm processing unit performs target detection and classification on the input building material location data, extracting the category, location, and size information of the building materials through a recognition algorithm, and labeling the recognition results and location information. The preliminary recognition results are optimized by the CBAM module. The CBAM module uses channel attention and spatial attention mechanisms to perform weighted calculations on the input features, enhancing the extraction and filtering of key features, and outputting the optimized recognition results. The data augmentation unit uses Mo... The SAIC data augmentation method expands and diversifies the training of building material positioning data, thereby improving the generalization ability of the deep learning model. The optimized recognition results are input to the intelligent sorting execution module 2 through the data interface. The intelligent sorting execution module 2 includes a six-axis collaborative robotic arm, a vacuum suction cup, and an adaptive gripper. The six-axis collaborative robotic arm performs path planning and motion control based on the recognition results and positioning information output by the deep learning processing module 3. The vacuum suction cup adapts to the surface characteristics of different types of building materials by adjusting the suction force. The adaptive gripper detects the gripping force and provides flexible protection through the built-in pressure sensor and silicone damping layer to avoid damage to the surface of the building materials.The intelligent sorting execution module 2 and the mobile carrier module 5 establish a data transmission channel through a communication interface. The mobile carrier module 5 includes a Mecanum wheel module and a laser SLAM navigation module. The Mecanum wheel module provides multi-directional movement capability, while the laser SLAM navigation module constructs an environmental map and performs path planning and positioning. The mobile carrier module 5 adjusts its own movement trajectory and positioning accuracy based on real-time acquired path control information to ensure that building materials accurately reach the target location during transportation. Through the collaborative work of the deep learning processing module 3 and the intelligent sorting execution module 2, the entire process control from building material identification, classification, grasping to transportation is achieved.

[0065] Furthermore, the multimodal vision module 1 includes a binocular stereo camera 11, a ToF depth sensor 12, a ring light 13, and an anti-glare curtain assembly 14. The binocular stereo camera 11 and the ToF depth sensor 12 work together, and the multimodal vision module 1 is connected to the deep learning processing module 3.

[0066] Specifically, the multimodal vision module 1 includes a binocular stereo camera 11, a ToF depth sensor 12, a ring light 13, and an anti-glare curtain assembly 14. The binocular stereo camera 11 is used to acquire image data of the target area from different perspectives, and the ToF depth sensor 12 is used to acquire depth data of the target area. By synchronously acquiring and fusing the image data and depth data, the generation of three-dimensional material distribution data of building materials is realized. The binocular stereo camera 11 and the ToF depth sensor 12 acquire data through a synchronous triggering mechanism. After the binocular stereo camera 11 acquires image data from the left and right perspectives, it transmits the image data to the deep learning processing module 3 through a serial port or parallel interface. When acquiring depth data, the ToF depth sensor 12 emits a modulated light signal to the surface of the target object and measures the reflected light signal. Phase difference or time delay is used to calculate the depth information of the object surface. The generated depth data is fused with the image data of the binocular stereo camera 11. A 3D point cloud model is constructed through a 3D reconstruction algorithm. The generated 3D material distribution data includes information such as the spatial location, size, shape, and surface texture of the building materials. To ensure the stability and accuracy of data acquisition, a ring light 13 is set in the multimodal vision module 1 to provide a uniform lighting environment. The light intensity and color temperature are automatically adjusted to adapt to the lighting conditions in different scenarios. The anti-glare curtain component 14 is used to eliminate glare interference caused by strong light sources or reflected light in the environment. By blocking or absorbing excess light, the clarity and contrast of the image data are improved. The multimodal vision module 1 transmits the generated 3D material distribution data to the deep learning processing module 3 through a data interface.

[0067] Furthermore, the intelligent sorting execution module 2 includes a six-axis collaborative robotic arm 21, a vacuum suction cup 22, and an adaptive gripper 23. The adaptive gripper 23 includes a pressure sensor 231 and a silicone damping layer 232. The intelligent sorting execution module 2 is connected to the deep learning processing module 3 and the mobile carrier module 5.

[0068] Specifically, the intelligent sorting execution module 2 includes a six-axis collaborative robotic arm 21, a vacuum suction cup 22, and an adaptive gripper 23. The six-axis collaborative robotic arm 21 is used to achieve multi-degree-of-freedom operation and precise control of the target building material. It provides high flexibility in motion by configuring six rotary joints and multiple servo motors. The end of the six-axis collaborative robotic arm 21 is equipped with a vacuum suction cup 22 and an adaptive gripper 23. The vacuum suction cup 22 is used for non-contact gripping when the surface of the target building material is relatively smooth or has good adsorption conditions. The magnitude of the adsorption force is adjusted by a built-in solenoid valve and a negative pressure generator to adapt to building materials of different weights and surface characteristics. The adaptive gripper 23 is used to clamp building materials with irregular shapes or uneven surfaces. A flexible silicone damping layer 232 is set in the clamping structure to provide cushioning and shock absorption functions. The silicone damping layer 232 can apply gentle pressure to the surface of the building material during the clamping process, thereby avoiding damage or deformation of the building material surface due to excessive clamping force. The adaptive gripper 23 has a built-in pressure sensor. Device 231 is used to detect pressure changes in real time during the clamping process. By transmitting the detected pressure data to the deep learning processing module 3, the clamping force is adaptively adjusted according to the pressure feedback information, thereby ensuring the stability and safety of clamping. The intelligent sorting execution module 2 is connected to the deep learning processing module 3 and exchanges data transmission and control commands through a communication protocol. When the deep learning processing module 3 outputs the optimized material identification results and target position coordinates, the intelligent sorting execution module 2 performs path planning and action execution based on the received data. It calculates the motion trajectory of the robotic arm and the gripping position of the end tool through a motion control algorithm, and adjusts the clamping force and suction force in real time to adapt to different types and sizes of building materials. The intelligent sorting execution module 2 is connected to the mobile carrier module 5 and communicates with the mobile carrier module 5 in real time through a data interface. When the target building material is successfully grasped, the mobile carrier module 5 moves and navigates autonomously according to the received path planning information, thereby transporting the target building material to the designated location.

[0069] Furthermore, the deep learning processing module 3 includes a YOLO algorithm processing unit 31, a CBAM module 32, and a data augmentation unit 33. The deep learning processing module 3 is connected to the multimodal vision module 1, the intelligent sorting execution module 2, and the mobile carrier module 5.

[0070] Specifically, the deep learning processing module 3 includes a YOLO algorithm processing unit 31, a CBAM module 32, and a data augmentation unit 33. The YOLO algorithm processing unit 31 is used to identify and classify the 3D material distribution data from the multimodal vision module 1. It performs feature extraction and target detection on the input 3D material distribution data by configuring a convolutional neural network. First, it segments and preprocesses the input 3D point cloud data, converting the data format to a standard format suitable for neural network input. Then, it performs layer-by-layer feature extraction and downsampling operations on the input data through a combination of convolutional layers, pooling layers, and activation functions, thereby obtaining the target building materials at different scales and... In the YOLO algorithm, hierarchical feature representation is used to train the model for object detection and classification by defining anchor boxes and loss functions. The recognition results are output in the form of class labels and target location coordinates. The CBAM module 32 optimizes and improves the accuracy of the initial recognition results output by the YOLO algorithm processing unit 31. The CBAM module 32 enhances the selectivity and sensitivity to target features by introducing channel attention and spatial attention mechanisms. First, global average pooling and global max pooling operations are performed on the input feature map to extract global information and maximum response values ​​of different channels, respectively. Then, the channel features are amplified by fully connected layers and activation functions. The system performs weighted calculations, multiplying the calculated result element-wise with the original feature map to generate an optimized feature map. Then, it uses a spatial attention mechanism to perform convolution and pooling operations on the optimized feature map, generating an attention weight matrix for localization. This matrix is ​​then multiplied with the feature map to enhance the focusing ability on the target region. The optimized recognition result is transmitted to the data augmentation unit 33 via a data interface for dataset augmentation training. The data augmentation unit 33 uses the Mosaic data augmentation method to randomly crop, rotate, scale, and mix the input data, thereby generating a diverse and robust training sample set. Augmentation training on different data samples further improves the recognition accuracy. To enhance the generalization ability and robustness of the model, the output of the data augmentation unit 33 is transmitted to the intelligent sorting execution module 2 and the mobile carrier module 5 through the data interface. The deep learning processing module 3 is connected to the multimodal vision module 1 to receive the three-dimensional material distribution data from the multimodal vision module 1. The deep learning processing module 3 is connected to the intelligent sorting execution module 2 to transmit the optimized recognition results and classification information to the intelligent sorting execution module 2 to guide the sorting and grasping operations of building materials. The deep learning processing module 3 is connected to the mobile carrier module 5 to transmit the path planning information and recognition results to the mobile carrier module 5 to guide the movement and positioning operations.

[0071] Furthermore, the rapid calibration module 4 includes a nine-point nonlinear calibration unit 41 and a calibration data output interface 42, which is connected to the deep learning processing module 3 and the intelligent sorting execution module 2, respectively.

[0072] Specifically, the rapid calibration module 4 includes a nine-point nonlinear calibration unit 41 and a calibration data output interface 42. The nine-point nonlinear calibration unit 41 is used to calibrate and correct the spatial coordinate system between the multimodal vision module 1, the deep learning processing module 3, and the intelligent sorting execution module 2. It sets nine calibration points with known positions and orientations within a specified calibration area, and uses the binocular stereo camera 11 and the ToF depth sensor 12 in the multimodal vision module 1 to image and detect the depth of the calibration points, generating 3D point cloud data and corresponding image data containing the calibration points. The collected calibration data is input to the nine-point nonlinear calibration unit 41. The nine-point nonlinear calibration unit 41 uses a nonlinear optimization algorithm and the least squares method to fit and solve the calibration point data, calculating the spatial mapping relationship and error parameters between the vision system and the execution module, including intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients. By comparing and iteratively optimizing the actual measured values ​​with the theoretical calibration values, calibration data is generated. According to the calibration data, the coordinate correction and error correction are performed on the three-dimensional material distribution data to ensure that the building material positioning data obtained from the vision system can be accurately mapped into the operating space of the intelligent sorting execution module 2 and the mobile carrying module 5. The calibration data output interface 42 is used to transmit the generated calibration data to the deep learning processing module 3 and the intelligent sorting execution module 2 through the data interface. The deep learning processing module 3 is used to receive the calibration data and correct and compensate for the error of the three-dimensional material distribution data according to the received data. By calculating, the calibration parameters and point cloud data are fused and mapped to ensure that the generated building material positioning data has high accuracy and consistency. The intelligent sorting execution module 2 is used to receive the calibration data and adjust and optimize the motion path and operation accuracy of the robotic arm based on the received calibration information. By comparing and calculating the calibration information with the actual operation data, it ensures that the end tool of the robotic arm can accurately reach the target position and perform precise grasping and handling operations on the target building material.

[0073] Furthermore, the mobile carrier module 5 includes a Mecanum wheel module 51, a laser SLAM navigation module 52, and a communication interface 501. The Mecanum wheel module 51 and the laser SLAM navigation module 52 work together to control the movement and positioning of the mobile carrier module 5. The communication interface 501 is connected to the deep learning processing module 3 and the intelligent sorting execution module 2, respectively.

[0074] Specifically, the mobile carrier module 5 includes a Mecanum wheel module 51, a laser SLAM navigation module 52, and a communication interface 501. The Mecanum wheel module 51 and the laser SLAM navigation module 52 work together to achieve omnidirectional movement and precise positioning of the mobile carrier module 5. The Mecanum wheel module 51 is formed by configuring multiple Mecanum wheels at the bottom of the mobile platform. Each Mecanum wheel consists of multiple tilted rollers. The rotation direction and speed of each Mecanum wheel are driven by an independent motor. The forward, backward, left and right translation, and rotational movements of the mobile platform are achieved by utilizing the frictional difference between the rollers and the ground. During the movement, the rotation speed and direction of each Mecanum wheel are adjusted in real time by a motion controller, thereby achieving high-precision omnidirectional movement and position control. The laser SLAM navigation module 52 is configured with a laser radar and an inertial measurement unit (IMU) for real-time scanning and map building of the surrounding environment. The laser radar emits a laser beam in a high-speed rotating manner and receives reflected light signals. The distance to surrounding objects is calculated by measuring the flight time or phase difference of the laser pulse. The distance between the objects is used to generate two-dimensional or three-dimensional point cloud data. The inertial measurement unit is used to detect the acceleration and angular velocity information of the mobile carrier module 5. The positioning accuracy and robustness are improved by fusing the data with the lidar data. The lidar SLAM navigation module 52 compares and matches the constructed environment map with the current position information to generate path planning and navigation instructions. The navigation information is transmitted to the deep learning processing module 3 and the intelligent sorting execution module 2 through the communication interface 501. The deep learning processing module 3 generates optimized path planning data based on the received navigation information and transmits the optimization results to the mobile carrier module 5 through the data interface. The intelligent sorting execution module 2 receives the path planning data and position information through the execution control terminal 201 to control the movement of the robotic arm and the grasping and handling of the target building materials. When the mobile carrier module 5 performs autonomous navigation and movement, the lidar SLAM navigation module 52 updates the environment map and path planning information in real time to ensure accurate obstacle avoidance and precise positioning in complex or dynamic environments, thereby achieving efficient building material handling and path optimization.

[0075] like Figure 2 As shown, a vision-based intelligent sorting method for building materials is applied to a vision-based intelligent sorting system for building materials. The vision-based intelligent sorting method for building materials includes:

[0076] S10: Obtain the first image data output by the binocular stereo camera 11 and the second depth data output by the ToF depth sensor 12, input the first image data and the second depth data into the YOLO algorithm processing unit (31) for preliminary identification and positioning, and obtain three-dimensional material distribution data.

[0077] Specifically, the system acquires first image data output by the binocular stereo camera 11 and second depth data output by the ToF depth sensor 12. By configuring the binocular stereo camera 11 and the ToF depth sensor 12, image and depth data within the target area are simultaneously acquired. The binocular stereo camera 11 consists of two cameras installed at different locations with a certain baseline distance, forming left and right perspectives. By capturing two images of the same scene, the spatial depth information of the target building material in the scene is calculated using the parallax principle. Feature point matching and stereo correction are performed on the left and right views to generate matching pixel pairs and calculate their parallax values. The parallax information is input into a 3D reconstruction algorithm for depth estimation and point cloud generation. The ToF depth sensor 12 emits modulated light signals to the target area and receives reflected light signals. The depth information of the object surface is calculated by measuring the phase difference or time delay of the light signals during propagation. The acquired depth data is aligned and fused with the point cloud data generated by the binocular stereo camera 11. The system integrates depth information and image data using a coordinate registration algorithm to generate three-dimensional material distribution data containing the location, shape, size, and surface features of building materials. To ensure the accuracy and completeness of data acquisition, a ring-shaped supplementary light 13 configured in the multimodal vision module 1 provides uniform illumination conditions, automatically adjusting the light intensity and color temperature to adapt to different lighting environments. Simultaneously, an anti-glare curtain assembly 14 shields and absorbs glare generated in the environment to reduce the impact of light reflection or strong light interference on image quality. The acquired first image data and second depth data are input to the YOLO algorithm processing unit 31. A convolutional neural network is configured to extract, detect, and classify the input data. The constructed neural network model performs convolution and pooling operations on the input data to extract and compress the high-dimensional features of the original data. Classification and target localization are performed through fully connected layers and activation functions to generate three-dimensional material distribution data containing the category information and spatial location of the target building materials.

[0078] S20: Obtain the calibration data output by the nine-point nonlinear calibration unit 41, and perform coordinate correction and error correction on the three-dimensional material distribution data based on the calibration data to obtain building material positioning data.

[0079] Specifically, nine calibration points with known coordinates are arranged within a predetermined calibration area. The binocular stereo camera 11 and the ToF depth sensor 12 in the multimodal vision module 1 are used to image and detect depth in the calibration area, generating 3D point cloud data and corresponding image data containing the calibration points. The collected calibration point data is input into the nine-point nonlinear calibration unit 41. The nine-point nonlinear calibration unit 41 processes and fits the calibration point data using a nonlinear optimization algorithm. Specifically, the calibration point data is first denoised and filtered to eliminate the influence of environmental noise and sensor error. An objective function is constructed using the least squares method, and the difference between the measured values ​​and theoretical values ​​of the calibration points is calculated and error analysis is performed. The error parameters are adjusted and compensated using an iterative optimization algorithm, thereby generating calibration data. The calibration data includes an intrinsic parameter matrix, an extrinsic parameter matrix, and an intrinsic parameter matrix. The calibration data includes parameters such as the intrinsic parameter matrix, principal point coordinates, and pixel scaling factor. The extrinsic parameter matrix describes the spatial relationship and rotation matrix between the camera and the calibration plane. The distortion coefficient corrects radial and tangential distortion caused by the lens. The generated calibration data is input into the deep learning processing module 3 for coordinate correction and error correction. By registering and transforming the calibration data with the 3D material distribution data, the 3D point cloud data is aligned with the world coordinate system, thereby eliminating coordinate deviations caused by sensor errors or system configuration. The error distribution of the calibration points is modeled and fitted using an error correction algorithm. By correcting the coordinate values ​​of each point cloud data point by point, the influence of systematic and random errors is eliminated, resulting in accurate building material positioning data. The building material positioning data includes the spatial location, size, shape, and orientation information of the building materials.

[0080] S30: Use data augmentation unit 33 to perform dataset augmentation training on building material positioning data to obtain an augmented material distribution dataset.

[0081] Specifically, the input building material positioning data is first preprocessed and formatted, including standardizing and normalizing the point cloud data and position coordinates in the 3D material distribution data to facilitate input and training of the deep learning network. Through normalization of the input data, building material data of different scales are uniformly scaled, and the input data is segmented and resampled to improve data uniformity and computational efficiency. The data augmentation unit 33 uses the configured Mosaic data augmentation method to diversify and expand the training of the building material positioning data. By randomly cropping, splicing, scaling, rotating, and mixing multiple sets of different building material positioning data, a new training sample set is generated. During the Mosaic data augmentation process, building material positioning data from four different scenarios are randomly selected and merged into a new scenario. First, the data from each scenario is processed... The data is randomly cropped and scaled, then transformed using coordinate translation and rotation matrices. The transformed data is then spliced ​​and merged to generate training samples with multiple building material objects and different backgrounds. By mixing and recombining data from different sources, the model's adaptability and generalization ability to complex scenes are improved. Random noise and brightness adjustment are introduced during data augmentation to increase the diversity and robustness of the dataset. The generated augmented sample set is input to the deep learning processing module 3 through the data interface for training and optimization. During training, the constructed deep neural network extracts and classifies the input data. Convolutional and pooling layers process and downsample the input data layer by layer to extract high-level features with discriminative power. Classification and prediction are performed through fully connected layers and activation functions to generate the augmented material distribution dataset.

[0082] S40: The enhanced material distribution dataset is identified and classified by the YOLO algorithm processing unit 31 to obtain the material identification result.

[0083] Specifically, the enhanced material distribution dataset is first input into the YOLO algorithm processing unit 31 for feature extraction and target detection. A configured convolutional neural network performs multi-level feature extraction and classification on the input 3D point cloud data and building material positioning data. In the input layer, the enhanced dataset undergoes formatting and standardization, including data normalization, coordinate alignment, and batch processing. Data samples from different sources are standardized to facilitate neural network input and computation. The YOLO algorithm processing unit 31 performs layer-by-layer feature extraction and dimensionality reduction on the input data through constructed convolutional layers, pooling layers, and activation functions, thereby generating a feature map with high-dimensional representation. During feature extraction, a sliding window operation is performed on the input data using convolutional kernels to extract feature information at different scales and resolutions. Pooling operations are used to downsample and compress the feature map to reduce computational complexity and the number of parameters. For target detection in the YOLO algorithm... In this process, anchor boxes and loss functions are defined to predict and classify the category, location, and size of building materials. Multiple anchor boxes are generated on the feature map of the input data, and each anchor box is classified and regressed. The prediction results include category confidence, bounding box coordinates, and relative scale information. The prediction results are compared with the true labeled values, and the error value is calculated using the defined loss function, which includes classification error, localization error, and confidence error. The backpropagation algorithm is used to update and optimize the network parameters to reduce the gap between the prediction results and the true values. During training, a multi-scale detection mechanism and feature fusion strategy are introduced to improve the model's ability to recognize building materials of different scales and shapes. The trained model is applied to the enhanced material distribution dataset for recognition and classification. The constructed neural network performs forward propagation and inference calculations on the input data to generate material recognition results containing the category information, location coordinates, and size information of the target building materials.

[0084] S50: Input the material identification result into the CBAM module 32 for optimization processing, and output the optimized material identification result.

[0085] Specifically, the CBAM module 32, composed of a channel attention mechanism and a spatial attention mechanism, improves the accuracy and optimizes the features of the material recognition results generated by the YOLO algorithm processing unit 31. First, it decomposes and normalizes the feature map of the input material recognition results. By separating and normalizing each channel of the input feature map, multiple independent channel feature matrices are generated. During the channel attention mechanism processing, global information is extracted from the input feature map through global average pooling and global max pooling operations. The feature information of each channel is compressed and aggregated, and the channel features are weighted using a fully connected layer and activation function to generate a channel weight matrix for optimization. The channel weight matrix is ​​then multiplied element-wise with the original feature map, thereby enhancing important features and removing redundant or insignificant features. Key features are suppressed. In the spatial attention mechanism, convolution and pooling operations are performed on the input feature map to compress and reconstruct its spatial dimensions. The convolution kernel is used to weight features at different locations to generate a spatial weight matrix for optimization. The spatial weight matrix is ​​multiplied by the channel-optimized feature map to highlight and optimize key areas. In the optimization process of CBAM module 32, the useful and useless information in the original feature map is effectively distinguished and extracted through multiple iterations and optimization calculations of the material recognition results. By weighting and fusing different dimensions of the feature map, the feature discrimination ability and model accuracy are improved. The generated optimized material recognition results include the target building material's category information, location coordinates, size information, and confidence value.

[0086] S60: The optimized material identification results are input into the intelligent sorting execution module 2. The intelligent sorting execution module 2 uses a six-axis collaborative robotic arm 21, a vacuum suction cup 22, and an adaptive clamp 23 to grasp and transport the target building materials.

[0087] Specifically, after receiving the optimized material identification results, the sorting and gripping operations are controlled and executed using the target building material's category information, location coordinates, size information, and confidence value contained in the identification results. The six-axis collaborative robotic arm 21 uses its built-in motion control system and path planning algorithm to calculate and plan the spatial position and gripping path of the target building material in real time. Through the cooperation of multiple servo motors and rotary joints, it achieves high-degree-of-freedom motion and precise positioning. Specifically, the six-axis collaborative robotic arm 21 calculates the target position and gripping posture of the end effector based on the received optimized material identification results, and solves and optimizes the angles and motion trajectories of each joint of the robotic arm through inverse kinematics algorithms and interpolation calculations. The calculation results are converted into drive signals for each joint motor, so that the end effector of the robotic arm accurately reaches the predetermined position of the target building material. The vacuum suction cup 22 uses the working state of the negative pressure generator and solenoid valve to adsorb and grip the surface of the target building material. When the surface of the target building material is relatively smooth or has good adsorption conditions, the vacuum suction cup 22 can adsorb and grip the surface of the target building material. During operation, the vacuum suction cup 22 automatically adjusts the suction force and contact area to adapt to building materials with different materials and surface characteristics. A built-in pressure sensor monitors and provides real-time feedback on the adsorption state to ensure stable and safe gripping. The adaptive clamp 23 is used to grip and transport building materials with irregular shapes or uneven surfaces. A flexible silicone damping layer 232 and a pressure sensor 231 are configured in the clamping structure. The flexible silicone damping layer 232 provides buffering and shock absorption, adjusting the clamping force and angle based on the sensed pressure changes to avoid damage or slippage to the building material surface. The adaptive clamp 23 monitors the pressure data in real-time during the gripping process through a closed-loop control system and adaptively adjusts the gripping state. Feedback control and iterative optimization ensure precise gripping and transport of the target building material. After completing the gripping and clamping of the target building material, the intelligent sorting execution module 2 transmits the gripping status and position information to the mobile carrier module 5 via a communication interface for subsequent transport and path planning operations.

[0088] S70: Acquire motion data output by the Mecanum wheel module 51 and global positioning data output by the laser SLAM navigation module 52.

[0089] Specifically, the configured Mecanum wheel module 51 works in conjunction with the laser SLAM navigation module 52 to provide precise motion control and global positioning capabilities. The Mecanum wheel module 51 consists of multiple Mecanum wheels, each driven by an independent motor to provide movement capabilities in different directions. Specifically, several inclined rollers are evenly distributed on the outer ring of the Mecanum wheel. By changing the speed and direction of each motor, the mobile carrier module 5 can move in all directions, including forward, backward, lateral, diagonal, and stationary rotation. The motion controller calculates and controls the speed and relative direction of each motor to achieve the desired motion path and attitude changes. During the movement, the Mecanum wheel module 51 uses configured speed and angle sensors to detect the rotational speed, rotation direction, and tilt angle of each Mecanum wheel in real time, and formats and encodes the detected motion data, transmitting it to the deep learning processing module through a data interface. Block 3 performs processing and analysis; simultaneously, the laser SLAM navigation module 52 provides global positioning data and environmental perception information through the configured laser radar and inertial measurement unit (IMU). The laser radar scans and measures the surrounding environment through a high-speed rotating laser emitter, and calculates the distance between itself and surrounding objects by measuring the flight time or phase difference of the laser pulse, thereby generating two-dimensional or three-dimensional point cloud data. The inertial measurement unit provides attitude and displacement information of the mobile carrier module 5 by detecting acceleration and angular velocity information. The data from the laser radar and the inertial measurement unit are fused and matched, and the current position information is compared and updated with the environmental map through the laser SLAM algorithm, thereby generating accurate global positioning data, including the current position coordinates, direction of motion and attitude angle. The generated global positioning data is transmitted to the deep learning processing module 3 and the intelligent sorting execution module 2 through the data interface for further optimization and adjustment of path planning and motion control.

[0090] S80: Input motion data and global positioning data into deep learning processing module 3 to obtain path planning and positioning control information.

[0091] Specifically, motion data and global positioning data are input into the deep learning processing module 3. Motion data output from the Mecanum wheel module 51 and global positioning data output from the laser SLAM navigation module 52 are synchronously transmitted to the deep learning processing module 3 via a data interface. The deep learning processing module 3 includes a configured path planning unit and a positioning control unit, used for fusion calculation and real-time optimization of path planning on the input data. In the preprocessing stage of the input data, the motion data and global positioning data are formatted and normalized to align and register data from different sources. The motion data from the Mecanum wheel module 51 is decomposed and parsed to extract the current velocity, acceleration, turning angle, and attitude information. A feature extraction algorithm is used to convert the raw data into a standardized motion state matrix. Simultaneously, the global positioning data output from the laser SLAM navigation module 52 is parsed and reconstructed to match and compare the current position, target position, and environmental map, generating a complete environmental model and global path map. In the path planning unit, a path planning unit is constructed... The deep neural network and reinforcement learning algorithms process and optimize the input data. First, feature extraction and convolution operations are performed on the input motion state matrix and global path map. Multi-scale feature extraction and dimensionality reduction operations are performed on the input data through configured convolutional and pooling layers to generate a high-dimensional feature map for path planning. Based on the feature map, the target point and path node of path planning are progressively inferred and predicted through the constructed fully connected layer and activation function. The network parameters are adjusted and optimized through backpropagation algorithm and reinforcement learning strategy to improve the accuracy and efficiency of path planning. In the positioning control unit, the path planning result is compared and corrected with the input data. The motion trajectory is corrected and adjusted in real time through the configured control algorithm and optimization module to ensure that the mobile carrier module 5 can accurately follow the predetermined trajectory during the path planning process. Real-time obstacle avoidance and path replanning are performed when encountering dynamic obstacles or environmental changes. The deep learning processing module 3 continuously optimizes and iteratively calculates the input data to generate path planning information and positioning control information.

[0092] S90: Input the path planning and positioning control information into the intelligent sorting execution module 2 to control the movement and positioning of the mobile carrier module 5.

[0093] Specifically, the path planning and positioning control information generated by the deep learning processing module 3 is transmitted to the intelligent sorting execution module 2 via a data interface. The intelligent sorting execution module 2 precisely controls the motion state and position of the mobile carrier module 5 based on the received path planning and positioning control information. By analyzing the target position coordinates, motion trajectory, and attitude angle in the path planning information, the intelligent sorting execution module 2 first analyzes and decomposes the path planning information, dividing the overall path into several continuous motion nodes. It then extracts and calculates the motion parameters of each node, including moving speed, acceleration, turning angle, and motion direction. During motion control, the constructed control algorithm is used to adjust and control the drive motor of the Mecanum wheel module 51 in real time. By controlling the rotational speed and direction of each Mecanum wheel, the omnidirectional movement and precise positioning of the mobile carrier module 5 are achieved. By calculating the error value between the current motion state and the target position, and using a PID control algorithm to dynamically adjust and compensate for the error, the mobile carrier module 5 is ensured to accurately reach the predetermined position. Simultaneously, through... The system compares and matches the global positioning data provided by the laser SLAM navigation module 52, detects the deviation between the current moving position and the target position in the global map, and updates and optimizes the path planning information in real time through the constructed adaptive control algorithm. When encountering dynamic obstacles or environmental changes, the system adjusts and corrects the current trajectory by introducing obstacle avoidance strategies and path replanning mechanisms. By recalculating the priority of the target position and path nodes, the system ensures that the mobile carrier module 5 can move and position according to the optimal path. During the execution of positioning control, the system collects and feeds back the real-time status information of the mobile carrier module 5, including parameters such as position coordinates, speed, acceleration, and attitude angle. By comparing and analyzing the data with the predetermined path planning information, the system detects and adjusts the error of each movement node. Through iterative optimization and feedback control, the system ensures the motion accuracy and positioning accuracy of the mobile carrier module 5. Finally, the optimized position information and path planning results are transmitted to the intelligent sorting execution module 2 through the communication interface to achieve precise grasping and handling of the target building materials.

[0094] In one embodiment, such as Figure 3 As shown, in step S30, the data augmentation unit 33 is used to perform dataset augmentation training on the building material positioning data to obtain an augmented material distribution dataset, including:

[0095] S301: Data augmentation unit 33 performs dataset augmentation training on building material positioning data based on the Mosaic data augmentation method to obtain an augmented material distribution dataset.

[0096] In this embodiment, the Mosaic data augmentation method refers to generating a training sample set with diversity and complexity by randomly cropping, scaling, rotating, splicing and merging building material positioning data from multiple different scenarios.

[0097] Specifically, four different sets of building material location data samples were randomly selected from the original building material location dataset. Each set of data samples included building material category information, location coordinates, size information, and depth data. To ensure the diversity and uniformity of the data samples, the category and size were randomized during the sample selection process. The four sets of data samples were then standardized and preprocessed, including normalization, coordinate alignment, and data format conversion for each set of data to facilitate subsequent splicing and fusion operations. Subsequently, each data sample was randomly cropped, scaled, and rotated. By transforming the input data at different scales and angles, the diversity and robustness of the data samples were enhanced. In the cropping operation, a randomly generated cropping window was used to truncate the original data, and the cropped data was scaled and rotated. An affine transformation matrix was then used to perform a spatial transformation on the data. The data processing involves several steps: First, the data is transformed and its position adjusted. Second, during the data stitching stage, four preprocessed data samples are stitched and mixed, mapping each sample to a quarter-area. These four areas are then combined to form a complete enhanced sample image. By stitching and recombining data from different sources, training samples with multiple building material objects and complex backgrounds are generated. Subsequently, brightness adjustment, noise introduction, and color transformation are applied to the stitched samples to further increase the diversity and adaptability of the data samples. Third, during the dataset enhancement training process, the generated enhanced samples are input into the deep learning processing module 3 for training and optimization. The constructed deep neural network extracts, classifies, and identifies features from the input data. Backpropagation algorithms and optimization strategies are used to adjust and update the network parameters, thereby generating an enhanced material distribution dataset.

[0098] In one embodiment, such as Figure 4 As shown, in step S50, the material identification result is input to the CBAM module 32 for optimization processing, and the optimized material identification result is output, including:

[0099] S501: The CBAM module 32 optimizes the material recognition results based on the channel attention mechanism and the spatial attention mechanism, and outputs the optimized material recognition results.

[0100] In this embodiment, the channel attention mechanism refers to improving the extraction and representation of key features by calculating and optimizing weights on the input feature map along the channel dimension. The spatial attention mechanism refers to improving the focusing ability and accuracy of the target region by calculating and optimizing weights on the input feature map along the spatial dimension.

[0101] Specifically, the CBAM module 32 optimizes the material recognition results based on channel attention and spatial attention mechanisms. It improves the model's recognition accuracy and feature extraction capabilities by extracting features and weighting the input material recognition results. In the implementation process, the input material recognition results are first preprocessed and standardized using feature maps. This involves decomposing and formatting the multi-dimensional feature maps, including normalization, channel separation, and dimensional adjustment, to accommodate subsequent channel attention and spatial attention calculations. In the channel attention mechanism, global information extraction and weighted calculation are performed on each channel of the input feature map. First, global average pooling and global max pooling operations are performed on the feature map to extract the global feature information and maximum response information of each channel. Then, the data for each channel is compressed and aggregated to generate channel feature vectors for optimization. These generated channel feature vectors are input to the fully connected layer, and nonlinear transformations and normalization are performed on the feature vectors using activation functions and regularization operations to generate channel feature vectors for weighted optimization. The channel weight matrix is ​​multiplied element-wise with the original feature map to enhance the response to important features and suppress irrelevant or redundant features. In the spatial attention mechanism, spatial features are extracted and optimized from the optimized channel feature map. First, convolution and pooling operations are performed on the optimized channel feature map to extract spatial feature information at different locations. The constructed convolution kernel performs weighted calculations and pixel-wise optimization on each location of the input feature map to generate a weight matrix for spatial optimization. The generated spatial weight matrix is ​​then multiplied with the optimized channel feature map to highlight and optimize important regions and suppress and filter irrelevant regions. During the optimization process, the input material recognition results are iterated and optimized multiple times to effectively distinguish and extract useful and useless information from the original feature map. Weighting and fusing different dimensions of the feature map improves feature discrimination ability and model accuracy. The optimized material recognition results include building material category information, location coordinates, size information, and confidence values.

[0102] In one embodiment, such as Figure 5 As shown, after step S90, i.e., a vision-based intelligent sorting method for building materials, it also includes;

[0103] S901: The remote monitoring system evaluates and provides feedback on the optimized material identification results, path planning, and positioning control information, and obtains the evaluation feedback results.

[0104] In this embodiment, the remote monitoring system refers to a computer system or cloud platform used to monitor, evaluate and optimize the material identification and path planning process in real time.

[0105] Specifically, the optimized material identification results and path planning and positioning control information are first uploaded to the remote monitoring system via a communication interface. The remote monitoring system, through its configured network communication module and data processing unit, parses and stores the received data. It categorizes and records the category information, location coordinates, size information, and confidence values ​​in the material identification results. Simultaneously, it formats and standardizes the target position, trajectory, speed, acceleration, and attitude information in the path planning and positioning control information for subsequent evaluation and analysis. During the evaluation process, the remote monitoring system uses a constructed data analysis algorithm to perform multi-dimensional comparisons and calculations on the input data. It evaluates the accuracy, stability, and consistency of the material identification results and path planning information, comparing the material identification results with pre-set standard or target values ​​and calculating the identification accuracy and error rate. Simultaneously, it evaluates the path planning and positioning control information. The deviation between the motion trajectory and the target position in the position control information is calculated and analyzed. The rationality and accuracy of the path planning are judged by calculating the error value of each node and the cumulative error value. In the feedback process, the remote monitoring system organizes and outputs the evaluation results through the constructed feedback control mechanism. By quantifying and modeling the evaluation results of the identification results and path planning information, the system generates evaluation feedback results containing error distribution, accuracy statistics and optimization suggestions. The generated evaluation feedback results are transmitted to the deep learning processing module 3 and the intelligent sorting execution module 2 through the data interface for optimization and adjustment of model parameters and control strategies. In this process, the remote monitoring system uses the constructed adaptive feedback mechanism to monitor and iteratively optimize the identification and control process in real time, thereby improving the stability and accuracy of the system and ensuring that the identification and handling of target building materials can be completed efficiently in complex or dynamic environments.

[0106] S902: Based on the evaluation feedback results, generate optimization information, input the optimization information into the deep learning processing module 3 and the intelligent sorting execution module 2 for optimization training, and obtain the optimized training model and execution control strategy.

[0107] Specifically, the evaluation feedback results output by the remote monitoring system are analyzed and extracted, including information such as recognition accuracy, path planning accuracy, positioning control accuracy, and error distribution. A data processing unit decomposes and formats the evaluation feedback results, extracting relevant data on recognition accuracy and path planning accuracy, and performing normalization and standardization to facilitate subsequent optimization training and parameter adjustment. During the generation of optimization information, error data in the evaluation feedback results is statistically analyzed and modeled to construct an error matrix containing recognition error and path planning error. The error matrix is ​​then analyzed and optimized using constructed optimization algorithms, including least squares optimization, gradient descent optimization, and adaptive learning algorithms. The error distribution is fitted and solved to generate optimization information for training. This generated optimization information is input into the deep learning processing module 3 and the intelligent sorting execution module 2. The constructed deep learning model trains and optimizes the input optimization information. In this process, the parameters of the YOLO algorithm processing unit 31, CBAM module 32, and data augmentation unit 33 are adjusted and optimized. The network weights are updated and corrected through the backpropagation algorithm. The error in the optimization information is quantified and fed back through the constructed loss function, thereby gradually reducing the recognition error and positioning error. At the same time, in the intelligent sorting execution module 2, the control strategies of the six-axis collaborative robotic arm 21, vacuum suction cup 22, and adaptive gripper 23 are optimized and adjusted. By introducing PID control algorithm, adaptive control algorithm, and path replanning algorithm, the input optimization information is corrected and adjusted in real time, generating an optimized execution control strategy and path planning parameters. The model and control strategy are continuously improved and optimized through iterative training and feedback optimization, thereby generating an optimized training model and execution control strategy. The optimization results are transmitted to the execution control end of the deep learning processing module 3 and the intelligent sorting execution module 2 through the data interface for subsequent recognition, classification, and grasping operations.

[0108] S903: The optimized training model is input into the YOLO algorithm processing unit 31 and the CBAM module 32 to update the recognition algorithm and optimization strategy, and obtain the updated material recognition result. The updated material recognition result is used to improve the recognition accuracy and target positioning accuracy of the YOLO algorithm processing unit 31 and the CBAM module 32.

[0109] Specifically, the optimized training model is input into the YOLO algorithm processing unit 31 and the CBAM module 32. The optimized training model and updated network parameters are transmitted to the YOLO algorithm processing unit 31 and the CBAM module 32 in the deep learning processing module 3 via a data interface. This is used to update and improve the recognition algorithm and optimization strategy. In the specific implementation process, the optimized training model is formatted and standardized, including updating and replacing network weights, bias values, convolution kernel parameters, and activation function parameters. The optimized training model is compared and fused with the original model to ensure that the structural and logical integrity of the original model is maintained during the model update process. In the YOLO algorithm processing unit 31, the optimized training model is loaded into a deep neural network, and feature extraction, detection, and classification are performed on the input material distribution dataset. Multi-scale feature extraction and downsampling operations are performed on the input data through constructed convolutional and pooling layers. The classification results are calculated and output through fully connected layers and activation functions. The network weights are updated through backpropagation algorithm and optimization strategy. The YOLO algorithm processing unit 31 is adjusted to improve the recognition accuracy and classification accuracy of target building materials. In the CBAM module 32, the optimized training model is input into the channel attention mechanism and the spatial attention mechanism to perform multi-dimensional feature optimization and enhancement on the optimized material recognition results. In the channel attention mechanism, the input feature map is compressed and aggregated through global average pooling and global max pooling operations. The weight matrix generated by the weighted calculation and optimization of the feature vector through fully connected layers and activation functions is multiplied element-wise with the input feature map to enhance the response of important features and suppress irrelevant features. In the spatial attention mechanism, the spatial features of the input feature map are extracted and optimized through the constructed convolution kernel and pooling layer. The extracted spatial feature information is fused and weighted with the original features to generate the optimized material recognition results. During the optimization process, the input material recognition results are iterated and optimized multiple times to effectively distinguish and extract useful and useless information in the original feature map, thereby improving the model's recognition accuracy and target positioning accuracy of target building materials.

[0110] S904: The optimized execution control strategy is input into the intelligent sorting execution module 2 to update the execution control parameters of sorting and handling operations, and to obtain updated path control information. The updated path control information is used to optimize the motion path and gripping accuracy of the intelligent sorting execution module 2.

[0111] Specifically, the optimized execution control strategy is input into the intelligent sorting execution module 2. The optimized execution control strategy and updated control parameters are transmitted to the intelligent sorting execution module 2 via a data interface for real-time updating and optimization of the control logic and execution parameters of sorting and handling operations. In the specific implementation process, the optimized execution control strategy is formatted and standardized, including the decomposition and reorganization of control parameters, path planning information, and grasping strategies. The optimized execution control strategy is then integrated with the control algorithm of the intelligent sorting execution module 2 to ensure system stability and consistency during control strategy updates. In the sorting and handling operations, the execution control parameters of the six-axis collaborative robotic arm 21, vacuum suction cup 22, and adaptive gripper 23 in the intelligent sorting execution module 2 are optimized and adjusted. The motion path, speed, acceleration, and attitude angle of the robotic arm are replanned and adjusted to improve the grasping accuracy and handling efficiency of the target building materials. Specifically, the optimized execution control strategy is analyzed and decomposed using the constructed path planning algorithm and motion control algorithm, dividing the entire sorting and handling operation process into several... Each motion node and execution step involves progressively adjusting and optimizing the control parameters of each node, including the attitude control, path tracking, and clamping force adjustment of the end effector. During path planning, the optimized execution control strategy is fused with motion data and global positioning data from the Mecanum wheel module 51 and the laser SLAM navigation module 52. PID control and adaptive control algorithms are used to correct and adjust the path planning information in real time, generating optimized path control information. By progressively optimizing and iteratively updating the motion path and positioning accuracy, the intelligent sorting execution module 2 can accurately grasp and transport target building materials in complex or dynamic environments. During the updating of execution control parameters, feedback information and real-time monitoring data are compared and analyzed to model and optimize error data. Backpropagation algorithms and optimization strategies are used to iteratively train and adjust the execution control strategy. The generated updated path control information includes optimized motion path, grasping accuracy, and comprehensive information on transport efficiency, and is transmitted to the intelligent sorting execution module 2 via a data interface to guide subsequent sorting and transport operations.

[0112] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0113] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0114] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A vision-based intelligent building materials sorting system, characterized in that, The vision-based intelligent building materials sorting system includes a multimodal vision module (1), an intelligent sorting execution module (2), a deep learning processing module (3), a rapid calibration module (4), and a mobile carrier module (5), wherein: The multimodal vision module (1) is used to communicate with the deep learning processing module (3) via data, and the multimodal vision module (1) is connected to the deep learning processing module (3); The multimodal vision module (1) includes a binocular stereo camera (11), a ToF depth sensor (12), a ring light (13), and an anti-glare curtain assembly (14). The binocular stereo camera (11) works in conjunction with the ToF depth sensor (12), and the multimodal vision module (1) is connected to the deep learning processing module (3). The intelligent sorting execution module (2) is used to communicate with the deep learning processing module (3) and is connected to the mobile carrier module (5); The deep learning processing module (3) is used to communicate with the multimodal vision module (1) and to communicate with the intelligent sorting execution module (2) and the mobile carrier module (5). The rapid calibration module (4) is used to communicate with the deep learning processing module (3) and the intelligent sorting execution module (2) via data. The communication interface of the mobile carrier module (5) is used to communicate with the intelligent sorting execution module (2) and the deep learning processing module (3) for data communication; The deep learning processing module (3) includes a YOLO algorithm processing unit (31), a CBAM module (32), and a data augmentation unit (33); The mobile carrier module (5) includes a Mecanum wheel module (51), a laser SLAM navigation module (52), and a communication interface (501); The first image data output by the binocular stereo camera (11) and the second depth data output by the ToF depth sensor (12) are obtained. The first image data and the second depth data are input to the YOLO algorithm processing unit (31) for preliminary identification and positioning to obtain three-dimensional material distribution data. The data augmentation unit (33) is used to perform dataset augmentation training on the building material positioning data to obtain an augmented material distribution dataset; The enhanced material distribution dataset is identified and classified by the YOLO algorithm processing unit (31) to obtain the material identification result; The material identification result is input into the CBAM module (32) for optimization processing, and the optimized material identification result is output. Obtain the motion data output by the Mecanum wheel module (51) and the global positioning data output by the laser SLAM navigation module (52); The motion data and the global positioning data are input into the deep learning processing module (3) to obtain path planning and positioning control information; Also includes; The remote monitoring system evaluates and provides feedback on the optimized material identification results and the path planning and positioning control information to obtain evaluation feedback results. Specifically, the optimized material identification results and path planning and positioning control information are first uploaded to the remote monitoring system via a communication interface. The remote monitoring system, through its configured network communication module and data processing unit, parses and stores the received data. It classifies and records the category information, location coordinates, size information, and confidence values ​​in the material identification results. Simultaneously, it formats and standardizes the target position, trajectory, speed, acceleration, and attitude information in the path planning and positioning control information for subsequent evaluation and analysis. During the evaluation process, the remote monitoring system uses a constructed data analysis algorithm to perform multi-dimensional comparisons and calculations on the input data. It evaluates the accuracy, stability, and consistency of the material identification results and path planning information, compares the material identification results with pre-set standard or target values, and calculates the identification accuracy and error rate. Simultaneously, it evaluates the path planning and positioning control information. The deviation between the motion trajectory and the target position in the control information is calculated and analyzed. The rationality and accuracy of the path planning are judged by calculating the error value of each node and the cumulative error value. In the feedback process, the remote monitoring system organizes and outputs the evaluation results through the constructed feedback control mechanism. By quantifying and modeling the evaluation results of the identification results and the path planning information, the system generates evaluation feedback results containing error distribution, accuracy statistics and optimization suggestions. The generated evaluation feedback results are transmitted to the deep learning processing module (3) and the intelligent sorting execution module (2) through the data interface for optimizing and adjusting the model parameters and control strategies. In this process, the remote monitoring system monitors and iteratively optimizes the identification and control process in real time through the constructed adaptive feedback mechanism, thereby improving the stability and accuracy of the system and ensuring that the identification and handling of target building materials can be completed efficiently in complex or dynamic environments.

2. The vision-based intelligent building materials sorting system according to claim 1, characterized in that, The intelligent sorting execution module (2) includes a six-axis collaborative robotic arm (21), a vacuum suction cup (22), and an adaptive fixture (23). The adaptive fixture (23) includes a pressure sensor (231) and a silicone damping layer (232). The intelligent sorting execution module (2) is connected to the deep learning processing module (3) and the intelligent sorting execution module (2) is connected to the mobile carrier module (5).

3. The vision-based intelligent building materials sorting system according to claim 2, characterized in that, The deep learning processing module (3) includes a YOLO algorithm processing unit (31), a CBAM module (32), and a data augmentation unit (33). The deep learning processing module (3) is connected to the multimodal vision module (1), the deep learning processing module (3) is connected to the intelligent sorting execution module (2), and the deep learning processing module (3) is connected to the mobile carrier module (5).

4. The vision-based intelligent building materials sorting system according to claim 3, characterized in that, The rapid calibration module (4) includes a nine-point nonlinear calibration unit (41) and a calibration data output interface (42), which is connected to the deep learning processing module (3) and the intelligent sorting execution module (2) respectively.

5. A vision-based intelligent building materials sorting system according to claim 4, characterized in that, The mobile carrier module (5) includes a Mecanum wheel module (51), a laser SLAM navigation module (52), and a communication interface (501). The Mecanum wheel module (51) works in conjunction with the laser SLAM navigation module (52) to control the movement and positioning of the mobile carrier module (5). The communication interface (501) is connected to the deep learning processing module (3) and the intelligent sorting execution module (2) respectively.

6. A vision-based intelligent sorting method for building materials, applied to a vision-based intelligent sorting system for building materials as described in claim 5, characterized in that, The vision-based intelligent sorting method for building materials includes: The first image data output by the binocular stereo camera (11) and the second depth data output by the ToF depth sensor (12) are obtained. The first image data and the second depth data are input to the YOLO algorithm processing unit (31) for preliminary identification and positioning to obtain three-dimensional material distribution data. Obtain the calibration data output by the nine-point nonlinear calibration unit (41), and perform coordinate correction and error correction on the three-dimensional material distribution data based on the calibration data to obtain building material positioning data; The data augmentation unit (33) is used to perform dataset augmentation training on the building material positioning data to obtain an augmented material distribution dataset; The enhanced material distribution dataset is identified and classified by the YOLO algorithm processing unit (31) to obtain the material identification result; The material identification result is input into the CBAM module (32) for optimization processing, and the optimized material identification result is output. The optimized material identification results are input to the intelligent sorting execution module (2), which uses the six-axis collaborative robotic arm (21), the vacuum suction cup (22), and the adaptive fixture (23) to grasp and transport the target building materials. Obtain the motion data output by the Mecanum wheel module (51) and the global positioning data output by the laser SLAM navigation module (52); The motion data and the global positioning data are input into the deep learning processing module (3) to obtain path planning and positioning control information; The path planning and positioning control information is input into the intelligent sorting execution module (2) to control the movement and positioning of the mobile carrier module (5).

7. The vision-based intelligent sorting method for building materials according to claim 6, characterized in that, The process of using the data augmentation unit (33) to perform dataset augmentation training on the building material positioning data to obtain an augmented material distribution dataset includes: The data augmentation unit (33) performs dataset augmentation training on the building material positioning data based on the Mosaic data augmentation method to obtain the augmented material distribution dataset.

8. A vision-based intelligent sorting method for building materials according to claim 6, characterized in that, The step of inputting the material identification result into the CBAM module (32) for optimization processing and outputting the optimized material identification result includes: The CBAM module (32) optimizes the material recognition result based on the channel attention mechanism and the spatial attention mechanism, and outputs the optimized material recognition result.

9. A vision-based intelligent sorting method for building materials according to claim 6, characterized in that, The vision-based intelligent sorting method for building materials also includes; The remote monitoring system evaluates and provides feedback on the optimized material identification results and the path planning and positioning control information to obtain evaluation feedback results; Based on the evaluation feedback results, optimization information is generated, and the optimization information is input into the deep learning processing module (3) and the intelligent sorting execution module (2) for optimization training to obtain the optimized training model and execution control strategy; The optimized training model is input into the YOLO algorithm processing unit (31) and the CBAM module (32) to update the recognition algorithm and optimization strategy, and obtain the updated material recognition result. The updated material recognition result is used to improve the recognition accuracy and target positioning accuracy of the YOLO algorithm processing unit (31) and the CBAM module (32). The optimized execution control strategy is input into the intelligent sorting execution module (2) to update the execution control parameters of sorting and handling operations, and to obtain updated path control information. The updated path control information is used to optimize the motion path and grasping accuracy of the intelligent sorting execution module (2).

Images