Unpacking action recognition system and method for automatic grading of recycled steel packing block material types
By combining multi-camera parallax and deep learning algorithms with edge processing and deep learning, the accuracy problem of recognizing the unpacking action of recycled steel block materials was solved, and high-precision evaluation of automatic classification was achieved.
Patent Information
- Application Number
- CN202511845930.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-03-17
AI Technical Summary
In the process of automatically classifying the shape of recycled steel baled blocks, it is difficult to accurately identify the degree of unpacking, especially in complex on-site backgrounds where lighting and occlusion issues affect the accuracy of image information acquisition.
Using multi-camera parallax as depth information, combined with deep learning and machine learning algorithms, an edge processing calculator controls an industrial camera to collect two-dimensional data, constructs depth information, and performs feature fusion and classification in a deep learning server to identify unpacking actions.
This improved the accuracy of unpacking actions, ensured the accuracy of automatic grading, and provided a precise basis for the subsequent evaluation of recycled steel baled block shapes.
Smart Images
Figure CN121671996A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of recycled steel baled block material type recognition technology, specifically to a system and method for automatically classifying the unpacking action of recycled steel baled blocks. Background Technology
[0002] In the actual automatic grading process of recycled steel baled blocks, it is necessary to detect whether the unpacking machine has completely unpacked the baled blocks. Since the compaction density of different scrap steel baled blocks varies, the degree of unpacking significantly affects the automatic grading results. This necessitates high precision in determining "whether the baled blocks have been completely unpacked" during the automatic scrap steel inspection process. However, complex background, lighting, and occlusion conditions at the scene make it difficult to obtain a detailed understanding of the scene using only color image information. Summary of the Invention
[0003] The purpose of this invention is to provide a system and method for automatically classifying the unpacking action of recycled steel bales, using the parallax of multiple cameras in the unpacking process as depth information to solve the problem of accurately obtaining the unpacking accuracy in intelligent scrap steel inspection.
[0004] To achieve the above objectives, the present invention provides the following technical solution:
[0005] An automatic classification system for unpacking actions of recycled steel baled blocks includes a data analysis server, an unpacking machine controller, an edge processing calculator A, an edge processing calculator B, an industrial camera A, and an industrial camera B. The data analysis server, unpacking machine controller, edge processing calculator A, and edge processing calculator B are all connected via a gigabit switch. The data analysis server issues unpacking commands to the unpacking machine controller, which controls the unpacking device for the recycled steel baled blocks. The unpacking machine controller sends signals to edge processing calculators A and B via the gigabit switch. Edge processing calculator A controls industrial camera A, and edge processing calculator B controls industrial camera B. Upon receiving a start signal, edge processing calculator A controls industrial camera A to collect data, and edge processing calculator B, upon receiving a start signal, collects data from industrial camera B and provides real-time feedback on the unpacking status to the data analysis server.
[0006] Furthermore, the industrial camera A forms the first data acquisition module, and the industrial camera B forms the second data acquisition module. The first data acquisition module and the second data acquisition module jointly perform two-dimensional data acquisition on the unpacking device of the recycled steel baled block material, and construct depth information based on this two-dimensional data.
[0007] This invention provides another technical solution: a method for automatically classifying the unpacking action of recycled steel baled blocks, comprising the following steps:
[0008] S1: The data analysis server sends unpacking commands to the unpacking machine controller via a gigabit switch;
[0009] S2: The unpacking machine controller determines that the unpacking machine has been successfully controlled and sends the start unpacking command to the edge processing calculator through the gigabit switch;
[0010] S3: Edge processing calculator controls industrial camera to acquire two-dimensional data of the unpacking device for the current recycled steel baled block shape;
[0011] S4: The edge processing calculator determines whether the image quality requires reshooting, then fuses the two-dimensional data and transforms it into the world coordinate system;
[0012] S5: The edge processing calculator calculates the degree of disassembly of the material type currently being unpacked and transmits it to the deep learning server. The deep learning server then determines whether the unpacking meets the automatic detection criteria.
[0013] S6: The data from all unpacking processes are fused, and the deep learning server extracts the fused data from the edge computing to trigger automatic grading of recycled steel.
[0014] Furthermore, the edge processing calculator in S4 includes scene determination using deep learning and machine learning algorithms on two-dimensional images, as well as joint calculation of the current scene determination by fusing two-dimensional data and deep data.
[0015] Furthermore, the S5 deep learning server internally constructs a deep network structure: one branch receives RGB images as input, using the Xcept version of the network structure; the other branch receives depth images as input, using a self-convolutional network structure composed of alternating convolutional and pooling layers. Simultaneously, the deep learning server uses both the image and the disparity image as input, extracts features from both through a deep neural network, and combines the appearance information of the RGB image with the depth information of the disparity image through feature fusion. Finally, it uses the fused features to classify each pixel of the image to obtain the predicted output.
[0016] Furthermore, when using a self-convolutional network to receive a depth image as input, the ratio of the input image resolution to the output rate is denoted as the output stride, where the expression for convolution multiplied by the output size is:
[0017]
[0018] In the formula, O is the size of the output feature map; I is the size of the input feature map; K is the size of the convolution kernel; P is the padding; and S is the stride.
[0019] Furthermore, a Spatial Pyramid Pooling (ASPP) module is introduced into the self-convolutional network structure. The ASPP module is used to capture multi-scale contextual features and extracts features through convolution operations with different dilation rates. The specific formula is as follows:
[0020]
[0021] In the formula, F ASPP For the output characteristics of the ASPP module; W i The weights are those of the i-th convolution operation; To use the expansion ratio d i The convolution operation; N is the number of parallel convolution operations.
[0022] Furthermore, the feature fusion expression is as follows:
[0023]
[0024] In the formula: F fused The fused feature map; F RGB F is a feature map extracted from an RGB image. Depth This is the feature map extracted from the depth image; α and β are weighting coefficients.
[0025] Compared with the prior art, the beneficial effects of the present invention are:
[0026] The present invention relates to an automatic classification system and method for identifying unpacking actions of recycled steel bales. By employing multi-edge processing calculators for joint calculation and combining the linkage control signals of on-site control equipment, it effectively solves the problem in practical applications where "automatic identification systems for unpacking actions of recycled steel bales are difficult to accurately determine the degree of unpacking." This lays the foundation for accurate evaluation of the subsequent automatic classification of recycled steel bales. At the same time, it uses the parallax of multiple cameras in the dismantling process of steel bales as depth information to solve the problem of accurately obtaining the unpacking accuracy of intelligent scrap steel inspection. Attached Figure Description
[0027] Figure 1 This is a system framework diagram of the present invention;
[0028] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation
[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0030] Please see Figure 1 This invention provides an automatic classification system for unpacking actions of recycled steel baled blocks, comprising a data analysis server, an unpacking machine controller, an edge processing calculator A, an edge processing calculator B, an industrial camera A, and an industrial camera B. The data analysis server, unpacking machine controller, edge processing calculator A, and edge processing calculator B are all connected via a gigabit switch. The data analysis server issues unpacking commands to the unpacking machine controller, which controls the unpacking device for recycled steel baled blocks. The unpacking machine controller sends signals to edge processing calculator A and edge processing calculator B via the gigabit switch. Edge processing calculator A controls industrial camera A, and edge processing calculator B controls industrial camera B. Upon receiving a start signal, edge processing calculator A controls industrial camera A to collect data, and edge processing calculator B receives a start signal to collect data from industrial camera B, providing real-time feedback on the unpacking status to the data analysis server. Industrial camera A forms the first data acquisition module, and industrial camera B forms the second data acquisition module. The first and second data acquisition modules work together to acquire two-dimensional data from the unpacking device of the recycled steel baled block material, and use this two-dimensional data to construct depth information.
[0031] In the above embodiments, due to the complex background, lighting, and occlusion issues at the scene, it is difficult to obtain a detailed understanding of the scene by relying solely on color image information. This embodiment proposes to use the parallax of multiple cameras in the dismantling process of steel ladle dismantling as depth information to solve the problem of accurately obtaining the dismantling accuracy of intelligent scrap steel inspection. The depth information is introduced as prior information into the semantic segmentation framework. With depth as prior information, convolutional neural networks are used to extract the depth features of the image and point cloud. Semantic segmentation is completed based on the fused features through feature fusion. Combined with the linkage control signals of the on-site control equipment, this effectively solves the problem that "the automatic recognition system for the dismantling action of recycled steel bale blocks is difficult to accurately determine the degree of dismantling" in practical applications, laying the foundation for the accurate evaluation of the subsequent automatic classification of recycled steel bale blocks.
[0032] Specifically, such as Figure 2 As shown, this embodiment of the invention provides another technical solution: a method for automatically classifying the unpacking action of recycled steel baled blocks, comprising the following steps:
[0033] S1: The data analysis server sends unpacking commands to the unpacking machine controller via a gigabit switch;
[0034] S2: The unpacking machine controller determines that the unpacking machine has been successfully controlled and sends the start unpacking command to the edge processing calculator through the gigabit switch;
[0035] S3: Edge processing calculator controls industrial camera to acquire two-dimensional data of the unpacking device for the current recycled steel baled block shape;
[0036] S4: The edge processing calculator determines whether the image quality requires reshooting, then fuses the two-dimensional data and transforms it into the world coordinate system;
[0037] S5: The edge processing calculator calculates the degree of disassembly of the material being unpacked and transmits it to the deep learning server. Deep learning is used to determine whether the unpacking meets the automatic detection criteria. Specifically, this includes using deep learning and machine learning algorithms to judge the scene on two-dimensional images, and fusing two-dimensional data and deep data to jointly calculate the current scene judgment.
[0038] S6: The data from all unpacking processes are fused, and the deep learning server extracts the fused data from the edge computing to trigger automatic grading of recycled steel.
[0039] In this embodiment, depth images can provide distance information from objects to the camera and positional relationships between objects, while RGB images can provide rich color information about objects. Both can provide more comprehensive information. Therefore, a deep learning model is used to simultaneously receive RGB images as input, extract their features separately, fuse the features, and then use the fused features to classify each pixel of the image.
[0040] Specifically, a deep network structure is built inside the deep learning server: one branch receives RGB images as input, using the Xcept version of the network structure; the other branch receives depth images as input, using a self-convolutional network structure composed of alternating convolutional and pooling layers. Simultaneously, the deep learning server uses both the RGB image and the disparity image as input, extracting features from both through a deep neural network. The appearance information of the RGB image and the depth information of the disparity image are then combined through feature fusion. Finally, the fused features are used to classify each pixel of the image to obtain the predicted output.
[0041] The RGB branch uses Xception to extract features from RGB images and modifies Xception to better suit semantic segmentation tasks. The modifications are mainly in four aspects: First, the number of Xception network layers is increased from 36 to 65 to enhance its learning ability. Since Xception uses depthwise separable convolutions as its basic components, the number of networks remains relatively low. Second, all max pooling operations are replaced with separable convolutions; downsampling can still be achieved by setting a reasonable kernel stride. Third, batch normalization is performed after each 3×3 depthwise convolutional layer, and ReLU activation is used to introduce more non-linear operations and alleviate gradient vanishing during network training. Finally, the stride in the depthwise convolutions is set to 1, effectively canceling the max pooling operation, and subsequent convolutions are converted to dilated convolutions, allowing the model to increase the receptive field while maintaining resolution. The final network output features... Figure 1 It contains 2048 channels, and the ratio of the resolution to the feature map resolution is set to 16. When using a self-convolutional network to receive depth images as input, the ratio of the input image resolution to the output rate is denoted as the output stride. The expression for convolution multiplication by the output size is:
[0042]
[0043] In the formula, O is the size of the output feature map; I is the size of the input feature map; K is the size of the convolution kernel; P is the padding; and S is the stride.
[0044] In the above embodiment, the disparity branch is used to extract features from the disparity map. A custom fully convolutional network is used to extract features. This model follows the design approach of Xception and consists of five stages composed of depthwise separable convolutions. In order to optimize the step parameters, the feature map output by the convolutional layer contains a small number of channels. The specific structure is shown in the table below. The output disparity feature map contains 256 channels. The ratio of input image resolution to output resolution is 16. The method of preserving the RGB branch is to directly treat the disparity map as a single-channel feature, change its resolution through bilinear interpolation, and use it as a new feature.
[0045]
[0046] In addition, this embodiment introduces a Spatial Pyramid Pooling (ASPP) module into the self-convolutional network structure. The ASPP module is used to capture multi-scale contextual features and extracts features through convolution operations with different dilation rates. The specific formula is as follows:
[0047]
[0048] In the formula, F ASPPFor the output characteristics of the ASPP module; W i The weights are those of the i-th convolution operation; To use the expansion ratio d i The convolution operation; N is the number of parallel convolution operations.
[0049] Specifically, it comprises five parallel channels: a 1×1 convolution, three 3×3 dilated convolutions, and a global average pooling operation. This enables it to capture multi-scale global contextual features, playing a crucial role in semantic segmentation tasks. The ASPP feature map has 256 channels and an output stride of 16. A concise decoding feature map resolution is then used, where the features output by the ASPP module are processed through 3×3 convolutions and bilinearly sampled to obtain the final semantic segmentation prediction.
[0050] The RGB branch and the disparity branch extract features from the RGB image and the disparity image respectively. These two feature branches are then fused so that the fused features simultaneously contain both color and depth information from the scene. The feature fusion expression is as follows:
[0051]
[0052] In the formula: F fused The fused feature map; F RGB F is a feature map extracted from an RGB image. Depth This is the feature map extracted from the depth image; α and β are weighting coefficients.
[0053] In the above method, the present invention uses pixel-wise multiplication. Due to the effect of the nonlinear activation function ReLU, the values of the elements on the feature map are all greater than or equal to 0. Therefore, using pixel-wise multiplication for feature fusion uses the disparity image as a mask to process the RGB features, which can highlight the key areas and suppress irrelevant areas.
[0054] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A system for recognizing unpacking action of automatic grade determination of recycled steel bale type, characterized by, The application relates to a data analysis server, a bale breaker controller, an edge processing calculator A, an edge processing calculator B, an industrial camera A and an industrial camera B; the data analysis server, the bale breaker controller, the edge processing calculator A and the edge processing calculator B are connected through a gigabit switch; the data analysis server is used for issuing a bale breaking instruction to the bale breaker controller, and the bale breaker controller is used for controlling a bale breaking device of a recycled steel bale; the bale breaker controller sends signals to the edge processing calculator A and the edge processing calculator B through the gigabit switch, the edge processing calculator A is used for controlling the industrial camera A, the edge processing calculator B is used for controlling the industrial camera B, the edge processing calculator A controls the industrial camera A to collect data after receiving a starting signal, the edge processing calculator B collects data of the industrial camera B after receiving a starting signal, and the bale breaking state of the bale breaker is fed back to the data analysis server in real time.
2. The system for identifying unpacking motion of a recycled steel bale type automatic grading according to claim 1, characterized in that: The industrial camera A forms a first data collection module, the industrial camera B forms a second data collection module, the first data collection module and the second data collection module jointly collect two-dimensional data of the bale breaking device of the recycled steel bale, and the two-dimensional data is used to construct depth information.
3. The unpacking action recognition method for automatic grading of recycled steel bale type, based on the unpacking action recognition system for automatic grading of recycled steel bale type according to claim 2, characterized in that: The application comprises the following steps: S1: the data analysis server issues a bale breaking instruction to the bale breaker controller through the gigabit switch; S2: the bale breaker controller judges whether the bale breaking control is successful, and issues a starting bale breaking instruction to the edge processing calculator through the gigabit switch; S3: the edge processing calculator controls the industrial camera to collect two-dimensional data of the bale breaking device of the recycled steel bale; S4: the edge processing calculator judges whether the image quality needs to be rephotographed, then fuses the two-dimensional data, and converts the two-dimensional data into a world coordinate system; S5: the edge processing calculator calculates the disassembling degree of the current bale being broken, and transmits the disassembling degree to a deep learning server, so that whether the bale breaking reaches an automatic judging standard is judged through deep learning; S6: all the data of the bale breaking process are fused, and the deep learning server extracts the fused data in the edge calculation and triggers automatic grading of the recycled steel.
4. The method of claim 3, wherein the method is characterized in that: In S4, the edge processing calculator comprises scene judgment using deep learning and machine learning algorithms on two-dimensional images, and joint calculation of the current scene judgment through fusion of two-dimensional data and depth data.
5. The method of claim 3, wherein the method is characterized in that: In S5, the deep learning server internally comprises a deep network structure: one branch is used for receiving an RGB image as input, and the network structure used is an Xcept version; another branch is used for receiving a depth image as input, and the network structure used is a self-convolution network which is composed of convolution layers and pooling layers alternately; meanwhile, the deep learning server uses an image and a parallax image as input, extracts features of the two through a deep neural network, comprehensively integrates apparent information of the RGB image and depth information of the parallax image in a feature fusion manner, and finally classifies each pixel of the image by using the fused features to obtain a prediction output result.
6. The method of claim 5, wherein the method is characterized in that: When the self-convolution network is used to receive the depth image as input, the ratio of the input image resolution to the output rate is recorded as an output step, wherein the convolution multiplication output size expression is: In the formula, O is the size of the output feature map; I is the size of the input feature map; K is the size of the convolution kernel; P is the padding; and S is the stride.
7. The method of claim 6, wherein the method is characterized in that: In the self-convolution network structure, a dilated space pyramid pooling module ASPP is further introduced, the ASPP module is used to capture multi-scale context features, and the features are extracted through convolution operations with different dilated rates, and the specific formula is as follows: In the formula, F ASPP is the output feature of the ASPP module; W i weights for the i-th convolution operation; convolution operation using dilation rate d i N is the number of parallel convolution operations.
8. The method of claim 5, wherein the method is characterized in that: The feature fusion expression is as follows: In the formula: F fused is the fused feature map; F RGB is the feature map extracted from the RGB image; F Depth is the feature map extracted from the depth image; and α and β are weight coefficients.