Scrap Steel Grabbing Guidance and Recognition System Based on Multi-Sensor Fusion and Deep Learning

Abstract

This invention discloses a scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning. The system includes: a main controller, a 3D point cloud image acquisition module, an RGB image acquisition module, an RGB-D image fusion module, an RGB-D image segmentation module, a grabbing point calculation module, a scrap steel identification module, and a data storage module. This invention uses multi-sensor fusion technology to overcome the shortcomings of previous single sensors, which could not effectively segment point cloud data and could not identify the grabbed scrap steel. It can effectively avoid safety problems caused by incorrect grabbing center of gravity of ultra-long scrap steel, and can calculate the grade of scrap steel grabbed each time, providing important information for storage and inventory in the warehouse, greatly improving the intelligence level of steel plants and reducing the risks of manual operation.

Description

Technical Field

[0001] This invention relates to the field of information technology in the steel smelting process, and more specifically, to a scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning. Background Technology

[0002] Due to the varying lengths and shapes of scrap steel, crane operators need a high level of skill to handle its entry and exit from the warehouse. Otherwise, strips of scrap steel can easily slip during the grabbing process, causing injury to equipment or personnel, or the scrap steel may shift, increasing the difficulty of placing it in the warehouse and reducing the utilization rate of the warehouse area. This invention provides guidance for the crane to grab scrap steel from entry to exit, and automatically identifies the scrap steel being grabbed. Compared to manual grabbing and sorting, it improves the automation and unmanned operation level of the warehouse, and has a fast calculation speed and high accuracy. Summary of the Invention

[0003] In view of the above problems, the purpose of this invention is to provide a scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning, which can solve the problem that unmanned overhead cranes cannot grab scrap steel of different types and volumes.

[0004] This invention provides a scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning, comprising:

[0005] The main controller includes a 3D point cloud image acquisition module, an RGB image acquisition module, an RGB-D image fusion module, an RGB-D image segmentation module, a grasping point calculation module, a scrap steel recognition module, and a data storage module.

[0006] The main controller is used to collect the crane coordinates recorded by the crane coding cable and determine the crane displacement value based on the crane coordinates.

[0007] The 3D point cloud image acquisition module is used to acquire 3D point cloud images during the crane's movement.

[0008] The RGB image acquisition module is used to acquire RGB images within the target area;

[0009] The RGB-D image fusion module is used to convert 3D point cloud images into depth maps and then fuse them with RGB images to generate RGB-D images;

[0010] The RGB-D image segmentation module is used to segment the scrap steel in the RGB-D image into individual scrap steel blocks;

[0011] The gripping point calculation module is used to calculate the gripping midpoint of the scrap steel blocks that have been divided into individual blocks.

[0012] The scrap steel identification module is used to identify the type and corresponding grade of the scrap steel blocks that have been divided into individual blocks;

[0013] The data storage module is used for data storage.

[0014] In this solution, the main controller is further configured to:

[0015] Determine whether the overhead crane has shifted; if so, generate a 3D point cloud image acquisition information.

[0016] The 3D point cloud acquisition signal is sent to the 3D point cloud image acquisition module, and the 3D point cloud is aligned with the crane coordinates, that is, each data acquisition will be bound to the corresponding crane coordinates.

[0017] After the overhead crane comes to a complete stop, determine whether the crane's position is within the set acquisition area. If so, generate an RGB image acquisition signal.

[0018] The RGB image acquisition signal is sent to the RGB image acquisition module.

[0019] In this solution, the RGB-D image fusion module includes a 3D point cloud image to depth map unit, an RGB image conversion unit, and a multi-channel image fusion unit;

[0020] The 3D point cloud to depth map unit is used to convert 3D point cloud data into crane coordinates according to the crane coordinates corresponding to each row of data, and the resulting depth map is aligned with the crane coordinate system.

[0021] The RGB image conversion unit is used to convert the RGB image into coordinates according to the crane position and calibration parameters. After conversion, the RGB image and the crane are in the same Cartesian coordinates.

[0022] The multi-channel image fusion unit is used to combine the converted depth map and the RGB image into an image containing 4 channels: red channel, green channel, blue channel, and depth channel; the corresponding image is in the same Cartesian coordinate system as the crane.

[0023] In this solution, the RGB-D image segmentation module includes: a scrap steel positioning unit and a scrap steel segmentation unit;

[0024] The scrap steel positioning unit is used to determine whether scrap steel exists in the RGB-D image and to determine the position coordinates of the corresponding scrap steel.

[0025] The scrap steel segmentation unit includes an RGB-D image segmentation subunit and a scrap steel segmentation subunit;

[0026] The RGB-D image segmentation subunit is used to calculate individual scrap steel block images based on the location coordinates of the scrap steel. The coordinate values ​​obtained in the RGB-D image are the coordinate values ​​corresponding to the overhead crane.

[0027] The scrap steel segmentation subunit is used to segment the scrap steel according to the individual scrap steel block images to obtain individual scrap steel blocks.

[0028] In this solution, the step of determining the location coordinates of the corresponding scrap steel is specifically as follows:

[0029] Retrieve historical RGB-D images;

[0030] Historical RGB-D images are preprocessed to obtain training samples;

[0031] Using SSD as the basic model framework, an input layer of a preset first specification is constructed, and the scrap steel region of the training samples is calibrated to train a localization model.

[0032] The RGB-D image is input into the localization model to determine whether scrap steel exists in the corresponding RGB-D image. If it does, the location coordinates of the scrap steel are output.

[0033] In this solution, the step of calculating an individual scrap steel block image based on the location coordinates of the scrap steel is specifically as follows:

[0034] Using the output coordinates of the scrap steel as a reference, the location of the scrap steel is expanded according to the preset first expansion range, and the image is cut out to obtain a rectangular image;

[0035] After the rectangular image is processed by histogram mean normalization, it is converted into an image a of the preset second size.

[0036] The image 'a' is input into a preset instance segmentation network to segment the scrap steel blocks, resulting in individual scrap steel block images.

[0037] In this solution, the grasping point calculation module is also used for:

[0038] Based on a preset point calculation algorithm, the highest point in the region of the RGB-D image is extracted as the optimal point for capturing.

[0039] Based on the preset grabbing radius, the grabbing range is constructed with the optimal point as the center;

[0040] Obtain the volume of the scrap steel block;

[0041] Determine whether the volume of the scrap steel block is greater than the grabbing range. If so, calculate the center of gravity of the corresponding scrap steel block and determine the grabbing center based on the corresponding center of gravity.

[0042] If not, then the entire scrap steel block is set as the grab center.

[0043] In this solution, the step of determining the grasping center based on the corresponding centroid point is specifically as follows:

[0044] When the corresponding center of gravity is on the corresponding scrap block, the corresponding center of gravity is set as the gripping center;

[0045] When the corresponding center of gravity is not on the corresponding scrap steel block, the corresponding scrap steel block is divided into multiple scrap steel sub-blocks with a volume smaller than or equal to the preset grabbing range.

[0046] The scrap steel blocks that are smaller than or equal to the preset grasping range are set as the grasping center as a whole.

[0047] In this solution, the scrap steel identification module further includes: a scrap steel type identification unit and a scrap steel grade identification unit;

[0048] The scrap steel type identification unit is used to identify the type of scrap steel that has been divided into individual scrap steel blocks;

[0049] The scrap steel grade identification unit is used to identify the grade of scrap steel that has been divided into individual scrap steel blocks;

[0050] The step of identifying the type of scrap steel that has been divided into individual blocks specifically includes:

[0051] Obtain the coordinates of the scrap steel block;

[0052] Determine the volume value of the corresponding scrap steel block based on its coordinates;

[0053] Obtain the weight value of the scrap steel block;

[0054] Divide the weight of the scrap steel block by its volume to obtain its average density.

[0055] The type of scrap steel block is determined based on the preset density range into which the average density value of the scrap steel block falls;

[0056] The step of identifying the grade of the scrap steel that has been divided into individual blocks specifically includes:

[0057] Obtain the external shape characteristics and corresponding characteristic values ​​of the scrap steel blocks;

[0058] Based on the preset weight coefficient, the corresponding feature value is multiplied by the corresponding preset weight coefficient to obtain the weight value of the corresponding feature;

[0059] The total weight value is obtained by summing the weight values ​​of all features.

[0060] The grade of the scrap steel block is obtained based on the preset weight range into which the total weight value falls.

[0061] This invention discloses a scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning. The multi-sensor fusion technology overcomes the shortcomings of previous single sensors, which could not effectively segment point cloud data and could not identify the grabbed scrap steel. It can effectively avoid safety problems caused by incorrect grabbing center of gravity of ultra-long scrap steel, and can calculate the grade of scrap steel grabbed each time, providing important information for storage and inventory in the warehouse, greatly improving the intelligence level of steel plants and reducing the risks of manual operation. Attached Figure Description

[0062] Figure 1 A block diagram of the scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning of the present invention is shown. Detailed Implementation

[0063] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0064] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0065] Figure 1 A block diagram of the scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning of the present invention is shown.

[0066] like Figure 1 As shown, this invention discloses a scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning, comprising:

[0067] The main controller includes a 3D point cloud image acquisition module, an RGB image acquisition module, an RGB-D image fusion module, an RGB-D image segmentation module, a grasping point calculation module, a scrap steel recognition module, and a data storage module.

[0068] The main controller is used to collect the crane coordinates recorded by the crane coding cable and determine the crane displacement value based on the crane coordinates.

[0069] The 3D point cloud image acquisition module is used to acquire 3D point cloud images during the crane's movement.

[0070] The RGB image acquisition module is used to acquire RGB images within the target area;

[0071] The RGB-D image fusion module is used to convert 3D point cloud images into depth maps and then fuse them with RGB images to generate RGB-D images;

[0072] The RGB-D image segmentation module is used to segment the scrap steel in the RGB-D image into individual scrap steel blocks;

[0073] The gripping point calculation module is used to calculate the gripping midpoint of the scrap steel blocks that have been divided into individual blocks.

[0074] The scrap steel identification module is used to identify the type and corresponding grade of the scrap steel blocks that have been divided into individual blocks;

[0075] The data storage module is used for data storage.

[0076] According to an embodiment of the present invention, a crane coding cable and a distance sensor are electrically connected at the input terminal of the main controller. The crane coordinates are obtained through the crane coding cable, and the displacement value of the crane is determined based on the crane coordinates and the distance sensor. The displacement value of the crane is judged, and the 3D point cloud image acquisition module and the RGB image acquisition module are triggered according to the corresponding judgment result, thereby collecting 3D point cloud data and RGB images of scrap steel. Based on the grabbing midpoint of the scrap steel block, the type of scrap steel block and the corresponding grade, the scrap steel block is accurately grabbed and placed in the accurate storage warehouse.

[0077] According to an embodiment of the present invention, the main controller is further configured to:

[0078] Determine whether the overhead crane has shifted; if so, generate a 3D point cloud image acquisition information.

[0079] The 3D point cloud acquisition signal is sent to the 3D point cloud image acquisition module, and the 3D point cloud is aligned with the crane coordinates, that is, each data acquisition will be bound to the corresponding crane coordinates.

[0080] After the overhead crane comes to a complete stop, determine whether the crane's position is within the set acquisition area. If so, generate an RGB image acquisition signal.

[0081] The RGB image acquisition signal is sent to the RGB image acquisition module.

[0082] It should be noted that by setting intervals for transportation, sufficient reaction time is allowed for each module. If the displacement value of the overhead crane is exactly within the preset displacement range, it indicates that there is scrap steel under the current overhead crane, and the scrap steel is exactly within the scanning range of the 3D point cloud acquisition device in the 3D point cloud image acquisition module on the overhead crane, and within the shooting range of the RGB image acquisition device located on the overhead crane. The scrap steel is scanned by the 3D point cloud acquisition device to obtain the corresponding 3D point cloud data of the scrap steel; the scrap steel is photographed by the RGB image acquisition device to obtain the RGB image of the scrap steel.

[0083] According to an embodiment of the present invention, the RGB-D image fusion module includes a 3D point cloud image to depth map unit, an RGB image conversion unit, and a multi-channel image fusion unit;

[0084] The 3D point cloud to depth map unit is used to convert 3D point cloud data into crane coordinates according to the crane coordinates corresponding to each row of data, and the resulting depth map is aligned with the crane coordinate system.

[0085] The RGB image conversion unit is used to convert the RGB image into coordinates according to the crane position and calibration parameters. After conversion, the RGB image and the crane are in the same Cartesian coordinates.

[0086] The multi-channel image fusion unit is used to combine the converted depth map and the RGB image into an image containing 4 channels: red channel, green channel, blue channel, and depth channel; the corresponding image is in the same Cartesian coordinate system as the crane.

[0087] It should be noted that during the crane's movement, the RGB-D image fusion module converts the collected 3D point cloud data into depth map data, and then aligns the coordinate system of the depth map data with the crane coordinate system through rigid RT transformation; the depth map and RGB image are then fused to obtain an RGB-D image, which contains four channels: red channel, green channel, blue channel and depth channel.

[0088] According to an embodiment of the present invention, the RGB-D image segmentation module includes: a scrap steel positioning unit and a scrap steel segmentation unit;

[0089] The scrap steel positioning unit is used to determine whether scrap steel exists in the RGB-D image and to determine the position coordinates of the corresponding scrap steel.

[0090] The scrap steel segmentation unit includes an RGB-D image segmentation subunit and a scrap steel segmentation subunit;

[0091] The RGB-D image segmentation subunit is used to calculate individual scrap steel block images based on the location coordinates of the scrap steel. The coordinate values ​​obtained in the RGB-D image are the coordinate values ​​corresponding to the overhead crane.

[0092] The scrap steel segmentation subunit is used to segment the scrap steel according to the individual scrap steel block images to obtain individual scrap steel blocks.

[0093] It should be noted that the scrap steel is located using the RGB-D image segmentation module. If segmentation is required, the scrap steel is segmented to obtain individual scrap steel blocks. After segmentation, the individual scrap steel blocks need to be relocated.

[0094] According to an embodiment of the present invention, the step of determining the position coordinates of the corresponding scrap steel specifically includes:

[0095] Retrieve historical RGB-D images;

[0096] Historical RGB-D images are preprocessed to obtain training samples;

[0097] Using SSD as the basic model framework, an input layer of a preset first specification is constructed, and the scrap steel region of the training samples is calibrated to train a localization model.

[0098] The RGB-D image is input into the localization model to determine whether scrap steel exists in the corresponding RGB-D image. If it does, the location coordinates of the scrap steel are output.

[0099] It should be noted that the SSD model is the Single Shot MultiBox Detector model. For example, if the preset first specification is 416*416, then a 416*416 input layer is constructed. The preset background image is the image without scrap steel.

[0100] According to an embodiment of the present invention, the step of calculating an image of a single scrap steel block based on the location coordinates of the scrap steel specifically includes:

[0101] Using the output coordinates of the scrap steel as a reference, the location of the scrap steel is expanded according to the preset first expansion range, and the image is cut out to obtain a rectangular image;

[0102] After the rectangular image is processed by histogram mean normalization, it is converted into an image a of the preset second size.

[0103] The image 'a' is input into a preset instance segmentation network to segment the scrap steel blocks, resulting in individual scrap steel block images.

[0104] It should be noted that, for example, if the first expansion range is preset to 35%, then the area is expanded by 35% based on the location coordinates of the scrap steel. This is to better obtain the corner positions of the area and prevent the influence of blind spots. For example, if the second specification size is preset to 224*224, then the input layer of the preset strength segmentation network is 224*224. Scrap steel is composed of multiple scrap steel blocks. If scrap steel is composed of a single scrap steel block, then the corresponding scrap steel block is scrap steel.

[0105] According to an embodiment of the present invention, the grab point calculation module is further configured to:

[0106] Based on a preset point calculation algorithm, the highest point in the region of the RGB-D image is extracted as the optimal point for capturing.

[0107] Based on the preset grabbing radius, the grabbing range is constructed with the optimal point as the center;

[0108] Obtain the volume of the scrap steel block;

[0109] Determine whether the volume of the scrap steel block is greater than the grabbing range. If so, calculate the center of gravity of the corresponding scrap steel block and determine the grabbing center based on the corresponding center of gravity.

[0110] If not, then the entire scrap steel block is set as the grab center.

[0111] It should be noted that when the volume of the scrap steel block is less than or equal to the grabbing range, the overhead crane can grab the entire scrap steel block without considering the center of gravity of the corresponding scrap steel block; when the volume of the scrap steel block is greater than the grabbing range, the weight of the corresponding scrap steel block may exceed the balance weight of the corresponding overhead crane. Therefore, it is necessary to stabilize the center of gravity of the scrap steel block to grab it smoothly.

[0112] According to an embodiment of the present invention, the step of determining the grasping center based on the corresponding centroid point specifically includes:

[0113] When the corresponding center of gravity is on the corresponding scrap block, the corresponding center of gravity is set as the gripping center;

[0114] When the corresponding center of gravity is not on the corresponding scrap steel block, the corresponding scrap steel block is divided into multiple scrap steel sub-blocks with a volume smaller than or equal to the preset grabbing range.

[0115] The scrap steel blocks that are smaller than or equal to the preset grasping range are set as the grasping center as a whole.

[0116] It should be noted that determining the gripping center of the corresponding scrap steel block by its volume and center of gravity is a simple and effective method.

[0117] According to an embodiment of the present invention, the scrap steel identification module further includes: a scrap steel type identification unit and a scrap steel grade identification unit;

[0118] The scrap steel type identification unit is used to identify the type of scrap steel that has been divided into individual scrap steel blocks;

[0119] The scrap steel grade identification unit is used to identify the grade of scrap steel that has been divided into individual scrap steel blocks;

[0120] The step of identifying the type of scrap steel that has been divided into individual blocks specifically includes:

[0121] Obtain the coordinates of the scrap steel block;

[0122] Determine the volume value of the corresponding scrap steel block based on its coordinates;

[0123] Obtain the weight value of the scrap steel block;

[0124] Divide the weight of the scrap steel block by its volume to obtain its average density.

[0125] The type of scrap steel block is determined based on the preset density range into which the average density value of the scrap steel block falls;

[0126] The step of identifying the grade of the scrap steel that has been divided into individual blocks specifically includes:

[0127] Obtain the external shape characteristics and corresponding characteristic values ​​of the scrap steel blocks;

[0128] Based on the preset weight coefficient, the corresponding feature value is multiplied by the corresponding preset weight coefficient to obtain the weight value of the corresponding feature;

[0129] The total weight value is obtained by summing the weight values ​​of all features.

[0130] The grade of the scrap steel block is obtained based on the preset weight range into which the total weight value falls.

[0131] It should be noted that the coordinates of the scrap steel block are all the coordinates of the outer surface of the corresponding scrap steel block. By using all the coordinates of the outer surface of the corresponding scrap steel block, the shape diagram of the corresponding scrap steel block is constructed, thereby determining the volume value of the corresponding scrap steel block. One preset density range corresponds to the type of scrap steel block. The type of scrap steel block is found according to the preset density range into which the average density value of the scrap steel block falls. Different preset weight ranges correspond to different grades of scrap steel blocks, and one preset weight range corresponds to one grade of scrap steel block. The shape characteristics of the scrap steel block include rust coverage rate, defect ratio, etc. For example, the rust coverage rate is the area of ​​rust on the scrap steel block divided by the total surface area of ​​the scrap steel block. The higher the grade of the scrap steel block, the lower the possibility that the corresponding scrap steel block can be reused, and the greater the possibility that the corresponding scrap steel block will be melted down and recast.

[0132] This invention discloses a scrap steel grabbing guidance and identification system based on multi-sensor fusion and deep learning. The system includes: a main controller, a 3D point cloud image acquisition module, an RGB image acquisition module, an RGB-D image fusion module, an RGB-D image segmentation module, a grabbing point calculation module, a scrap steel identification module, and a data storage module. This invention uses multi-sensor fusion technology to overcome the shortcomings of previous single sensors, which could not effectively segment point cloud data and could not identify the grabbed scrap steel. It can effectively avoid safety problems caused by incorrect grabbing center of gravity of ultra-long scrap steel, and can calculate the grade of scrap steel grabbed each time, providing important information for storage and inventory in the warehouse, greatly improving the intelligence level of steel plants and reducing the risks of manual operation.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0134] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0136] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A scrap steel grabbing, guiding, and identification system based on multi-sensor fusion and deep learning, characterized in that, include: The main controller includes a 3D point cloud image acquisition module, an RGB image acquisition module, an RGB-D image fusion module, an RGB-D image segmentation module, a grasping point calculation module, a scrap steel recognition module, and a data storage module. The main controller is used to collect the crane coordinates recorded by the crane coding cable and determine the crane displacement value based on the crane coordinates. The 3D point cloud image acquisition module is used to acquire 3D point cloud images during the crane's movement. The RGB image acquisition module is used to acquire RGB images within the target area; The RGB-D image fusion module is used to convert 3D point cloud images into depth maps and then fuse them with RGB images to generate RGB-D images; The RGB-D image segmentation module is used to segment the scrap steel in the RGB-D image into individual scrap steel blocks; The gripping point calculation module is used to calculate the gripping midpoint of the scrap steel blocks that have been divided into individual blocks. The grab point calculation module is also used for: Based on a preset point calculation algorithm, the highest point in the region of the RGB-D image is extracted as the optimal point for capturing. Based on the preset grabbing radius, the grabbing range is constructed with the optimal point as the center; Obtain the volume of the scrap steel block; Determine whether the volume of the scrap steel block is greater than the grabbing range. If so, calculate the center of gravity of the corresponding scrap steel block and determine the grabbing center based on the corresponding center of gravity. If not, then the entire scrap metal block is set as the grab center; The step of determining the grasping center based on the corresponding centroid point is as follows: When the corresponding center of gravity is on the corresponding scrap block, the corresponding center of gravity is set as the gripping center; When the corresponding center of gravity is not on the corresponding scrap steel block, the corresponding scrap steel block is divided into multiple scrap steel sub-blocks with a volume smaller than or equal to the preset grabbing range. The scrap steel blocks that are smaller than or equal to the preset grasping range are set as the grasping center as a whole; The scrap steel identification module is used to identify the type and corresponding grade of the scrap steel blocks that have been divided into individual blocks; The data storage module is used for data storage.

2. The scrap steel grabbing, guiding, and identification system based on multi-sensor fusion and deep learning according to claim 1, characterized in that, The main controller is also used for: Determine whether the overhead crane has shifted; if so, generate a 3D point cloud image acquisition information. The 3D point cloud acquisition signal is sent to the 3D point cloud image acquisition module, and the 3D point cloud is aligned with the crane coordinates, that is, each data acquisition will be bound to the corresponding crane coordinates. After the overhead crane comes to a complete stop, determine whether the crane's position is within the set acquisition area. If so, generate an RGB image acquisition signal. The RGB image acquisition signal is sent to the RGB image acquisition module.

3. The scrap steel grabbing, guiding, and identification system based on multi-sensor fusion and deep learning according to claim 1, characterized in that, The RGB-D image fusion module includes a 3D point cloud to depth map unit, an RGB image conversion unit, and a multi-channel image fusion unit; The 3D point cloud to depth map unit is used to convert 3D point cloud data into crane coordinates according to the crane coordinates corresponding to each row of data, and the resulting depth map is aligned with the crane coordinate system. The RGB image conversion unit is used to convert the RGB image into coordinates according to the crane position and calibration parameters. After conversion, the RGB image and the crane are in the same Cartesian coordinates. The multi-channel image fusion unit is used to combine the converted depth map and the RGB image into an image containing 4 channels: red channel, green channel, blue channel, and depth channel. The corresponding image is in the same Cartesian coordinate system as the overhead crane.

4. The scrap steel grabbing, guiding, and identification system based on multi-sensor fusion and deep learning according to claim 1, characterized in that, The RGB-D image segmentation module includes: a scrap steel positioning unit and a scrap steel segmentation unit; The scrap steel positioning unit is used to determine whether scrap steel exists in the RGB-D image and to determine the position coordinates of the corresponding scrap steel. The scrap steel segmentation unit includes an RGB-D image segmentation subunit and a scrap steel segmentation subunit; The RGB-D image segmentation subunit is used to calculate individual scrap steel block images based on the location coordinates of the scrap steel. The coordinate values ​​obtained in the RGB-D image are the coordinate values ​​corresponding to the overhead crane. The scrap steel segmentation subunit is used to segment the scrap steel according to the individual scrap steel block images to obtain individual scrap steel blocks.

5. The scrap steel grabbing, guiding, and identification system based on multi-sensor fusion and deep learning according to claim 4, characterized in that, The step of determining the location coordinates of the corresponding scrap steel is as follows: Retrieve historical RGB-D images; Historical RGB-D images are preprocessed to obtain training samples; Using SSD as the basic model framework, an input layer of a preset first specification is constructed, and the scrap steel region of the training samples is calibrated to train a localization model. The RGB-D image is input into the localization model to determine whether scrap steel exists in the corresponding RGB-D image. If it does, the location coordinates of the scrap steel are output.

6. The scrap steel grabbing, guiding, and identification system based on multi-sensor fusion and deep learning according to claim 4, characterized in that, The step of calculating an individual scrap steel block image based on the location coordinates of the scrap steel is as follows: Using the output coordinates of the scrap steel as a reference, the location of the scrap steel is expanded according to the preset first expansion range, and the image is cut out to obtain a rectangular image; After the rectangular image is processed by histogram mean normalization, it is converted into an image a of the preset second size. The image a is input into a preset instance segmentation network to segment the scrap steel blocks, resulting in individual scrap steel block images.

7. The scrap steel grabbing, guiding, and identification system based on multi-sensor fusion and deep learning according to claim 1, characterized in that, The scrap steel identification module further includes: a scrap steel type identification unit and a scrap steel grade identification unit; The scrap steel type identification unit is used to identify the type of scrap steel that has been divided into individual scrap steel blocks; The scrap steel grade identification unit is used to identify the grade of scrap steel that has been divided into individual scrap steel blocks; The step of identifying the type of scrap steel that has been divided into individual blocks specifically includes: Obtain the coordinates of the scrap steel block; Determine the volume value of the corresponding scrap steel block based on its coordinates; Obtain the weight value of the scrap steel block; Divide the weight of the scrap steel block by its volume to obtain its average density. The type of scrap steel block is determined based on the preset density range into which the average density value of the scrap steel block falls; The step of identifying the grade of the scrap steel that has been divided into individual blocks specifically includes: Obtain the external shape characteristics and corresponding characteristic values ​​of the scrap steel blocks; Based on the preset weight coefficient, the corresponding feature value is multiplied by the corresponding preset weight coefficient to obtain the weight value of the corresponding feature; The weight values ​​of all features are summed to obtain the total weight value; the grade of the corresponding scrap steel block is obtained according to the preset weight range in which the total weight value falls.

Images