Binocular camera and laser radar-based strip-shaped steel scrap three-dimensional detection method

Abstract

The invention provides a strip-shaped steel scrap three-dimensional detection method based on a binocular camera and a laser radar, and relates to the field of deep learning and computer vision. The method comprises the following steps: (1) constructing a transformer-based multi-modal three-dimensional detection network, (2) making a data set associated with an image and a point cloud, (3) carrying out fine tuning training on a decoding output network by adopting a transfer learning strategy, and (4) carrying the model on a detection system, thereby realizing real-time acquisition of the category probability and attitude estimation of the strip-shaped scrap steel by using a binocular camera and a laser radar. The device can be used for recycling strip-shaped steel and has the advantages of being accurate in reaction and high in speed.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a posture detection method for three-dimensional measurement of strip scrap steel based on a binocular camera and a laser radar. Background Art

[0002] Target posture recognition and positioning of scrap steel strips refers to the identification of the spatial position, three-dimensional size, and angular orientation of scrap steel strips as they accumulate. Scrap steel recycling is a vital component of my country's steel industry and has played a significant role in its transformation and upgrading.

[0003] Scrap steel recycling comes from complex sources and in a variety of shapes, with strips comprising a significant portion. When stacked, these strips present complex postures and are prone to occlusion, bending, overlapping, and rotational issues. These issues severely impact the accuracy of their classification and assessment, as well as the efficiency of their grasping. Traditional object detection algorithms primarily rely on cameras or radar for measurement and recognition. Traditional cameras can capture rich environmental information and semantic information about objects, effectively enabling object classification and assessment. However, they are sensitive to factors like lighting and humidity, resulting in limited distance perception and poor adaptability to industrial dust. While binocular cameras offer significant improvements in distance perception, they are also subject to interference from occlusion and reflections. Furthermore, equipment vibration, temperature fluctuations, and mechanical shock can cause offsets in the camera's calibration parameters, impacting algorithm accuracy. While lidar (LiDAR) can obtain precise distance information and accurately measure the three-dimensional pose of objects, its sparse sampling and poor semantic perception make it difficult to assess the surface quality of scrap steel. These sensors often encounter problems such as non-recognition, incorrect recognition, deviations in recognized posture and size, and large errors in estimated target distance when used in actual industrial production scenarios. These issues hinder effective recognition and resolution in real-world production scenarios, hindering the automation and efficiency of scrap steel recycling. Therefore, multimodal recognition can leverage the strengths of both sensors to overcome their weaknesses and achieve precise recognition.

[0004] The fusion of binocular cameras and lidar, along with a new algorithm, can significantly improve the recognition accuracy and robustness of piled steel bars. It has extremely strong recognition stability and is not easily affected by the environment. It provides reliable data support for subsequent quality assessment of scrap steel bars and automatic grabbing and recycling using robotic arms.

[0005] CN111951305A, a method for target detection and motion state estimation based on vision and laser radar, includes the following steps: 1. Installing the camera and laser radar positions, and performing calibration between each pair of binocular cameras, as well as joint calibration between the front left camera and the laser radar. The sparse point clouds generated by the stereo vision system and the laser radar are then fused and supplemented to form a global three-dimensional point cloud map. 2. Using deep learning methods to detect the two-dimensional image and three-dimensional point cloud, the three-dimensional motion state of the target image generated by stereo vision is fused with the three-dimensional point cloud of the laser radar to obtain comprehensive three-dimensional feature points of the target. 3. Based on the target's three-dimensional feature points, the target is tracked using a Kalman filter to analyze its motion state.

[0006] This patent requires the collection and annotation of a large dataset to train the network. However, we used a public dataset for pre-training, followed by fine-tuning using a smaller, self-built dataset, effectively reducing costs. Furthermore, this pre-trained model has strong generalization performance and can be fine-tuned to a variety of similar object recognition tasks. We used a transformer-based feature fusion network to interact with image and point cloud features, and its attention mechanism enabled the extraction of more refined features. Summary of the Invention

[0007] The present invention aims to solve the above problems of the prior art. It proposes a three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar. The technical solution of the present invention is as follows:

[0008] A three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar comprises the following steps:

[0009] Step S1: Calibrate the binocular camera to obtain the camera's internal and external parameters, and perform distortion correction and stereo correction on the image; jointly calibrate the camera and radar to obtain the conversion relationship between the camera and radar coordinate systems, thereby associating and aligning the image and point cloud data; use the binocular camera and lidar to collect data on steel strips in different shapes, postures, and environments, and construct a self-built dataset for steel strips;

[0010] Step S2: construct a transformer-based multimodal 3D object detection network;

[0011] Step S3: Use a multimodal alignment strategy to align the input image and point cloud data, use the image and point cloud feature extraction networks to extract features from the processed data respectively, concatenate the extracted features with the aligned data, and use the transformer encoder architecture to encode the concatenated data to extract more refined features;

[0012] In step S4, the encoded features and pre-trained object query are input into the transformer decoder, and three-dimensional object detection is performed using several multi-layer perception modules (MLP).

[0013] In step S5, transfer learning is performed based on the pre-trained model. The encoder weight parameters of the feature extractor and transformer in the pre-trained model are frozen, and the decoder network is fine-tuned using a self-built dataset. The trained network is quantized and accelerated, and deployed on a detection system consisting of a binocular camera and a lidar. Point cloud and image data are acquired in real time and input into the network to obtain the position, posture, and shape parameters of the scrap steel strips.

[0014] Furthermore, the step S1 includes the following steps:

[0015] A binocular camera is used to capture the calibration plate and collect image data. The Zhang Zhengyou calibration method is used to determine the camera's internal and external parameters and distortion coefficients. Distortion correction is first performed to eliminate image distortion, and then the binocular image is stereo corrected. The camera and lidar are used to synchronously collect images and 3D point cloud data of the calibration plate. The image and point cloud are jointly calibrated using the calibration plate to obtain a point cloud-to-image conversion matrix. The binocular camera and lidar are used to collect data for different materials, different stacking methods, different colors, and different stains and occlusions under different weather conditions, different ground conditions, and different backgrounds. The collected point cloud and image are associated according to the image-to-point cloud conversion matrix to obtain a self-built dataset.

[0016] Furthermore, the step S2 specifically includes the following steps:

[0017] A transformer-based multimodal 3D object detection network, including a feature extraction network composed of the VIT backbone (VisionTransformer feature extraction backbone network) and the SECOND backbone (a sparse convolution-based feature extraction network), a transformer-based feature fusion network, and a decoding output network. After the model is built, it is trained using a public dataset.

[0018] Furthermore, the step S3 specifically includes the following steps:

[0019] The image data and point cloud data are positionally encoded and embedded using a linear projection layer to obtain a high-dimensional vector. The linear projection layer is a fully connected layer in a neural network used to map input data to a specific dimensional space, implemented by linear transformation weight matrix multiplication. The multimodal alignment strategy uses the farthest point sampling (FPS) and K-nearest neighbor (KNN) algorithm to sample the point cloud data, process the point cloud into clusters, separate the image into N*N patches, and then project the center point of the sampled point cloud cluster onto the image through the 2D and 3D relationship between the image and the point cloud. The projection function is used for projection, and the formula is:

[0020]

[0021] Where K and R are the camera's internal and external matrices, (X, Y, Z) and (u, v) are the original three-dimensional coordinates and projected two-dimensional coordinates of the point cloud respectively;

[0022] Then, mask alignment is performed, and the image blocks corresponding to the sampled point cloud clusters are masked so that the areas corresponding to the visible point clouds on the blocks are invisible. A feature extractor for the image branch is constructed, and the unmasked image data is input into the VIT model skeleton for feature extraction. The image input is [H, W, C], and the output is [a0, 3*P*P], where H is the height of the image, W is the width, C is the number of channels, P is the patch size, and a0 is a constant related to the input image. A feature extractor for the point cloud branch is constructed, and the unmasked point cloud data is voxelized. The unmasked voxels are input into the SECOND model backbone for feature extraction. The output point cloud features have the same dimension as the image features. The extracted point cloud features and image features are spliced ​​with the projected masked point cloud and image, and then input into the transformer encoder for modal interaction to learn the features of both parties and obtain more refined features.

[0023] Furthermore, the step S4 specifically includes:

[0024] The target query is added to the refined features after modal interaction. Its dimension is consistent with the encoder output. It is used to interact with the input high-dimensional vector in the decoder and input it into the transformer decoder. The query vector output by the decoder is input into the MLP network for classification and regression. The loss function is:

[0025] L total =λ1L cls +λ2L coord +λ3L size +λ4L angle

[0026] Among them, λ1, λ2, λ3, and λ4 are the weight coefficients of the loss function, and L cls is the classification loss function, Lcoord is the coordinate loss function, L size is the target size loss function, L angle is the angle loss function, L cls The formula is:

[0027]

[0028] Where N is the batch size, n is the number of targets, is the classification result of the actual target, is the predicted target classification result, L coord The formula is:

[0029]

[0030] in is the center point coordinate of the predicted j-th target coordinate of the i-th batch, is the actual center point coordinate, L size The formula is:

[0031]

[0032] in is the size value of the predicted j-th target in the i-th batch, is the actual size value, L angle The formula is:

[0033]

[0034] in include is the rotation angle deviation around the x-axis, is the rotation angle deviation around the y-axis, is the rotation angle deviation around the z-axis; the deviation calculation formula is:

[0035]

[0036] Furthermore, the specific steps of step S5 are:

[0037] The trained model is deployed on a three-dimensional inspection system for steel strips consisting of a binocular camera and a lidar. It can acquire point cloud and image data in real time and input them into the network, enabling it to estimate category probabilities and position, posture, and shape parameters, including its seven parameters (x, y, z, d, l, α, β, γ), where (x, y, z) are the coordinates of the center point, (d, l) are the diameter and length, α is the rotation angle around the x-axis, β is the rotation angle around the y-axis, and γ is the rotation angle around the z-axis, to achieve unmanned grasping of the steel grabber.

[0038] An electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar as described in any one of the above items is implemented.

[0039] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar as described in any one of the above.

[0040] The advantages and beneficial effects of the present invention are as follows:

[0041] 1. This invention adopts the fusion method of binocular camera and laser radar, and adopts the strategy of modal alignment, which fully utilizes the advantages of image information density and laser point cloud distance accuracy to improve the accuracy of target recognition and the precision of posture estimation.

[0042] 2. The transformer-based feature fusion network interacts with image features and point cloud features, and the attention mechanism can extract more refined features.

[0043] 2. Pre-training on a public dataset was used, followed by fine-tuning on a smaller, self-built dataset, effectively reducing costs. Furthermore, this pre-trained model exhibits strong generalization performance and can be fine-tuned to a variety of similar target recognition tasks. A unique dataset for strip scrap steel recognition was constructed, featuring refined data samples, comprehensive annotations, and multi-task applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention provides a preferred embodiment of a flow chart of a three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar;

[0045] Figure 2 Schematic diagram of the data acquisition device;

[0046] Figure 3 Schematic diagram of the transformer-based multimodal 3D detection network model structure. DETAILED DESCRIPTION

[0047] The following will describe the technical solutions in the embodiments of the present invention in detail with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention.

[0048] The technical solution of the present invention to solve the above technical problems is:

[0049] Step S1: Install a binocular camera and a lidar module, calibrate the binocular camera, obtain the camera's internal and external parameters, and perform distortion correction and stereo correction on the image; jointly calibrate the camera and lidar to obtain the conversion relationship between the camera and lidar coordinate systems, thereby associating and aligning the image and point cloud data; use the binocular camera and lidar to collect data on steel strips in different shapes, postures, and environments, and construct a unique self-built dataset for steel strips;

[0050] Step S2: Build a transformer-based multimodal 3D object detection network and pre-train it using a public dataset.

[0051] Step S3: Use a multimodal alignment strategy to align the input image and point cloud data, use the image and point cloud feature extraction networks to extract features from the processed data respectively, concatenate the extracted features with the aligned data, and use the transformer encoder architecture to encode the concatenated data to extract more refined features;

[0052] Step S4: Input the encoded features and pre-trained object query into the transformer decoder and use MLP to perform 3D object detection respectively;

[0053] Step S5: Perform transfer learning based on the pre-trained model, freeze the encoder weight parameters of the feature extractor and transformer in the pre-trained model, and fine-tune the decoder network using a self-built dataset; quantize and accelerate the trained network, deploy it on a detection system consisting of a binocular camera and lidar, acquire point cloud and image data in real time and input them into the network to obtain the position, posture, and shape parameters of the scrap steel strips.

[0054] A flow chart of a three-dimensional detection method for strip scrap steel based on binocular camera and laser radar is shown in the figure. Figure 1 shown.

[0055] As a possible implementation of this embodiment, the data acquisition device in step S1 is as follows: Figure 2 As shown, the specific steps include:

[0056] A binocular camera was used to capture the calibration plate and collect image data. The Zhang Zhengyou calibration method was used to determine the camera's intrinsic and extrinsic parameters and distortion coefficients. Distortion correction was first performed to eliminate image distortion, and then the binocular image was stereo-rectified. Images and 3D point cloud data of the calibration plate were simultaneously acquired using a camera and lidar. The images and point cloud were jointly calibrated using the calibration plate to obtain a point cloud-to-image conversion matrix. Data was collected using the binocular camera and lidar in different weather conditions (sunny, cloudy, rainy, and night), on different floor surfaces (wooden, concrete, dirt, sand, and other materials, sloped surfaces, stairs, and flat ground), and in different backgrounds (indoor and outdoor scenes), including different materials, stacking methods, colors, and stain occlusion. The collected point cloud and image were correlated using the image-to-point cloud conversion matrix to obtain a self-built dataset.

[0057] As a possible implementation of this embodiment, the schematic diagram of the transformer-based multimodal 3D detection network model structure of step S2 is as follows: Figure 3 As shown, the specific steps include:

[0058] A transformer-based multimodal 3D object detection network, including a feature extraction network composed of a VIT backbone and a SECOND backbone, a transformer-based feature fusion network, and a decoding output network. After the model is built, it is trained using a public dataset.

[0059] As a possible implementation of this embodiment, step S3 specifically includes the following steps:

[0060] The image data and point cloud data are positionally encoded and embedded using a linear projection layer to obtain a high-dimensional vector. The linear projection layer is a fully connected layer in a neural network used to map input data to a specific dimensional space. It is implemented by linear transformation (weight matrix multiplication). The multimodal alignment strategy samples the point cloud data using the Far Point Sampling (FPS) and K-Nearest Neighbor (KNN) algorithms, processes the point cloud into clusters, separates the image into N*N patches, and then projects the center points of the sampled point cloud clusters onto the image using the 2D and 3D relationships between the image and the point cloud. The projection function is used for projection, as shown in the formula:

[0061]

[0062] Where K and R are the camera's internal and external matrices, (X, Y, Z) and (u, v) are the original three-dimensional coordinates and projected two-dimensional coordinates of the point cloud, respectively.

[0063] Mask alignment is then performed, masking the image patches corresponding to the sampled point cloud clusters (the remaining clusters are masked), rendering the areas on the patches corresponding to the visible point cloud invisible. A feature extractor for the image branch is constructed, and the unmasked image data is fed into the VIT model backbone for feature extraction. The image input is [H, W, C], and the output is [a0, 3*P*P], where H is the image height, W is the width, C is the number of channels, P is the patch size, and a0 is a constant dependent on the input image. A feature extractor for the point cloud branch is constructed, voxelizing the unmasked point cloud data. The unmasked voxels are fed into the SECOND model backbone for feature extraction, and the output point cloud features have the same dimensionality as the image features. The extracted point cloud and image features are concatenated with the projected masked point cloud and image, and then fed into the transformer encoder for modal interaction, learning features from both sides to obtain more refined features.

[0064] As a possible implementation of this embodiment, step S4 specifically includes:

[0065] The target query is added to the refined features after modal interaction. Its dimension is consistent with the encoder output. It is used to interact with the input high-dimensional vector in the decoder and input it into the transformer decoder. The query vector output by the decoder is input into the MLP network for classification and regression. The loss function is:

[0066] L total =λ1L cls +λ2L coord +λ3L size +λ4L angle

[0067] Among them, λ1, λ2, λ3, and λ4 are the weight coefficients of the loss function, and L cls is the classification loss function, L coord is the coordinate loss function, L size is the target size loss function, L angle is the angle loss function, L cls The formula is:

[0068]

[0069] Where N is the batch size, n is the number of targets, is the classification result of the actual target, is the predicted target classification result. L coord The formula is:

[0070]

[0071] in is the center point coordinate of the predicted j-th target coordinate of the i-th batch, is the actual center point coordinate. size The formula is:

[0072]

[0073] in is the size value of the predicted j-th target in the i-th batch, is the actual size value. angle The formula is:

[0074]

[0075] in include is the rotation angle deviation around the x-axis, is the rotation angle deviation around the y-axis, is the rotation angle deviation around the z-axis. The deviation calculation formula is:

[0076]

[0077] As a possible implementation of this embodiment, the specific steps of step S5 are:

[0078] The trained model is deployed on a three-dimensional inspection system for steel strips consisting of a binocular camera and a lidar. It can acquire point cloud and image data in real time and input them into the network, enabling it to estimate category probabilities and position, posture, and shape parameters, including its seven parameters (x, y, z, d, l, α, β, γ), where (x, y, z) are the coordinates of the center point, (d, l) are the diameter and length, α is the rotation angle around the x-axis, β is the rotation angle around the y-axis, and γ is the rotation angle around the z-axis, to achieve unmanned grasping of the steel grabber.

[0079] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions.

[0080] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0081] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0082] The above embodiments should be understood as merely illustrating the present invention and not as limiting the scope of protection of the present invention. After reading the contents of the present invention, technicians may make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A three-dimensional detection method for strip scrap steel based on binocular camera and laser radar, characterized in that: The following steps are involved: Step S1: Calibrate the binocular camera to obtain the camera's internal and external parameters, and perform distortion correction and stereo correction on the image; jointly calibrate the camera and radar to obtain the conversion relationship between the camera and radar coordinate systems, thereby associating and aligning the image and point cloud data; use the binocular camera and lidar to collect data on steel strips in different shapes, postures, and environments, and construct a self-built dataset for steel strips; Step S2: construct a transformer-based multimodal 3D object detection network; Step S3: Use a multimodal alignment strategy to align the input image and point cloud data, use the image and point cloud feature extraction networks to extract features from the processed data respectively, concatenate the extracted features with the aligned data, and use the transformer encoder architecture to encode the concatenated data to extract more refined features; Step S4: Input the encoded features and pre-trained target query into the transformer decoder, and use the multi-layer perception and several modules MLP to perform 3D target detection respectively; In step S5, transfer learning is performed based on the pre-trained model. The encoder weight parameters of the feature extractor and transformer in the pre-trained model are frozen, and the decoder network is fine-tuned using a self-built dataset. The trained network is quantized and accelerated, and deployed on a detection system consisting of a binocular camera and a lidar. Point cloud and image data are acquired in real time and input into the network to obtain the position, posture, and shape parameters of the scrap steel strips.

2. The three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar according to claim 1, characterized in that: The step S1 comprises the following steps: A binocular camera is used to capture the calibration plate and collect image data. The Zhang Zhengyou calibration method is used to determine the camera's internal and external parameters and distortion coefficients. Distortion correction is first performed to eliminate image distortion, and then the binocular image is stereo corrected. The camera and lidar are used to synchronously collect images and 3D point cloud data of the calibration plate. The image and point cloud are jointly calibrated using the calibration plate to obtain a point cloud-to-image conversion matrix. The binocular camera and lidar are used to collect data for different materials, different stacking methods, different colors, and different stains and occlusions under different weather conditions, different ground conditions, and different backgrounds. The collected point cloud and image are associated according to the image-to-point cloud conversion matrix to obtain a self-built dataset.

3. The three-dimensional detection method for strip scrap steel based on binocular camera and laser radar according to claim 1, characterized in that: The step S2 specifically includes the following steps: A transformer-based multimodal 3D object detection network, including a feature extraction network consisting of a VIT backbone feature extraction network and a sparse convolution-based feature extraction network SECOND, a transformer-based feature fusion network, and a decoding output network. After the model is built, it is trained using a public dataset.

4. The three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar according to claim 1, characterized in that: The step S3 specifically includes the following steps: The image data and point cloud data are positionally encoded and embedded using a linear projection layer to obtain a high-dimensional vector. The linear projection layer is a fully connected layer in a neural network used to map input data to a specific dimensional space, implemented by linear transformation weight matrix multiplication. The multimodal alignment strategy uses the farthest point sampling (FPS) and K-nearest neighbor (KNN) algorithm to sample the point cloud data, process the point cloud into clusters, separate the image into N*N patches, and then project the center point of the sampled point cloud cluster onto the image through the 2D and 3D relationship between the image and the point cloud. The projection function is used for projection, and the formula is: Where K and R are the camera's internal and external matrices, (X, Y, Z) and (u, v) are the original three-dimensional coordinates and projected two-dimensional coordinates of the point cloud respectively; Then, mask alignment is performed, and the image blocks corresponding to the sampled point cloud clusters are masked so that the areas corresponding to the visible point clouds on the blocks are invisible. A feature extractor for the image branch is constructed, and the unmasked image data is input into the VIT model skeleton for feature extraction. The image input is [H, W, C], and the output is [a0, 3*P*P], where H is the height of the image, W is the width, C is the number of channels, P is the patch size, and a0 is a constant related to the input image. A feature extractor for the point cloud branch is constructed, and the unmasked point cloud data is voxelized. The unmasked voxels are input into the SECOND model backbone for feature extraction. The output point cloud features have the same dimension as the image features. The extracted point cloud features and image features are spliced ​​with the projected masked point cloud and image, and then input into the transformer encoder for modal interaction to learn the features of both parties and obtain more refined features.

5. The three-dimensional detection method for strip scrap steel based on binocular camera and laser radar according to claim 1, characterized in that: The step S4 specifically includes: The target query is added to the refined features after modal interaction. Its dimension is consistent with the encoder output. It is used to interact with the input high-dimensional vector in the decoder and input it into the transformer decoder. The query vector output by the decoder is input into the MLP network for classification and regression. The loss function is: L total =λ1L cls +λ2L coord +λ3L size +λ4L angle Among them, λ1, λ2, λ3, and λ4 are the weight coefficients of the loss function, and L cls is the classification loss function, L coord is the coordinate loss function, L size is the target size loss function, L angle is the angle loss function, L cls The formula is: Where N is the batch size, n is the number of targets, is the classification result of the actual target, is the predicted target classification result, L coord The formula is: in is the center point coordinate of the predicted j-th target coordinate of the i-th batch, is the actual center point coordinate, L size The formula is: in is the size value of the predicted j-th target in the i-th batch, is the actual size value, L angle The formula is: in include is the rotation angle deviation around the x-axis, is the rotation angle deviation around the y-axis, is the rotation angle deviation around the z-axis; the deviation calculation formula is:

6. The three-dimensional detection method for strip scrap steel based on binocular camera and laser radar according to claim 1, characterized in that: The specific steps of step S5 are: The trained model is deployed on a three-dimensional inspection system for steel strips consisting of a binocular camera and a lidar. It can acquire point cloud and image data in real time and input them into the network, enabling it to estimate category probabilities and position, posture, and shape parameters, including its seven parameters (x, y, z, d, l, α, β, γ), where (x, y, z) are the coordinates of the center point, (d, l) are the diameter and length, α is the rotation angle around the x-axis, β is the rotation angle around the y-axis, and γ is the rotation angle around the z-axis, to achieve unmanned grasping of the steel grabber.

7. An electronic device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar is implemented as claimed in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional detection method for strip scrap steel based on a binocular camera and a laser radar is implemented as described in any one of claims 1 to 6.

Images