Automatic rubber block grabbing method, device, equipment and storage medium

Abstract

The application discloses an automatic rubber block grabbing method, device, equipment and storage medium, target area image and point cloud data of a target area are acquired in real time; the target area image is input into a pre-trained instance segmentation model, so that the instance segmentation model outputs rubber block image data, and a target area global depth image is acquired based on the point cloud data; after it is determined that no first rubber block exists in the target area based on the rubber block image data, whether the target area is a null frame area is judged according to the target area global depth image; after it is determined that the target area is not a null frame area, the rubber block image data and the target area global depth image are analyzed to obtain the spatial position of a target rubber block in the target area; the target rubber block is grabbed by a mechanical arm according to the spatial position; compared with the prior art, the technical scheme of the application can improve the feeding efficiency of the rubber block and reduce the production cost.

Description

Technical Field

[0001] This invention relates to the field of image recognition processing technology, and in particular to an automatic method, apparatus, device, and storage medium for grasping rubber blocks. Background Technology

[0002] Rubber blocks are a common material in industrial production, mainly used to make rubber products such as automobile tires. In the automobile tire manufacturing industry, the width of commonly used rubber block raw materials is generally 300-500mm, the length is generally 400-700mm, and the weight is about 35kg.

[0003] Currently, the loading of rubber blocks into the next process still relies on manual handling. Obviously, the traditional manual method of loading rubber blocks is inefficient, which is not conducive to manufacturers improving their competitiveness and increases production costs. With the development of artificial intelligence technology, improving the automation level of rubber block loading production lines has become a very urgent need. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an automatic gripping method, device, equipment and storage medium for rubber blocks, so as to improve the feeding efficiency of rubber blocks and reduce production costs.

[0005] To address the aforementioned technical problems, this invention provides an automatic gripping method for rubber blocks, comprising:

[0006] Real-time acquisition of target area images and point cloud data;

[0007] The target region image is input into a pre-trained instance segmentation model so that the instance segmentation model outputs rubber block image data, and a global depth image of the target region is obtained based on the point cloud data;

[0008] Based on the rubber block image data, after determining that there is no first rubber block in the target area, it is determined whether the target area is an empty frame area based on the global depth image of the target area;

[0009] After determining that the target area is not an empty frame area, the image data of the rubber block and the global depth image of the target area are analyzed to obtain the spatial position of the target rubber block in the target area;

[0010] Based on the spatial position, the robotic arm is controlled to grasp the target rubber block.

[0011] The present invention provides an automatic gripping method for rubber blocks, which further includes:

[0012] Based on the rubber block image data, after detecting the presence of a first rubber block in the rubber block image data, it is determined whether there is a rubber block in the rubber block image data whose category is NG;

[0013] When it is determined that there is no rubber block of the NG category in the rubber block image data, the rubber block image data and the global depth image of the target area are analyzed to obtain the spatial location of the target rubber block in the target area;

[0014] When it is determined that there is a rubber block of the NG category in the rubber block image data, the presence of an NG category rubber block is confirmed, and an alarm prompt is output.

[0015] In one possible implementation, after determining whether the target region is an empty frame region, the method further includes:

[0016] After determining that the target area is an empty frame area, confirm that there are no grabbable rubber blocks in the target area, and output a prompt for manual material replacement.

[0017] In one possible implementation, pre-training the instance segmentation model specifically includes:

[0018] Collect images of the rubber block at different exposure values;

[0019] Rubber block identification is performed on each exposed rubber block image to obtain a rubber block mask image corresponding to each rubber block in the exposed rubber block image;

[0020] All rubber blocks are categorized so that rubber blocks with tilt angles exceeding a preset threshold are categorized as NG; otherwise, rubber blocks with tilt angles not exceeding the preset threshold are categorized as OK.

[0021] Based on the exposed rubber block image, the rubber block mask image, and the rubber block category, an initial training dataset is generated, and the initial training dataset is augmented to obtain an augmented training dataset.

[0022] The initial neural network model is trained based on the expanded training dataset until the model converges, resulting in an instance segmentation model.

[0023] In one possible implementation, based on the point cloud data, a global depth image of the target region is obtained, specifically including:

[0024] Based on the mapping relationship between the rubber block point cloud and the depth map, the point cloud data is mapped to the depth map to obtain a global depth image of the target area. The mapping relationship is as follows:

[0025]

[0026] In the formula, u and v are arbitrary coordinate points on the image, u0 and v0 are the center coordinates of the image, and x... w y w z w The z-axis represents the three-dimensional coordinates of a point cloud. c The z-axis value represents the camera coordinates, i.e., the distance from the target to the camera. dx, dy, and f are built-in parameters of the camera.

[0027] In one possible implementation, the image data of the rubber block and the global depth image of the target region are analyzed to obtain the spatial location of the target rubber block in the target region, specifically including:

[0028] The rubber block image data includes a first rubber block mask image, a first rubber block category, and a first confidence level;

[0029] The first rubber block mask image is aligned and integrated with the global depth image of the target region to obtain the spatial position of the rubber block in the target region;

[0030] Based on the first confidence level, the rubber block with the highest first confidence level is selected from the highest layer of the spatial location of the rubber block as the target rubber block, and the spatial location of the target rubber block in the target area is obtained.

[0031] The present invention also provides an automatic rubber block gripping device, comprising: a target area data acquisition module, a data processing module, a first target area judgment module, a target rubber block position determination module, and a target rubber block gripping module;

[0032] The target area data acquisition module is used to acquire the target area image and point cloud data of the target area in real time.

[0033] The data processing module is used to input the target region image into a pre-trained instance segmentation model so that the instance segmentation model outputs rubber block image data and obtains a global depth image of the target region based on the point cloud data.

[0034] The first target region determination module is used to determine whether the target region is an empty frame region based on the target region global depth image of the target region after determining that there is no first rubber block in the target region based on the rubber block image data.

[0035] The target rubber block location determination module is used to analyze the rubber block image data and the global depth image of the target area after determining that the target area is not an empty frame area, so as to obtain the spatial location of the target rubber block in the target area.

[0036] The target rubber block grasping module is used to control the robotic arm to grasp the target rubber block according to the spatial position.

[0037] The present invention provides an automatic rubber block gripping device, which further includes: a second target area judgment module;

[0038] The second target region determination module is used to determine whether there is a rubber block of the NG category in the rubber block image data after detecting the presence of a first rubber block in the rubber block image data; when it is determined that there is no rubber block of the NG category in the rubber block image data, the module analyzes the rubber block image data and the global depth image of the target region to obtain the spatial location of the target rubber block in the target region; when it is determined that there is a rubber block of the NG category in the rubber block image data, the module confirms the presence of an NG category rubber block and outputs an alarm prompt.

[0039] In one possible implementation, the first target region determination module, after determining whether the target region is an empty frame region, further includes:

[0040] After determining that the target area is an empty frame area, confirm that there are no grabbable rubber blocks in the target area, and output a prompt for manual material replacement.

[0041] In one possible implementation, the data processing module, when pre-training the instance segmentation model, specifically includes:

[0042] Collect images of the rubber block at different exposure values;

[0043] Rubber block identification is performed on each exposed rubber block image to obtain a rubber block mask image corresponding to each rubber block in the exposed rubber block image;

[0044] All rubber blocks are categorized so that rubber blocks with tilt angles exceeding a preset threshold are categorized as NG; otherwise, rubber blocks with tilt angles not exceeding the preset threshold are categorized as OK.

[0045] Based on the exposed rubber block image, the rubber block mask image, and the rubber block category, an initial training dataset is generated, and the initial training dataset is augmented to obtain an augmented training dataset.

[0046] The initial neural network model is trained based on the expanded training dataset until the model converges, resulting in an instance segmentation model.

[0047] In one possible implementation, the data processing module is used to acquire a global depth image of the target area based on the point cloud data, specifically including:

[0048] Based on the mapping relationship between the rubber block point cloud and the depth map, the point cloud data is mapped to the depth map to obtain a global depth image of the target area. The mapping relationship is as follows:

[0049]

[0050] In the formula, u and v are arbitrary coordinate points on the image, u0 and v0 are the center coordinates of the image, and x... w y w z w The z-axis represents the three-dimensional coordinates of a point cloud. c The z-axis value represents the camera coordinates, i.e., the distance from the target to the camera. dx, dy, and f are built-in parameters of the camera.

[0051] In one possible implementation, the target rubber block location determination module is used to analyze the rubber block image data and the global depth image of the target region to obtain the spatial location of the target rubber block in the target region, specifically including:

[0052] The rubber block image data includes a first rubber block mask image, a first rubber block category, and a first confidence level;

[0053] The first rubber block mask image is aligned and integrated with the global depth image of the target region to obtain the spatial position of the rubber block in the target region;

[0054] Based on the first confidence level, the rubber block with the highest first confidence level is selected from the highest layer of the spatial location of the rubber block as the target rubber block, and the spatial location of the target rubber block in the target area is obtained.

[0055] The present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the automatic gripping method for rubber blocks as described in any of the preceding claims.

[0056] The present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the automatic gripping method for rubber blocks as described in any of the preceding claims.

[0057] The present invention provides an automatic gripping method, apparatus, device, and storage medium for rubber blocks, which, compared with the prior art, has the following advantages:

[0058] By acquiring target region images and point cloud data in real time, the target region images are input into a pre-trained instance segmentation model, which outputs rubber block image data. Based on the point cloud data, a global depth image of the target region is obtained. After determining that the first rubber block does not exist in the target region based on the rubber block image data, the global depth image of the target region is used to determine whether the target region is an empty frame region. After determining that the target region is not an empty frame region, the rubber block image data and the global depth image of the target region are analyzed to obtain the spatial position of the target rubber block in the target region. Based on the spatial position, a robotic arm is controlled to grasp the target rubber block. Compared with the prior art, the technical solution of this invention combines instance segmentation models and point cloud data to obtain the spatial position of the target rubber block in the target region, and then realizes automatic grasping and feeding of the rubber block based on the robotic arm, which improves the efficiency of rubber block grasping and feeding and reduces the dependence on manual labor. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating an embodiment of an automatic rubber block gripping method provided by the present invention;

[0060] Figure 2 This is a schematic diagram of an embodiment of an automatic rubber block gripping device provided by the present invention;

[0061] Figure 3 This is a schematic diagram of the exposed rubber block images corresponding to five different exposure values ​​according to an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the tilted state of a rubber block according to an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of an exposed rubber block image based on data rotation and mirroring augmentation, according to an embodiment of the present invention.

[0064] Figure 6 This is a schematic diagram of an exposed rubber block image based on data fusion augmentation, according to an embodiment of the present invention.

[0065] Figure 7 This is a schematic diagram of the network structure of PointRend, an instance segmentation model provided by an embodiment of the present invention;

[0066] Figure 8 This is a schematic diagram of the output result of an instance segmentation model according to an embodiment of the present invention;

[0067] Figure 9 This is a schematic diagram of the rubber block grasping algorithm of one embodiment of the present invention. Detailed Implementation

[0068] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1

[0070] See Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the automatic rubber block gripping method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 101-105, as detailed below:

[0071] Step 101: Acquire the target area image and point cloud data in real time.

[0072] In one embodiment, a 3D camera and a robotic arm are set up in the target area. Preferably, the camera is fixed directly above the target area, which is the material frame area for placing rubber blocks.

[0073] In one embodiment, after detecting that the robotic arm and camera are in place, the system receives and collects image point cloud data signals in real time, and triggers the 3D camera to collect target area images and point cloud data of the target area after receiving the image point cloud data signals.

[0074] Step 102: Input the target region image into the pre-trained instance segmentation model so that the instance segmentation model outputs rubber block image data, and obtains the global depth image of the target region based on the point cloud data.

[0075] In one embodiment, when pre-training the instance segmentation model, it is necessary to collect images of rubber blocks with different exposure values.

[0076] Specifically, since the lighting environment in a factory is not constant, images of the target area taken under different lighting conditions may affect subsequent image processing, leading to a decrease in the model's segmentation accuracy. Therefore, to adapt the model to the changing lighting environment in the factory and enhance its generalization performance, images of the rubber block with various exposure values ​​were collected during data acquisition, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the rubber block images corresponding to five different exposure values.

[0077] In one embodiment, after obtaining the exposed rubber block images corresponding to different exposure values, it is also necessary to perform rubber block recognition on each exposed rubber block image to obtain the rubber block mask image corresponding to each rubber block in the exposed rubber block image.

[0078] In one embodiment, all rubber blocks are categorized so that rubber blocks with tilt angles exceeding a preset threshold are categorized as NG, and otherwise, rubber blocks with tilt angles not exceeding the preset threshold are categorized as OK.

[0079] Specifically, the preset threshold can be automatically defined according to user needs; preferably, the preset threshold is set to 20°. Based on the annotation tool, all collected rubber blocks are labeled with polygons. If the tilt angle of the rubber block exceeds 20°, it is labeled as NG category; otherwise, it is labeled as OK category.

[0080] Specifically, during the actual handling process of the robotic arm, the adhesive nature of the rubber causes it to pull up surrounding rubber blocks, resulting in the surrounding rubber blocks not lying flat but tilted, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the rubber block in a tilted state.

[0081] If the rubber block is tilted at too large an angle, the stability of the robotic arm in grasping the rubber block will be greatly reduced, which may easily lead to safety accidents. Therefore, when collecting data, all the rubber blocks are also labeled with rubber block categories. In the actual production process, once such a tilted rubber block appears, an alarm will be triggered and the machine will stop, allowing the staff to make adjustments and avoid safety accidents.

[0082] In one embodiment, an initial training dataset is generated based on the exposed rubber block image, the rubber block mask image, and the rubber block category.

[0083] In one embodiment, when generating the initial training dataset, in order to increase the generalization performance of the model, it is necessary to expand the data diversity in the initial training dataset, that is, to expand the initial training dataset to obtain an expanded training dataset.

[0084] In one embodiment, the initial training dataset is augmented, wherein the augmentation includes data rotation and mirroring augmentation and data fusion augmentation.

[0085] Specifically, when performing data rotation and mirroring augmentation on the initial training dataset, the exposed rubber block images and the rubber block categories in the initial training dataset are rotated by 180°, horizontally mirrored, and vertically mirrored, respectively, to obtain the first augmented training dataset, such as... Figure 5 As shown, Figure 5This is a schematic diagram of an exposed rubber block image based on data rotation and mirroring expansion; the first expanded training data volume is 4 times the initial data volume during acquisition.

[0086] Specifically, when performing data fusion and augmentation on the initial training dataset, a first exposed rubber block image of the rubber block category NG is obtained, and the first exposed rubber block image is fused into the exposed rubber block image of the initial training dataset to generate a second exposed rubber block image, thus obtaining a second augmented training dataset.

[0087] To improve the recognition accuracy of NG (Not Good) rubber blocks, NG data needs to be artificially generated during data collection. However, rubber blocks weigh up to 35kg, making it difficult to artificially generate a large number of NG data. Therefore, it is necessary to fuse the first-exposure rubber block image of the NG category into images of other exposed rubber blocks to expand the number of first-exposure rubber block images corresponding to the NG category in the dataset. Theoretically, this data fusion can generate an infinite number of NG samples, from which data that matches reality can be selected for model training. Figure 6 As shown, Figure 6 This is a schematic diagram of an exposed rubber block image based on data fusion augmentation.

[0088] In one embodiment, the generalization performance of the model can be increased by expanding the data diversity in the initial training dataset when generating the training dataset.

[0089] In one embodiment, the initial neural network model is trained based on the expanded training dataset until the model converges, thus obtaining an instance segmentation model.

[0090] Specifically, the PointRend instance segmentation model from the deep learning domain is used for model training, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of the network structure of the PointRend instance segmentation model. By inputting the acquired image, the model can output the position of the mask corresponding to the target.

[0091] Specifically, the exposed rubber block images in the expanded training dataset are used as input to the instance segmentation model, and the rubber block mask image, rubber block category, and confidence score of the classification as the rubber block category are used as the model output. The model is trained until it converges. After the instance segmentation model is trained, it can be deployed on industrial control computer hardware for actual production deployment.

[0092] In one embodiment, the target region image is input into a pre-trained instance segmentation model, so that the instance segmentation model outputs rubber block image data of the target region. The rubber block image data includes a first rubber block mask image, a first rubber block category, and a first confidence score. The first confidence score is the confidence score corresponding to the first rubber block category to which any first rubber block in the target region image is classified. Figure 8 As shown, Figure 8 This is a schematic diagram of the output results of the instance segmentation model. Figure 8 It can be clearly seen that in the results, one tilted rubber block is considered to be in the NG category, while the rest of the rubber blocks are considered to be in the OK category.

[0093] In one embodiment, a global depth image of the target area is obtained based on the point cloud data. Specifically, based on the mapping relationship between the rubber block point cloud and the depth map, the point cloud data is mapped onto the depth map to obtain a global depth image of the target area, such as... Figure 9 As shown, Figure 9 This is a schematic diagram of the global depth image.

[0094] In one embodiment, the mapping relationship is as follows:

[0095]

[0096] In the formula, u and v are arbitrary coordinate points on the image, u0 and v0 are the center coordinates of the image, and x... w y w z w The z-axis represents the three-dimensional coordinates of a point cloud. c The z-axis value represents the camera coordinates, i.e., the distance from the target to the camera. dx, dy, and f are built-in parameters of the camera.

[0097] Step 103: Based on the rubber block image data, after determining that there is no first rubber block in the target area, determine whether the target area is an empty frame area based on the global depth image of the target area.

[0098] In one embodiment, since the rubber block image data output by the instance segmentation model may result in missed detections, that is, a rubber block exists in the target area but is not detected by the instance segmentation model. Therefore, after determining that there is no first rubber block in the target area based on the rubber block image data, the global depth image of the target area is also combined to determine whether the target area is an empty frame area, so as to combine the model detection and depth information for verification and improve the accuracy of rubber block detection.

[0099] In one embodiment, when determining whether the target region is an empty frame region by combining the global depth image of the target region, effective depth information is detected by performing effective depth information detection on the global depth image of the target region. When effective depth information is detected, a manual re-inspection alarm notification is output; when no effective depth information is detected, the target region is determined to be an empty frame region.

[0100] In one embodiment, after determining that the target area is an empty frame area, it is confirmed that there are no grabbable rubber blocks in the target area, and a manual material replacement prompt is output.

[0101] Specifically, if the instance segmentation model does not detect the rubber block, and combined with the global depth image of the target area, if no valid depth information is detected in the depth direction, it means that the material box is empty, and the manual replacement is notified. If valid depth information is detected in the depth direction, it means that the instance segmentation model did not detect the rubber block, and an alarm is triggered to notify manual intervention. If the manual confirms that there is indeed a missed detection, the manual can select the rubber block mask area for further processing.

[0102] In one embodiment, the method further includes, based on the rubber block image data, after detecting the presence of a first rubber block in the rubber block image data, determining whether there is a rubber block of the NG category in the rubber block image data; when it is determined that there is a rubber block of the NG category in the rubber block image data, then confirming the presence of an NG category rubber block and outputting an alarm prompt.

[0103] In one embodiment, when it is determined that there is no rubber block of the NG category in the rubber block image data, the rubber block image data and the global depth image of the target region are analyzed to obtain the spatial location of the target rubber block in the target region.

[0104] Step 104: After determining that the target area is not an empty frame area, analyze the rubber block image data and the global depth image of the target area to obtain the spatial position of the target rubber block in the target area.

[0105] In one embodiment, when analyzing the rubber block image data and the global depth image of the target region, the first rubber block mask image and the global depth image of the target region are aligned and integrated to obtain the spatial position of the rubber block in the target region; based on the first confidence level, the rubber block with the highest first confidence level is selected from the highest layer of the spatial position of the rubber block as the target rubber block to obtain the spatial position of the target rubber block in the target region.

[0106] Specifically, if the first rubber block detected by the instance segmentation model is an OK category first rubber block, the rubber block image data and the global depth image of the target area are aligned and integrated to obtain the spatial position of the rubber block, and the rubber block with the highest confidence in the highest layer is selected as the pre-grabbing target.

[0107] Step 105: Based on the spatial position, control the robotic arm to grasp the target rubber block.

[0108] In one embodiment, since the graspable rubber blocks are all labeled based on their category to achieve tilt angle filtering, and the robotic arm is positioned directly above the target area, after obtaining the spatial position, the robotic arm is controlled to vertically downwards to grasp the target rubber block. After the robotic arm grasps the target rubber block, the processing of the target rubber block ends, and the next round of automatic rubber block grasping process begins, such as... Figure 9 As shown, Figure 9 This is a schematic diagram of the rubber block grasping algorithm.

[0109] In summary, the automatic rubber block grasping method provided by this invention combines 2D image processing with 3D vision technology, and combines instance segmentation model and point cloud data to obtain the spatial position of the target rubber block in the target area. Then, based on the robotic arm, the automatic grasping and feeding of the rubber block is realized, which improves the efficiency of rubber block grasping and feeding and reduces the dependence on manual labor.

[0110] Example 2

[0111] See Figure 2 , Figure 2 This is a schematic diagram of one embodiment of the automatic rubber block gripping device provided by the present invention, as shown below. Figure 2 As shown, the device includes a target area data acquisition module 201, a data processing module 202, a first target area judgment module 203, a target rubber block position determination module 204, and a target rubber block grasping module 205, as detailed below:

[0112] The target area data acquisition module 201 is used to acquire the target area image and point cloud data of the target area in real time.

[0113] The data processing module 202 is used to input the target region image into a pre-trained instance segmentation model so that the instance segmentation model outputs rubber block image data and obtains a global depth image of the target region based on the point cloud data.

[0114] The first target region determination module 203 is used to determine whether the target region is an empty frame region based on the target region global depth image of the target region after determining that there is no first rubber block in the target region based on the rubber block image data.

[0115] The target rubber block location determination module 204 is used to analyze the rubber block image data and the global depth image of the target area after determining that the target area is not an empty frame area, so as to obtain the spatial location of the target rubber block in the target area.

[0116] The target rubber block gripping module 205 is used to control the robotic arm to grip the target rubber block according to the spatial position.

[0117] The automatic gripping device for rubber blocks provided in this embodiment also includes a second target area judgment module.

[0118] In one embodiment, the second target region determination module is used to determine whether there is a rubber block of the NG category in the rubber block image data after detecting the presence of a first rubber block in the rubber block image data; when it is determined that there is no rubber block of the NG category in the rubber block image data, the module analyzes the rubber block image data and the global depth image of the target region to obtain the spatial location of the target rubber block in the target region; when it is determined that there is a rubber block of the NG category in the rubber block image data, the module confirms the presence of an NG category rubber block and outputs an alarm prompt.

[0119] In one embodiment, the first target area determination module 203, after determining whether the target area is an empty frame area, further includes: after determining that the target area is an empty frame area, confirming that there are no grabbable rubber blocks in the target area, and outputting a manual material replacement prompt.

[0120] In one embodiment, the data processing module 202, when pre-training the instance segmentation model, specifically includes: acquiring exposed rubber block images corresponding to different exposure values; performing rubber block recognition on each exposed rubber block image to obtain a rubber block mask image corresponding to each rubber block in the exposed rubber block image; labeling all rubber blocks with rubber block categories, such that rubber blocks with tilt angles exceeding a preset threshold are labeled as NG category, otherwise, rubber blocks with tilt angles not exceeding the preset threshold are labeled as OK category; generating an initial training dataset based on the exposed rubber block images, the rubber block mask images, and the rubber block categories, and augmenting the initial training dataset to obtain an augmented training dataset; training the initial neural network model based on the augmented training dataset until the model converges to obtain the instance segmentation model.

[0121] In one embodiment, the data processing module 202 is used to obtain a global depth image of the target area based on the point cloud data, specifically including: mapping the point cloud data to the depth map based on the mapping relationship between the rubber block point cloud and the depth map to obtain a global depth image of the target area, wherein the mapping relationship is as follows:

[0122]

[0123] In the formula, u and v are arbitrary coordinate points on the image, u0 and v0 are the center coordinates of the image, and x... w y w z w The z-axis represents the three-dimensional coordinates of a point cloud. c The z-axis value represents the camera coordinates, i.e., the distance from the target to the camera. dx, dy, and f are built-in parameters of the camera.

[0124] In one embodiment, the target rubber block location determination module 204 is used to analyze the rubber block image data and the global depth image of the target region to obtain the spatial location of the target rubber block in the target region. Specifically, the rubber block image data includes a first rubber block mask image, a first rubber block category, and a first confidence level; the first rubber block mask image is aligned and integrated with the global depth image of the target region to obtain the spatial location of the rubber block in the target region; based on the first confidence level, the rubber block with the highest first confidence level is selected from the highest layer of the rubber block spatial location as the target rubber block to obtain the spatial location of the target rubber block in the target region.

[0125] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0126] It should be noted that the above-described embodiment of the automatic rubber block gripping device is merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Based on the above-described embodiments of the automatic rubber block gripping method, another embodiment of the present invention provides an automatic rubber block gripping terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the automatic rubber block gripping method of any embodiment of the present invention.

[0128] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the automatic gripping terminal device for the rubber block.

[0129] The automatic rubber block gripping terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The automatic rubber block gripping terminal device may include, but is not limited to, a processor and a memory.

[0130] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the automatic rubber block grasping terminal device, connecting all parts of the device via various interfaces and lines.

[0131] The memory can be used to store the computer programs and / or modules. The processor, by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, realizes various functions of the automatic rubber block grasping terminal device. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0132] Based on the above-described embodiments of the automatic rubber block gripping method, another embodiment of the present invention provides a storage medium, the storage medium including a stored computer program, wherein, when the computer program is running, the device where the storage medium is located controls the execution of the automatic rubber block gripping method of any embodiment of the present invention.

[0133] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0134] In summary, the present invention provides an automatic rubber block grasping method, apparatus, device, and storage medium that acquires target region images and point cloud data in real time; inputs the target region image into a pre-trained instance segmentation model so that the instance segmentation model outputs rubber block image data, and obtains a global depth image of the target region based on the point cloud data; after determining that there is no first rubber block in the target region based on the rubber block image data, it determines whether the target region is an empty frame region based on the global depth image of the target region; after determining that the target region is not an empty frame region, it analyzes the rubber block image data and the global depth image of the target region to obtain the spatial position of the target rubber block in the target region; and controls a robotic arm to grasp the target rubber block based on the spatial position. Compared with the prior art, the technical solution of the present invention can improve the rubber block loading efficiency and reduce production costs.

[0135] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. An automatic gripping method for rubber blocks, characterized in that, include: Real-time acquisition of target area images and point cloud data; The target region image is input into a pre-trained instance segmentation model so that the instance segmentation model outputs rubber block image data, and a global depth image of the target region is obtained based on the point cloud data; Based on the rubber block image data, after determining that there is no first rubber block in the target area, it is determined whether the target area is an empty frame area based on the global depth image of the target area; After determining that the target area is not an empty frame area, the image data of the rubber block and the global depth image of the target area are analyzed to obtain the spatial position of the target rubber block in the target area; Based on the spatial position, the robotic arm is controlled to grasp the target rubber block; Also includes: Based on the rubber block image data, after detecting the presence of a first rubber block in the rubber block image data, it is determined whether there is a rubber block in the rubber block image data whose category is NG; When it is determined that there is no rubber block of the NG category in the rubber block image data, the rubber block image data and the global depth image of the target area are analyzed to obtain the spatial location of the target rubber block in the target area; When it is determined that there is a rubber block of the NG category in the rubber block image data, the presence of an NG category rubber block is confirmed, and an alarm is output. Rubber blocks whose tilt angle exceeds a preset threshold are marked as NG; otherwise, rubber blocks whose tilt angle does not exceed the preset threshold are marked as OK.

2. The automatic gripping method for rubber blocks as described in claim 1, characterized in that, After determining whether the target area is an empty frame area, the method further includes: After determining that the target area is an empty frame area, confirm that there are no grabbable rubber blocks in the target area, and output a prompt for manual material replacement.

3. The automatic gripping method for rubber blocks as described in claim 1, characterized in that, Pre-training the instance segmentation model specifically includes: Collect images of the rubber block at different exposure values; Rubber block identification is performed on each exposed rubber block image to obtain a rubber block mask image corresponding to each rubber block in the exposed rubber block image; All rubber blocks are categorized so that rubber blocks whose tilt angle exceeds a preset threshold are categorized as NG; otherwise, rubber blocks whose tilt angle does not exceed the preset threshold are categorized as OK. Based on the exposed rubber block image, the rubber block mask image, and the rubber block category, an initial training dataset is generated, and the initial training dataset is augmented to obtain an augmented training dataset. The initial neural network model is trained based on the expanded training dataset until the model converges, resulting in an instance segmentation model.

4. The automatic gripping method for rubber blocks as described in claim 1, characterized in that, Based on the point cloud data, a global depth image of the target area is obtained, specifically including: Based on the mapping relationship between the rubber block point cloud and the depth map, the point cloud data is mapped to the depth map to obtain a global depth image of the target area. The mapping relationship is as follows: ； In the formula, , Let be any coordinate point on the image. , The center coordinates of the image. , , Represents the three-dimensional coordinates of a point cloud. The z-axis value represents the camera coordinates, i.e., the distance from the target to the camera. , , These are parameters that come with the camera.

5. The automatic gripping method for a rubber block as described in claim 1, characterized in that, Analyzing the image data of the rubber block and the global depth image of the target region yields the spatial location of the target rubber block within the target region, specifically including: The rubber block image data includes a first rubber block mask image, a first rubber block category, and a first confidence level; The first rubber block mask image is aligned and integrated with the global depth image of the target region to obtain the spatial position of the rubber block in the target region; Based on the first confidence level, the rubber block with the highest first confidence level is selected from the highest layer of the spatial location of the rubber block as the target rubber block, and the spatial location of the target rubber block in the target area is obtained.

6. An automatic gripping device for rubber blocks, characterized in that, include: The module includes a target area data acquisition module, a data processing module, a first target area judgment module, a target rubber block location determination module, and a target rubber block grasping module. The target area data acquisition module is used to acquire the target area image and point cloud data of the target area in real time. The data processing module is used to input the target region image into a pre-trained instance segmentation model so that the instance segmentation model outputs rubber block image data and obtains a global depth image of the target region based on the point cloud data. The first target region determination module is used to determine whether the target region is an empty frame region based on the target region global depth image of the target region after determining that there is no first rubber block in the target region based on the rubber block image data. The target rubber block location determination module is used to analyze the rubber block image data and the global depth image of the target area after determining that the target area is not an empty frame area, so as to obtain the spatial location of the target rubber block in the target area. The target rubber block grasping module is used to control the robotic arm to grasp the target rubber block according to the spatial position; It also includes: a second target area determination module; The second target region determination module is used to determine whether there is a rubber block of the NG category in the rubber block image data after detecting the presence of a first rubber block in the rubber block image data; when it is determined that there is no rubber block of the NG category in the rubber block image data, the module analyzes the rubber block image data and the global depth image of the target region to obtain the spatial location of the target rubber block in the target region; when it is determined that there is a rubber block of the NG category in the rubber block image data, the module confirms the presence of an NG category rubber block and outputs an alarm prompt. Rubber blocks whose tilt angle exceeds a preset threshold are marked as NG; otherwise, rubber blocks whose tilt angle does not exceed the preset threshold are marked as OK.

7. A terminal device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the automatic gripping method for rubber blocks as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the automatic gripping method for rubber blocks as described in any one of claims 1 to 5.

Images