A method and system for judging a grabbing state of a mechanical arm grabbing a small-size object

Abstract

A mechanical arm grasps small size object grasping state judgment method and system, the method comprises the following steps: S1: constructing a virtual data set to train a convolutional neural network; S2: judging whether the grasping target is located on the hand; S3: judging the distance between two fingers; S4: judging the grasping state of the mechanical arm. The application can still accurately judge the grasping state of the mechanical arm in the case of large changes in light, background and target by combining two kinds of judgment methods.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and in particular to a method and system for determining the grasping state of a robotic arm when grasping small objects. Background Technology

[0002] In practical applications, such as industrial production lines and logistics warehousing, robotic arms are needed to accurately grasp target objects for subsequent processing and transportation. However, in complex environments, the grasping state of a robotic arm is often affected by various factors, such as the shape, material, position, and posture of the target object. This necessitates using visual information to determine the grasping state of the robotic arm.

[0003] Research on vision-based robotic arm grasping state determination faces several challenges. First, the diverse shapes and materials of target objects lead to significant variations in generated images, impacting model performance. Second, factors such as the robotic arm's position and posture affect visual information, making grasping state determination even more difficult.

[0004] To address these issues, researchers have proposed a series of methods for determining the grasping state of robotic arms. For example,

[0005] Force sensor-based methods utilize force sensors on a robotic arm to detect the force information between the robotic arm and the object being grasped, thereby determining the grasping status. For example, by monitoring the magnitude and stability of the grasping force, it can be determined whether the object has been successfully grasped, and whether the object has slipped or been damaged. However, for practical applications, this method suffers from high sensor costs, increasing the overall system cost, and it cannot determine the specific state of the grasped object.

[0006] Traditional vision-based methods utilize computer vision technology to process and analyze object images to determine the object's pose and position. This method is often used to identify object edges, feature points, or perform template matching to determine the grasping position and direction. However, this method's accuracy may be affected by uneven lighting, complex object shapes, or unclear surface textures, leading to inaccurate grasping status assessments.

[0007] Deep learning-based methods utilize deep learning technology to train the robotic arm on a large dataset, enabling it to learn the features and states of objects it grasps. This method can extract more complex object features, thus achieving more accurate grasping state judgment. However, this method requires a large training dataset for model training and optimization, and collecting and labeling this data is a time-consuming and resource-intensive task. Furthermore, using a single method, such as determining whether a target is on the gripper or judging the distance between two fingers, presents some challenges. For example, if the target is completely black and blends into the fingers, or if the target blends into the background, it can easily lead to errors in target recognition and positioning. Dim lighting may cause errors in identifying key points at the fingertips, and grasping a very flat target can result in a small tolerance for accurate judgment of the distance between the two fingers. Summary of the Invention

[0008] The present invention aims to overcome the above-mentioned defects of the prior art and provide a method and system for judging the grasping state of a robotic arm when grasping small objects.

[0009] This invention uses a robotic arm's hand-eye camera to determine the grasping status in real time, such as whether the target has been successfully grasped, fallen, or placed in the collection box. This invention employs a combined approach of both methods to improve accuracy.

[0010] The present invention provides a method for determining the grasping state of a robotic arm when grasping a small object, comprising:

[0011] S1: Construct a virtual dataset and train a convolutional neural network;

[0012] S2: Determine if the target to be grasped is in your hand;

[0013] S3: Determine the distance between two fingers;

[0014] S4: Determine the gripping status of the robotic arm.

[0015] Furthermore, step S1 specifically includes:

[0016] S11. Use Blender software to generate a virtual environment;

[0017] S12. Change the camera's intrinsic and extrinsic parameters; by changing parameters such as the camera's position, rotation angle, and focal length, different angles of view and field of view can be obtained, thereby generating diverse visual data; adjust the camera's exposure and focus parameters to obtain clearer and more accurate images;

[0018] S13. Change the lighting and background; simulate different lighting conditions by adjusting parameters such as the position, brightness, and color of the light source, thereby generating various lighting effects; change different backgrounds to create different scenes and environments;

[0019] S14. Change the target object; By changing the parameters of the target object's position, size, shape, and pose, various different target objects are generated, thereby constructing a diverse visual dataset;

[0020] S15. Generate photos and annotation information using the annotation software BlenderProc. The annotation information includes the target and key points of the fingertip.

[0021] S16. Finally, on a virtual dataset, train a convolutional neural network, YOLOX, for object recognition.

[0022] Furthermore, step S2 specifically includes:

[0023] S21. By using a hand-eye camera that follows hand movements, it provides RGB information and depth information within a preset distance and preset precision from the camera, and provides grasping status information throughout the process;

[0024] S22. Use a convolutional neural network to perform 2D detection on the image to determine whether the target is located between the two fingers, including: first, identify the target through the trained neural network, filter interference information according to the target size, estimate the center position of the target, and determine whether the nearest target position is within the grasping range of the hand, thereby determining whether the target is located on the hand.

[0025] Furthermore, the preset distance mentioned in step S21 is 30cm, and the preset accuracy is 1%.

[0026] Furthermore, step S3 specifically includes:

[0027] The hand-eye camera is also used to determine the distance between the two fingers. The key points of the two fingertips are identified through a neural network. The key points are selected based on obvious features. The key point recognition uses 2D detection instead of key point detection because there may be point cloud loss or large deviation in real scene. Using 2D bounding box to estimate the center is more robust than using key point detection to return only one point.

[0028] The robotic arm's gripper is initially in an open state. When it grasps an object, it encounters an obstacle during the closing process and cannot close completely. When it does not grasp an object, both fingers are fully closed. Based on point cloud information and recognition detection boxes, the key points of the two finger joints are identified. The outer envelope of the two detection boxes containing the point cloud is calculated, and the center of the outer envelope is used as the center of the key point. The distance between the two centers is calculated. If the distance is greater than a distance threshold, the gripper is judged to be in an open state, and the open state is judged as grasping an object.

[0029] Furthermore, the key point is selected from the finger joints. If there are no obvious points on the fingers, colored paper can be pasted or the positions can be marked with colored pens. The distance threshold is 2.8cm.

[0030] Furthermore, step S4 specifically involves determining the robotic arm's grasping state based on the following two conditions: 1) the target object is located between the two fingers; 2) the distance between the two fingers reaches a distance threshold. As long as the above two conditions are met, the grasping is considered successful. The detection box is output according to the set recognition confidence threshold value.

[0031] Furthermore, the detection box is output when the confidence threshold value is greater than 0.6.

[0032] A second aspect of the present invention relates to a system for determining the grasping state of a robotic arm when grasping a small object, comprising:

[0033] The virtual dataset construction and convolutional neural network training module is used to construct virtual datasets and train convolutional neural networks.

[0034] The target position determination module is used to determine whether the target is in the hand.

[0035] The two-finger distance judgment module is used to judge the distance between two fingers;

[0036] The robotic arm grasping status determination module is used to determine the robotic arm's grasping status.

[0037] A third aspect of the invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a method for determining the grasping state of a robotic arm grasping a small object.

[0038] A fourth aspect of the present invention relates to a device for determining the grasping state of a robotic arm grasping a small object, characterized in that it includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a method for determining the grasping state of a robotic arm grasping a small object according to the present invention.

[0039] This invention uses a virtual dataset as the data source for training neural networks. Virtual datasets offer several advantages, including: data controllability: training the network on a virtual dataset allows complete control over the data generation process. Factors such as object pose, lighting, and background can be adjusted as needed to obtain specific training samples. Diversity and richness: Virtual datasets can generate a large amount of diverse data, covering different scenes and variations. This helps the network better adapt to various situations and improves its generalization ability. Cost-effectiveness: Compared to collecting and labeling real data, using virtual datasets saves significant time and costs. Especially for complex or dangerous scenarios, virtual datasets avoid risks and resource waste. Data scalability: Virtual datasets can be easily expanded to generate any number of training samples. This is particularly important for deep learning tasks requiring large-scale data. Algorithm verification and debugging: Using virtual datasets facilitates algorithm verification and debugging, especially when real datasets are unavailable. Algorithms can be iterated and optimized by quickly generating and modifying virtual datasets. By using a convolutional neural network to identify and locate targets and calculate the distance between two fingers, and further combining two features, accurate judgment of the grasping state can be achieved.

[0040] The working principle of this invention is:

[0041] By constructing virtual environments using 3D virtual modeling technology to generate datasets and annotations, the cost of data collection and annotation is reduced, and the data is controllable and abundant. Visual technology combined with deep learning is used to recognize images, improving the accuracy of judgment in complex environments.

[0042] The innovation of this invention is:

[0043] The robotic arm's gripper structure is well-defined, making modeling relatively easy. Using virtual modeling to generate datasets greatly reduces the cost of data collection and annotation. Visual technology is used for judgment, which can capture the state of the grasped target more comprehensively than mechanical sensors. At the same time, the use of deep learning and a combination of two judgment conditions improves the accuracy of recognition in different environments.

[0044] The advantages of this invention are:

[0045] This method is easy to implement, highly adaptable to different environments, and has a high accuracy rate in determining the grasping status of small objects by a robotic arm. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of a 3D virtual environment built using Blender.

[0047] Figures 2a to 2f It is a dataset consisting of six virtual photographs with different backgrounds and lighting intensities, among which: Figure 2aUsing a yellow sand background, a light intensity of 5000, and smooth yellow pebbles, Figure 2b Using a yellow sand background, 8000 light intensity, and smooth blue pebbles, Figure 2c Using a loess background, light intensity of 2000, and white flat pebbles, Figure 2d The background is made of bluestone slabs, the light intensity is 6000, and the background is made of rough brown pebbles. Figure 2e Using yellow sand background, light intensity 5000, and dark blue smooth pebbles, Figure 2f Use dark bluestone background, light intensity 50, and no pebbles.

[0048] Figure 3 This is a diagram illustrating the annotation of a virtual dataset. Figure 4 This is a schematic diagram of a robotic arm grasping an object in a real-world scenario.

[0049] Figure 5 This is a diagram showing the target object recognition effect under real captured photos.

[0050] Figure 6 This is a diagram illustrating the distance between two fingers of a robotic arm. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0053] Example 1

[0054] Reference Figure 1 The present invention provides a method for determining the grasping state of a robotic arm when grasping small objects, the specific implementation steps of which are as follows:

[0055] 1. Construct a virtual dataset to train a convolutional neural network.

[0056] This invention uses Blender as a 3D scene construction tool. Blender is a powerful 3D modeling and animation software that can generate virtual environments and modify factors such as camera intrinsics and extrinsic parameters, lighting, background, and target objects to create visual training datasets. In the field of vision, the quality of the dataset has a crucial impact on the performance of the algorithm; therefore, creating high-quality visual training datasets is essential.

[0057] First, use Blender to generate a virtual environment. Blender has powerful modeling and rendering capabilities, allowing you to create objects of various shapes and materials, and set different lighting and backgrounds to build realistic virtual environments.

[0058] Secondly, modify the camera's intrinsic and extrinsic parameters. By changing parameters such as the camera's position, rotation angle, and focal length, different angles of view and field of view can be obtained, thus generating diverse visual data. Simultaneously, camera parameters such as exposure and focus can be adjusted to obtain clearer and more accurate images.

[0059] Next, the lighting and background are changed. By adjusting parameters such as the position, brightness, and color of the light source, different lighting conditions can be simulated, thus generating various lighting effects. At the same time, different backgrounds can be changed to create different scenes and environments. The constructed virtual 3D scene is as follows: Figure 1 As shown.

[0060] Finally, the target object is modified. By changing parameters such as the target object's position, size, shape, and pose, various different target objects can be generated, thereby constructing diverse visual datasets.

[0061] This embodiment generated one thousand photos with different parameters such as background, lighting, and target object, such as... Figures 2a to 2f As shown. Further, the image and annotation information are generated using the annotation software BlenderProc, such as... Figure 3 As shown, the annotation information includes the target and fingertip keypoints. Finally, a convolutional neural network, YOLOv, was trained for target recognition on a virtual dataset.

[0062] 2. Determine if the target is in your hand.

[0063] The determination of whether the target is located on the gripper is made using a RealSense D405 depth camera on the robotic arm's hand, installed in the following position: Figure 4 As shown, it can of course be replaced with other cameras. It can provide depth information with an accuracy of less than 1% within 20cm of the camera, and the camera can follow the hand movement, providing grasping status information throughout the process.

[0064] Using convolutional neural networks to classify or detect images can determine whether the target is on the hand. However, image classification makes it difficult to combine the target's depth information to locate the target. For example, when the target is directly below the finger, it may be difficult to distinguish whether the target is on the hand or on the ground from the camera's perspective. Therefore, 2D detection is used here.

[0065] First, a trained neural network is used to identify and detect targets within a certain size range. Interference information is filtered out based on the target size, and the center position of the target is estimated using point cloud data. Further, it is determined whether the nearest target is within the grasping range of the hand, thus determining whether the target is on the hand. The data acquisition and recognition results in real-world scenarios are as follows: Figure 5 As shown.

[0066] 3. Determine the distance between two fingers.

[0067] The hand-eye camera is also used to determine the distance between the two fingers, and a neural network is used to identify key points on the two fingertips, such as... Figure 6 As shown, key points are selected based on their distinctive features, such as knuckles. If there are no distinctive features on the fingers, colored paper or colored pens can be used to mark the positions. Key point recognition uses 2D detection instead of key point detection because point cloud loss or large deviations may occur in real-world scenarios. Using 2D bounding boxes to estimate the center is more robust than using key point detection, which only returns a single point.

[0068] Further, based on the point cloud information and the recognition detection boxes, the outer envelope boxes containing the point cloud of the two detection boxes are calculated, and the center of the outer envelope box is used as the center of the key point. The distance between the two centers is further calculated. If the distance is greater than a certain distance (such as 2.8cm), it is judged as an open state, and the open state is judged as the object being grasped.

[0069] 4. Determine the gripping status of the robotic arm.

[0070] Since both the target location and the distance between the two fingers are determined using neural network detection, its robustness is low when the environment changes significantly. Therefore, a higher confidence threshold is set during recognition; a detection box is output when the threshold is greater than 0.6. Furthermore, a successful capture is determined as long as the above two conditions are met.

[0071] Example 2

[0072] This embodiment relates to a robotic arm grasping small objects grasping state determination system, used to implement the robotic arm grasping small objects grasping state determination method of Embodiment 1, including:

[0073] The virtual dataset construction and convolutional neural network training module is used to construct virtual datasets and train convolutional neural networks.

[0074] The target position determination module is used to determine whether the target is in the hand.

[0075] The two-finger distance judgment module is used to judge the distance between two fingers;

[0076] The robotic arm grasping status determination module is used to determine the robotic arm's grasping status.

[0077] Example 3

[0078] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements a method for determining the grasping state of a robotic arm grasping a small object, as described in Embodiment 1.

[0079] Example 4

[0080] This embodiment relates to a device for determining the grasping state of a robotic arm grasping a small object, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a method for determining the grasping state of a robotic arm grasping a small object according to Embodiment 1.

[0081] At the hardware level, the device for determining the grasping state of the robotic arm when grasping small objects includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for the operation. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0082] Improvements in a technology can be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology can now be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement in methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0083] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0084] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0085] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, in implementing this invention, the functions of each unit can be implemented in one or more software and / or hardware components.

[0086] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0091] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0092] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0093] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0094] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0096] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0097] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for determining the grasping state of a robotic arm when grasping a small object, characterized in that, include: S1: Construct a virtual dataset and train a convolutional neural network; S2: Determine if the target to be grasped is in hand; specifically: S21. By using a hand-eye camera that follows hand movements, it provides RGB information and depth information within a preset distance and preset precision from the camera, and provides grasping status information throughout the process; S22. Use a convolutional neural network to perform 2D detection on the image to determine whether the target is located between the two fingers, including: first, identify the target through the trained neural network, filter interference information according to the target size, estimate the center position of the target, and determine whether the nearest target position is within the grasping range of the hand, thereby determining whether the target is located on the hand. S3: Determine the distance between two fingers; S4: Determine the robotic arm's grasping status; The robotic arm's grasping status is determined by the following two conditions: 1) the target object is located between the two fingers; 2) the distance between the two fingers reaches the distance threshold. As long as the above two conditions are met, the grasping is considered successful. The detection box is output according to the set recognition confidence threshold value.

2. The method for determining the grasping state of a robotic arm grasping a small object according to claim 1, characterized in that, Step S1 is as follows: S11. Use Blender software to generate a virtual environment; S12. Change the camera's intrinsic and extrinsic parameters; by changing the camera's position, rotation angle, and focal length parameters, different angles of view and field of view can be obtained, thereby generating diverse visual data; adjust the camera's exposure and focus parameters to obtain clearer and more accurate images; S13. Change the lighting and background; simulate different lighting conditions by adjusting the position, brightness, and color parameters of the light source to generate various lighting effects; change the background to create different scenes and environments; S14. Change the target object; By changing the parameters of the target object's position, size, shape, and pose, various different target objects are generated, thereby constructing a diverse visual dataset; S15. Generate photos and annotation information using the annotation software BlenderProc. The annotation information includes the target and key points of the fingertip. S16. Finally, on a virtual dataset, train a convolutional neural network, YOLOX, for object recognition.

3. The method for determining the grasping state of a robotic arm grasping a small object according to claim 1, characterized in that, The preset distance mentioned in step S21 is 30cm, and the preset accuracy is 1%.

4. The method for determining the grasping state of a robotic arm grasping a small object according to claim 1, characterized in that, Step S3 is as follows: The hand-eye camera is also used to determine the distance between the two fingers. The key points of the two fingertips are identified through a neural network. The key points are selected based on obvious features. The key point recognition uses 2D detection instead of key point detection because there may be point cloud loss or large deviation in real scene. Using 2D bounding box to estimate the center is more robust than using key point detection to return only one point. The robotic arm's gripper is initially in an open state. When it grasps an object, it encounters an obstacle during the closing process and cannot close completely. When it does not grasp an object, both fingers are fully closed. Based on point cloud information and recognition detection boxes, the key points of the two finger joints are identified. The outer envelope of the two detection boxes containing the point cloud is calculated, and the center of the outer envelope is used as the center of the key point. The distance between the two centers is calculated. If the distance is greater than a distance threshold, the gripper is judged to be in an open state, and the open state is judged as grasping an object.

5. The method for determining the grasping state of a robotic arm grasping a small object according to claim 4, characterized in that, The key points are selected from the finger joints. If there are no obvious points on the fingers, mark the positions with colored paper or colored pens.

6. The method for determining the grasping state of a robotic arm grasping a small object according to claim 4, characterized in that, The distance threshold is 2.8cm.

7. The method for determining the grasping state of a robotic arm grasping a small object according to claim 1, characterized in that, If the confidence threshold is greater than 0.6, output the detection box.

8. A system for determining the grasping state of a robotic arm when grasping a small object, used to implement the method for determining the grasping state of a robotic arm when grasping a small object as described in claim 1, characterized in that, include: The virtual dataset construction and convolutional neural network training module is used to construct virtual datasets and train convolutional neural networks. The target position determination module is used to determine whether the target is in the hand. The two-finger distance judgment module is used to judge the distance between two fingers; The robotic arm grasping status determination module is used to determine the robotic arm's grasping status.

9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements a method for determining the grasping state of a robotic arm grasping a small object, as described in any one of claims 1-7.

10. A device for determining the grasping state of a robotic arm when grasping a small object, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a method for determining the grasping state of a robotic arm grasping a small object, as described in any one of claims 1-7.

Images