Methods for generating training data for robotic arms, training methods, media, and robotic arm systems

Abstract

This application provides a method for generating training data for a robotic arm, a training method, a medium, and a robotic arm system, belonging to the field of data processing technology. The method for generating training data for a robotic arm includes: capturing the process of a robotic arm grasping a target object in the environment using multiple cameras to form multi-view videos; locating the start and end positions of each complete grasping cycle based on the grasping motion trajectory of the robotic arm; segmenting the multi-view videos based on the start and end positions to form one or more single-cycle grasping videos, each single-cycle grasping video corresponding to a complete grasping cycle; performing target object grasping analysis on the single-cycle grasping videos to identify whether the robotic arm successfully completed the target object grasping cycle in the single-cycle grasping video; and generating training data for robotic arm training based on the single-cycle grasping videos that successfully completed the target object grasping cycle. This application can improve the quality of generated training data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for generating training data for a robotic arm, a training method, a medium, and a robotic arm system. Background Technology

[0002] In recent years, the demand for dexterous hand robots in industrial automation, service robots, and special operations has been growing. The core challenge lies in achieving the fine, robust, and adaptive manipulative capabilities of a human hand. Deep learning-based control strategies (such as reinforcement learning and imitation learning) have become the mainstream approach to improving dexterous hand performance. However, the effectiveness of such methods is highly dependent on high-quality, large-scale, and diverse training data.

[0003] There are two existing methods for generating training data. One is to use real machine data. However, real machine experiments (such as grasping and manipulating objects) are usually time-consuming, labor-intensive, and consume hardware (e.g., wear and tear, potential damage), resulting in extremely high data acquisition costs. Furthermore, the complexity of the physical world and limitations of experimental conditions (such as time and resources) mean that datasets obtained through real machine experiments often have limited scene coverage, sparse state spaces, and uneven action distribution. This directly limits the generalization ability of the trained model and its robustness in complex and unknown environments. The second method is to use simulated data. However, the realism of simulated data still falls short of the training requirements for dexterous hand robots. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, medium, and equipment for processing on-board information of environmentally friendly vehicles, so as to solve at least one of the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for generating training data for a robotic arm, the method comprising:

[0006] The process of a robotic arm grasping target objects in the environment is captured by multiple cameras, forming a multi-view video.

[0007] The starting and ending positions of each complete grasping cycle are located based on the grasping motion trajectory of the robotic arm.

[0008] The multi-view captured video is segmented based on the start position and the end position to form one or more single-cycle capture videos, each single-cycle capture video corresponding to a complete capture cycle;

[0009] The single-cycle grasping video is analyzed to determine whether the robotic arm in the single-cycle grasping video has successfully completed the target item grasping cycle.

[0010] Training data for training the robotic arm is generated based on a single-cycle grasping video that successfully completes the target object grasping cycle.

[0011] Optionally, the step of segmenting the multi-view captured video based on the start and end positions to form one or more single-cycle capture videos includes:

[0012] Obtain the start time corresponding to the start position and the end time corresponding to the end position;

[0013] Video segments within the start and end time intervals of a capture cycle are extracted from the multi-view captured video, and these video segments are used as the single-cycle capture video.

[0014] Optionally, the step of performing target item grasping analysis on the single-cycle grasping video and identifying whether the robotic arm in the single-cycle grasping video successfully completed the target item grasping cycle includes:

[0015] For each single-cycle video capture, image frames are extracted to form data slices;

[0016] Identify whether each frame in the data slice represents a successful capture of the target item, and assign a capture value to each frame based on the capture success identification result.

[0017] The capture success confidence of a data slice for a complete capture cycle is obtained by weighted summation based on the capture assignment of each frame image.

[0018] Based on the confidence level of successful grasping, it is determined whether the robotic arm has successfully completed the target item grasping cycle.

[0019] Optionally, identifying whether each frame in the data slice represents a successful capture of the target item includes:

[0020] The positional relationship between the robotic arm and the target object is identified based on sub-images from each shooting perspective in each frame;

[0021] The image to be analyzed is taken as the current frame image. When the robotic arm is identified to have contacted the target object based on the positional relationship, it is analyzed whether there is a change between the first position of the target object in the current frame image and the second position in the adjacent frame image. If there is a change, it is determined that the current frame image indicates that the target object has been successfully grasped. If there is no change, it is determined that the current frame image indicates that the object has failed to be grasped.

[0022] When it is determined that the robotic arm has not contacted the target item based on the positional relationship, it is determined whether the first position of the target item in the current frame image matches the preset position. Based on the matching between the first position and the preset position, it is determined whether the current frame image indicates successful grabbing of the item. The preset position is set according to the position of the current frame image in the data slice according to the order of timestamps.

[0023] Optionally, identifying the positional relationship between the robotic arm and the target object based on sub-images from each shooting perspective in each frame includes:

[0024] Identify the first sub-region of the target object in a sub-image under the first shooting view of the current frame image, and form a mask map under the first shooting view based on the first sub-region.

[0025] Identify the second sub-region in a sub-image within the first shooting perspective of the current frame image;

[0026] When the first sub-region and the second sub-region come into contact with each other, the third sub-region in which the target item is located in the sub-image under the second shooting perspective of the current frame image is identified;

[0027] Based on the perspective relationship between the first shooting perspective and the second shooting perspective, the third sub-region is transformed to obtain the fourth sub-region corresponding to the third sub-region in the masking image, and the fourth sub-region is marked in the masking image. The first sub-region and the fourth sub-region are taken as the first position of the target item in the current frame image.

[0028] Optionally, the step of assigning a capture value to each frame image based on the capture success recognition result of each frame image includes: assigning a value of 1 to images that indicate successful capture and assigning a value of 0 to images that indicate failed capture.

[0029] Optionally, the weighted summation based on the captured values ​​of each image frame includes: assigning weights to each image frame according to the timestamp of each image frame in the data slice, wherein the later the image in the data slice is ordered by timestamp, the greater its corresponding weight.

[0030] Optionally, when the target item grabbing cycle fails to complete, the type of the target item that failed to be grabbed is identified; the type of the target item that failed to be grabbed is used as the type to be trained; the target number of grabbing attempts for the target item of the type to be trained is determined based on the number of samples in the existing training data; and the target item is grabbed again for the target number of grabbing attempts.

[0031] A second aspect of this application provides a robotic arm grasping training method, the method comprising:

[0032] Obtain a training dataset, wherein the training data in the training dataset is training data generated by the robotic arm training data generation method described in any embodiment of this application;

[0033] The preset robotic arm grasping model is iteratively trained based on the training dataset to form a trained robotic arm grasping model.

[0034] In a third aspect, this application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, cause the processor to perform the robotic arm training data generation method and / or robotic arm grasping training method as described in any embodiment of this application.

[0035] A fourth aspect of this application provides a robotic arm system, comprising: a robot body; a robotic arm; a camera disposed on the robot body or the robotic arm for capturing environmental images; and one or more processors for controlling the robotic arm to grasp objects based on a trained robotic arm grasping model generated according to the robotic arm grasping training method in the embodiments of this application, and / or

[0036] A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the robotic arm training data generation method and / or robotic arm grasping training method described in any embodiment of this application.

[0037] The robotic arm training data generation method, training method, medium, and robotic arm system in this application control the robotic arm to grasp a target object. Video of the grasping process is captured from multiple angles, and the start and end positions of a complete grasping cycle are determined based on the grasping motion trajectory. Then, based on the start and end positions, each single-cycle grasping video is segmented from the multi-angle video, improving the accuracy of single-cycle grasping video segmentation. Successful grasping analysis of the target object is performed on the single-cycle grasping video, and training data is generated based on the single-cycle grasping videos that successfully grasp the target object, thus improving the quality of training data generated for the robotic arm's grasping model. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0039] Figure 1This is a flowchart illustrating a method for generating training data for a robotic arm in one embodiment;

[0040] Figure 2 This is a schematic diagram of a robotic arm grasping a target object in one embodiment;

[0041] Figure 3 This is a schematic diagram of multi-view images in one embodiment;

[0042] Figure 4 This is a schematic diagram of a multi-view captured image including a mask image in one embodiment;

[0043] Figure 5 This is a schematic diagram of a multi-view image including a masking image in another embodiment;

[0044] Figure 6 This is a flowchart illustrating the process of analyzing a single-cycle grasping video to identify whether the robotic arm in the single-cycle grasping video has successfully completed the target item grasping cycle, as shown in one embodiment.

[0045] Figure 7 This is a schematic diagram of a process for identifying the positional relationship between a robotic arm and a target object based on sub-images from each shooting perspective in each frame of an image, as shown in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0048] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0049] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0050] This application provides a method for generating training data for a robotic arm, such as... Figure 1 As shown, the method includes:

[0051] Step 110: The process of the robotic arm grasping the target object in the environment is captured by multiple cameras to form a multi-view video.

[0052] In this embodiment, combined with Figure 2 As shown, a gripping scenario is pre-set up for a robotic arm to grasp objects. This scenario includes one or more target objects that the robotic arm needs to grasp, and a placement area for placing these target objects. Figure 2 As shown, multiple items can be placed on a platform as target items, and one or more placement areas, such as placement area A and placement area B, can be set on this platform. The robotic system can control a robotic arm to grasp the target items and move them to placement area A or placement area B. Specifically, for example, it can grasp one item from placement area A as the target item, move it to placement area B, and then grasp one target item from placement area B and move it to placement area A. By periodically grasping items back and forth between the two placement areas, the grasping efficiency of the target items can be improved.

[0053] Optionally, there are multiple ways to arrange the robotic arm to grasp the target object. For example, you can set up only one placement area for the grasped object, and control the robotic arm to grasp the target object from the starting point and move it to a unified placement area.

[0054] The robot system is also equipped with multiple cameras, each with a fixed relative position. These cameras can be fixedly mounted at different locations on a rigid component, ensuring that the shooting angles are all the same, such as mounting them on a rigid fixture on the robot body. All cameras simultaneously capture the robotic arm's grasping process, forming a multi-view video. This multi-view video contains sub-videos captured by each camera.

[0055] Because the cameras are positioned differently, the robotic arm captures different gripping angles, resulting in videos taken from multiple perspectives.

[0056] For example, the robot system is equipped with a hand camera and a head camera. The relative positions of the two cameras are fixed. During the movement of the robotic arm, these two cameras capture the movement of the robotic arm in real time, forming multi-view video. Figure 3The image shown is an example of one frame from this multi-view video. The left image is a sub-image captured by the hand camera, and the right image is a sub-image captured by the head camera. These two sub-images constitute one frame of the multi-view video. It is understood that any other suitable method can be used to set the number and position of the cameras.

[0057] Step 120: Based on the grasping motion trajectory of the robotic arm, locate the start and end positions of each complete grasping cycle.

[0058] Step 130: Cut the multi-view video based on the start and end positions to form one or more single-cycle capture videos.

[0059] In this embodiment, each single-cycle grasping video corresponds to a complete grasping cycle. A multi-view video may contain one or more complete grasping cycles of the robotic arm. One complete grasping cycle represents the process of the robotic arm picking up the target object from the starting position and placing it to the ending position. For example... Figure 2 As shown, the process of picking up a target item from placement area A and placing it in placement area B is one complete picking cycle. Similarly, the process of picking up a target item from placement area B and placing it in placement area A is also one complete picking cycle.

[0060] The robot system can identify the position of the robotic arm and / or the target object in multi-view video. Based on the changes in the position of the robotic arm and the position of the target object, the start and end times of a complete grasping cycle can be determined. The position of the robotic arm at the start time is the start position, and the position of the robotic arm at the end time is the end position.

[0061] The multi-view video is segmented based on the identified start and end times (start and end positions) to form one or more single-cycle capture videos. The first frame of the single-cycle capture video is the image at the corresponding start time (start position), and the last frame is the image at the corresponding end time (end position).

[0062] In one embodiment, step 130 includes: obtaining the start time corresponding to the start position and the end time corresponding to the end position; extracting video segments from the multi-view captured video that fall within the start time and end time interval of a capture cycle, and using the video segments as single-cycle capture videos.

[0063] The starting position and the ending position can be fixed positions that are preset. For example, the starting position and the ending position of the robotic arm can be preset during a complete grasping cycle, and the starting time of the robotic arm when it is in the starting position and the ending time when it is in the ending position can be recorded.

[0064] Furthermore, the starting and ending positions do not have to be preset fixed positions. The robotic arm system can adaptively determine the starting and ending positions based on the grasping motion trajectory of the robotic arm. When the robotic arm moves to the starting position, it indicates the start of a corresponding grasping process; when the robotic arm moves to the preset ending position, it indicates the end of the corresponding grasping process.

[0065] Taking a complete grasping cycle as the process of picking up a target item from placement area A and placing it into placement area B, the corresponding start time is when the target item is within placement area A and the robotic arm is at the preset starting position before grasping the target item, and the end time is when the target item is within placement area B and the robotic arm is at the preset ending position after the item has been placed. Conversely, the start and end positions for a complete grasping cycle can also be set according to the process of picking up a target item from placement area B and placing it into placement area A.

[0066] For this multi-view video, identify the image frames captured at the start and end times, and segment the multi-view video according to the start and end times so that the first frame of the segmented video is the image frame at the start time and the last frame is the image frame at the end time. Then the image segment is a single-cycle capture video.

[0067] In one embodiment, when the robotic arm is in the starting position, it is located in a specific first region in the corresponding image frame of the multi-view captured image. Similarly, when the robotic arm is in the ending position, it is located in a specific second region in the corresponding image frame. By identifying the region where the robotic arm is located in each frame of the multi-view captured video, the image frames in the first and second regions can be located respectively. Based on the located image frames, the video is segmented to form a corresponding single-cycle capture video.

[0068] The image of the robotic arm located in the first region is used as the first frame of the single-cycle grasping video. The image of the robotic arm located in the second region, which is first identified after this frame, is used as the second frame of the single-cycle grasping video.

[0069] For image frames in the multi-view video that do not contain a complete capture cycle, they are discarded.

[0070] Step 140: Analyze the target object grasping in the single-cycle grasping video to identify whether the robotic arm in the single-cycle grasping video has successfully completed the target object grasping cycle.

[0071] Step 150: Generate training data for robotic arm training based on a successful single-cycle grasping video of the target object grasping cycle.

[0072] The robotic arm does not always successfully grasp the target object. Image recognition is performed on the single-cycle grasping video to detect whether the image frames demonstrate successful grasping and placement of the target object. Based on this, the success of the grasping cycle is determined. For example, if the target object is detected to have fallen during the grasping process or not been correctly placed in the designated area, the grasp is considered a failure. Success is determined when the object is successfully grasped and placed in the preset area. Optionally, the system can also detect whether the target object is damaged during the grasping process; if damage is caused, the grasp is considered a failure.

[0073] The location of the robotic arm, the placement area, and the damage / integrity of the items can all be comprehensively judged by combining images taken from multiple perspectives, thereby improving the accuracy of the placement area judgment.

[0074] If a capture fails, the single-cycle capture video is discarded. If a capture is successful, the single-cycle capture video can be directly used as training data, or it can be processed accordingly, and the processed data can be used as training data. The processing of the single-cycle capture video can involve frame extraction, using the resulting data slices as training data; or it can involve performing pose analysis on the robotic arm based on the video and other relevant data, using the resulting pose data as training data.

[0075] The robotic arm training data generation method in this application controls the robotic arm to grasp the target object, collects video of the grasping process from multiple angles, and determines the start and end positions of a complete grasping cycle based on the grasping motion trajectory of the robotic arm. Then, based on the start and end positions, each single-cycle grasping video is cut out from the multi-angle video, which improves the accuracy of single-cycle grasping video cutting.

[0076] By analyzing the success rate of target object grasping in this single-cycle grasping video, and using the single-cycle grasping video that successfully grasped the target object to form training data, the quality of training data generated for the robotic arm's grasping model can be improved.

[0077] In one embodiment, such as Figure 6 As shown, step 140 includes:

[0078] Step 610: Extract frames from each single-cycle captured video to form data slices.

[0079] In this embodiment, each camera in the robotic arm system can capture video at a preset frame rate. Each camera can have the same frame rate, such as 5 FPS, 7.5 FPS, 10 FPS, 15 FPS, 20 FPS, or any other suitable frame rate. For a single-cycle video capture, images are uniformly extracted based on the required number of frames for capture and analysis, forming data slices with a suitable number of frames, ensuring that each data slice contains the same number of frames.

[0080] For example, if the frame rate of a single-cycle video capture is 10 FPS and the duration of each video cycle is 10 seconds, the robot system extracts images according to a preset extraction interval, such as extracting one frame every 10 frames, and extracts 10 frames as data slices.

[0081] Step 620: Identify whether each frame in the data slice represents a successful capture of the target item, and assign a capture value to each frame based on the capture success identification result.

[0082] In this embodiment, target item capture and recognition are performed on each frame of the data slice to determine the target item capture result presented in each frame of the image, to obtain whether the target item was successfully captured in each frame of the image, and to assign a capture value to it based on the recognition result.

[0083] Optionally, the successful capture result can be represented as a successful capture or a failed capture, or it can be represented as the probability of a successful capture. The larger the assigned value, the higher the success rate of capturing the identified target item. For example, an image representing a successful capture can be assigned a value of 1, and an image representing a failed capture can be assigned a value of 0. Optionally, the value can also be assigned based on the probability of a successful capture; the higher the probability, the larger the value.

[0084] Specifically, for each frame of an image, one or more of the following can be identified: whether the target object was successfully grasped by the robotic arm; whether the target object was grasped to the correct target position; and whether the target object remained intact before and after being grasped by the robotic arm. This yields the target object grasping recognition result for the corresponding frame. This analysis can be performed independently on each frame based on images from multiple perspectives, or it can be combined with the current frame (the image to be analyzed) and its preceding and following frames for correlation analysis.

[0085] Step 630: Perform a weighted summation based on the capture assignment of each image frame to obtain the capture success confidence of a data slice for a complete capture cycle.

[0086] Step 640: Determine whether the robotic arm has successfully completed the target item grasping cycle based on the grasping success confidence level.

[0087] In this embodiment, the robotic arm system further sets the weights of each frame in the data slice. The calculated values ​​are then weighted and summed with the corresponding weights to obtain the corresponding confidence level of successful capture.

[0088] The calculated confidence level is compared with a preset confidence threshold. When the confidence level exceeds the preset confidence threshold, it is determined that the robotic arm in the data slice has successfully completed the target item grasping cycle.

[0089] Furthermore, for data slices that successfully complete the grasping cycle, relevant labels can be applied, including labels for one or more dimensions such as the grasped target item information, grasping environment, and robotic arm motion characteristics. The target item information includes one or more of the following: target item identifier, item category, item material, item geometry, and item pose. The grasping environment includes one or more of the following: current lighting level and grasping background (e.g., black background, white background, complex background lighting). The robotic arm motion characteristics can include one or more of the following: picking up, transferring, placing, and moving.

[0090] In one embodiment, identifying whether each frame in a data slice represents a successful grasp of a target object includes: identifying the positional relationship between the robotic arm and the target object based on sub-images from each shooting angle in each frame, and determining whether each frame represents a successful grasp of the target object based on the positional relationship.

[0091] In this embodiment, the positional relationship between the robotic arm and the target object reflects whether the robotic arm is in contact with the target object. When the robotic arm contacts the target object, it indicates that the robotic arm needs to move the target object. When the robotic arm does not contact the target object, it indicates that the robotic arm is moving towards the target object to grasp it, or that the robotic arm has completed moving the target object to the placement area and placing it down.

[0092] Optionally, the positions / regions of the robotic arm and the target object in the corresponding sub-images will differ depending on the shooting angle. The robotic arm system can perform image recognition on each sub-image within each frame to identify the robotic arm and the target object in each sub-image, thus determining the regions where the robotic arm and the target object are located in each sub-image. When the regions where the robotic arm and the target object are located overlap in each sub-image of a frame, or when mutual occlusion between the target object and the robotic arm is detected, it indicates that the corresponding robotic arm has made contact with the target object. Figure 3 and Figure 4 / Figure 5 As shown, it can be identified Figure 3 If the robotic arm and the target object occlude each other in both sub-images, then it indicates that... Figure 3 The robotic arm and the target object are in contact. Figure 4 / Figure 5 In the two sub-images, the robotic arm with fingers in the left sub-image does not contact the target object, while in the middle sub-image, the robotic arm occludes the target object (the area where the robotic arm is located in the sub-image and the area where the target object is located in the sub-image overlap), indicating that the robotic arm does not contact the target object.

[0093] In one embodiment, such as Figure 7 As shown, the positional relationship between the robotic arm and the target object is identified based on sub-images from each shooting perspective in each frame, including:

[0094] Step 710: Identify the first sub-region of the target object in the sub-image under the first shooting view of the current frame image, and form a mask map under the first shooting view based on the first sub-region.

[0095] In this embodiment, image transformation is performed on a sub-image under the first shooting view to form a mask image of that sub-image. The mask image reflects the area (i.e., the first sub-region) where the target object is located under the first shooting view. Target objects identified in the sub-image under the first shooting view are displayed in the mask image with a first pixel value, while non-target objects are displayed in the mask image with a second pixel value. For example... Figure 4 As shown, for example, for a target item in a sub-image at a identified middle position, in the corresponding mask image ( Figure 4 In the sub-image on the right side of the image, the same location is shown in green, while other areas that are not the target item are shown in black. By performing a masking transformation, the location of the target item can be visually determined.

[0096] Step 720: Identify the second sub-region in the sub-image of the current frame image where the robotic arm is located.

[0097] Similar to the identification of target objects, the robotic arm is also identified in this sub-image to obtain the region where the robotic arm is located in this sub-image (i.e., the second sub-region).

[0098] Step 730: When the first sub-region and the second sub-region come into contact with each other, identify the third sub-region in which the target object is located in the sub-image under the second shooting perspective of the current frame image.

[0099] After obtaining the first and second sub-regions, it is possible to detect whether there is contact between them. If there is contact, it indicates that the robotic arm has touched the target object or that the robotic arm has obstructed the target object. In this case, the sub-image from the second shooting perspective can be used for further confirmation. At this point, the position of the target object in the sub-image from the second shooting perspective can be identified, and its third sub-region in that sub-image can be obtained based on its position.

[0100] Specifically, it can detect whether the edges between the first sub-region and the second sub-region overlap, and if they overlap, it determines that they are in contact.

[0101] Step 740: Based on the perspective relationship between the first shooting view and the second shooting view, the position of the third sub-region is transformed to obtain the fourth sub-region corresponding to the third sub-region in the mask image, and the fourth sub-region is marked in the mask image. The first sub-region and the fourth sub-region are taken as the first position of the target object in the current frame image.

[0102] By combining the perspective relationship between the first and second shooting perspectives, the positions of pixels at various locations in the sub-image from the second shooting perspective within the sub-image from the first shooting perspective can be obtained. Based on this positional relationship, the third sub-region is mapped onto the mask image, resulting in the corresponding fourth sub-region. The fourth sub-region and the first sub-region are then merged; the merged region represents the first position of the target object in the current frame image, which is its position under the first shooting perspective. Figure 5 As shown, the green area in the right-hand image is the first sub-region of the target object, and the orange area is the third sub-region of the target object identified in the left-hand sub-image. This third sub-region is the area added after being converted to the mask image and merged with the first sub-region. The green and orange areas together constitute the first position of the target object.

[0103] By combining sub-images from two perspectives to identify the location of a target object, the accuracy of the target object location identification can be improved.

[0104] In one embodiment, when the first sub-region and the second sub-region do not touch, the position of the first sub-region is directly taken as the first position of the target object. In this case, there is no need to identify the target object for the sub-image under the second shooting perspective.

[0105] In one embodiment, the image to be analyzed is used as the current frame image. When the robotic arm is identified to have contacted the target object based on the positional relationship, it is analyzed whether there is a change between the first position of the target object in the current frame image and the second position in the adjacent frame image. If there is a change, it is determined that the current frame image indicates that the target object has been successfully grasped. If there is no change, it is determined that the current frame image indicates that the object has failed to be grasped.

[0106] In this embodiment, adjacent frame images can be either the previous frame image or the next frame image of the current frame image. The position of the target object identified in the current frame image is taken as the first position, and the position of the target object identified in the adjacent frame image of the current frame image is taken as the second position. The first position and the second position are positions under the same shooting perspective. By comparing the two positions, it is identified whether the position has changed. When a change occurs, it indicates that the robotic arm has moved the target object, thus determining that the object in the current frame image has been successfully grasped. If no change occurs, it indicates that although the robotic arm has contacted the target object, it has not moved it, and the object grasping has failed.

[0107] Furthermore, when it is determined that the robotic arm has not contacted the target object based on the positional relationship, it is also analyzed whether there is a change between the first position of the target object in the current frame image and the second position in the adjacent frame image. If there is a change, it is determined that the target object in the current frame image has been successfully grasped; if there is no change, it is determined that the target object in the current frame image has failed to be grasped.

[0108] In this embodiment, position change recognition is performed on all images in the data slice. If a position change occurs, the capture is considered successful; if no position change occurs, the capture is considered unsuccessful, thus simplifying the method of recognizing successful capture.

[0109] In one embodiment, when it is determined that the robotic arm has not contacted the target item based on the positional relationship, it is determined whether the first position of the target item in the current frame image matches the preset position. Based on the matching between the first position and the preset position, it is determined whether the current frame image indicates successful grabbing of the item. The preset position is set according to the position of the current frame image in the data slice according to the order of timestamps.

[0110] In this embodiment, the preset position may include the initial placement position and the target placement position of the target item. The initial placement position is the position of the target item before the robotic arm grasps it; the target placement position is the target position to which the robotic arm moves the target item. The robotic arm system can determine whether the preset position is the initial placement position or the target placement position based on the position of the current frame image in the data slice. For example, for the first frame image in the data slice, its corresponding preset position is the initial placement position, and for the last frame image in the data slice, its corresponding preset position is the target placement position.

[0111] By comparing the identified first position with the preset position, if the two match, it means that the robotic arm has successfully grasped the target object; if they do not match, it means that the target object has failed to be grasped.

[0112] In one embodiment, weighted summation is performed based on the captured values ​​of each image frame, including: assigning weights to each image frame according to the timestamp of each image frame in the data slice, wherein the later the image in the data slice is ordered by timestamp, the greater its corresponding weight.

[0113] In this embodiment, the later an image is sorted, the more it reflects the success rate of the corresponding capture cycle, and therefore it is assigned a larger value.

[0114] Optionally, the weight of the i-th frame image in the data slice can be set. The α weight attenuation factor can be between 0 and 1, and its value can be set according to the actual situation. The assignment of the i-th frame image... When the i-th frame indicates a successful capture, ri = 1; when it indicates a failed capture, r = 1. i =0.

[0115] Based on the above settings and weights, the corresponding confidence level K = , where n represents the total number of frames in the data slice.

[0116] Taking n=5 and α=0.8 as an example, the assignment vector r=[0, 0, 1, 1,1] formed by the assignments corresponding to each frame image, and the weight vector formed by the weights corresponding to each frame image. = [0.4096,0.512,0.64,0.8,1], then the corresponding confidence level K = 72.6%.

[0117] In one embodiment, when the target item grabbing cycle fails to complete, the type of the target item that failed to be grabbed is identified; the type of the target item that failed to be grabbed is used as the type to be trained; the number of target grabbing times for the target item of the type to be trained is determined based on the number of samples of the existing training data; and the target item is grabbed again for the target number of grabbing times.

[0118] In this embodiment, for each failed grasping cycle, the type of the target item that failed to be grasped can be recorded. Based on the pre-planned amount of training data required for each item type, the number of grasps for that failed target item is determined. The robotic arm is then controlled to grasp the corresponding target item according to the required number of grasps, thereby generating the required sample size. By allowing the robotic arm to autonomously collect grasping data of poorly performing target items and then autonomously allocate the proportion of training data according to a preset label filtering logic, the number of grasps for each label's target item can be determined, thus improving the quality of the samples subsequently used for model training.

[0119] In one embodiment, a robotic arm grasping training method is also proposed, the method comprising: acquiring a training dataset; iteratively training a preset robotic arm grasping model based on the training dataset to form a trained robotic arm grasping model.

[0120] The training dataset consists of training data generated according to the robotic arm training data generation method described in this application. The training data in this dataset is labeled accordingly, and the robotic arm grasping model can specifically be a VLA (Visual Language Action) model.

[0121] The training data generated using the aforementioned robotic arm training data generation method is labeled to form a training dataset. This training dataset is then used to iterate the pre-trained robotic arm grasping model a predetermined number of times. The iteration ends when the number of iterations reaches a pre-defined threshold, or when the accuracy of the trained model reaches a predetermined level, resulting in a trained robotic arm grasping model.

[0122] In one embodiment, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the above method embodiments.

[0123] In one embodiment, a robotic arm system is also provided, comprising: a robot body; a robotic arm; a camera configured on the robot body or the robotic arm for capturing environmental images; and one or more processors for controlling the robotic arm to grasp objects based on a trained robotic arm grasping model generated by the robotic arm grasping training method of this application.

[0124] In one embodiment, the robotic arm system further includes a memory for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to perform the method described in any embodiment of this application.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0126] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments. For example, all the embodiments above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for generating training data of a robot arm, characterized in that, The method comprises: photographing the target object in the environment by a plurality of cameras to form a multi-view video; locating a starting position and an ending position of each complete grabbing cycle based on the grabbing motion trajectory of the mechanical arm; cutting the multi-view video based on the starting position and the ending position to form one or more single-cycle grabbing videos, each single-cycle grabbing video corresponding to a complete grabbing cycle; image frame extraction is performed on each single-cycle grabbing video to form a data slice; identifying the positional relationship between the mechanical arm and the target object based on the sub-image under each shooting angle in each frame image, judging whether each frame image represents successful grabbing of the target object based on the positional relationship, and performing grabbing assignment on each frame image according to the grabbing success identification result of each frame image; performing weighted summation according to the grabbing assignment of each frame image to obtain a grabbing success confidence of the data slice of the complete grabbing cycle; determining whether the mechanical arm successfully completes the target object grabbing cycle based on the grabbing success confidence; generating training data for training the mechanical arm based on the single-cycle grabbing video in which the target object grabbing cycle is successfully completed; The method comprises:

2. The method of claim 1, wherein, obtaining a starting time corresponding to the starting position and an ending time corresponding to the ending position; extracting a video segment in the starting time and the ending time interval of a grabbing cycle from the multi-view video, and taking the video segment as the single-cycle grabbing video. The method comprises:

3. The method of claim 1, wherein, ​ the image to be analyzed is taken as a current frame image, when it is identified based on the position relationship that the mechanical arm contacts the target object, it is analyzed whether a first position where the target object appears in the current frame image and a second position where the target object appears in a neighboring frame image are changed, when there is a change, it is determined that the current frame image represents successful grabbing of the target object, when there is no change, it is determined that the current frame image represents failed grabbing of the target object; when it is identified based on the position relationship that the mechanical arm does not contact the target object, it is identified whether a first position where the target object is located in the current frame image matches a preset position, and it is determined whether the current frame image represents successful grabbing of the target object according to matching between the first position and the preset position, the preset position being set according to a position where the current frame image is located in the data slice according to time stamp sorting.

4. The method of claim 1, wherein, the grabbing assignment of each frame image according to the grabbing success identification result of each frame image includes: assigning an image representing successful grabbing as 1 and assigning an image representing failed grabbing as 0; the weighted summation according to the grabbing assignment of each frame image includes: setting a weight of each frame image according to a time stamp of each frame image in the data slice, and in images in the data slice according to time stamp sorting, the later the sorting is, the greater the corresponding weight is.

5. The method according to any one of claims 1 to 4, characterized in that, when a target object grabbing cycle is completed in failure, the type of the target object of which grabbing fails is identified; the type of the target object of which grabbing fails is taken as a type to be trained, a target grabbing number of the target object of the type to be trained is determined based on a sample number of existing training data, and grabbing of the target object is performed again for the target grabbing number. 6.A robot arm grasping training method, characterized by, The method comprises: obtaining a training data set, training data in the training data set being training data generated by the method in any one of claims 1 to 5; iteratively training a preset mechanical arm grabbing model according to the training data set to form a trained mechanical arm grabbing model.

7. A computer readable storage medium characterized in that, The computer-readable storage medium stores executable instructions, and the executable instructions are executed by the processor to cause the processor to execute the method in any one of claims 1 to 6.

8. A robotic arm system, comprising: comprise: a robot body; a mechanical arm; a camera configured on the robot body or the mechanical arm to capture environment images; one or more processors configured to control the mechanical arm to grab objects according to the trained mechanical arm grabbing model generated by the method in claim 6, and / or a memory configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method in any one of claims 1 to 6.

Images