Method, device, equipment, storage medium and product for performing grabbing task

Abstract

The application discloses an execution method and device of a grabbing task, equipment, a storage medium and a product. The method comprises the following steps: acquiring a multi-modal image, attribute information of an article and state information of a mechanical arm; inputting the attribute information into a mode discrimination model, determining a grabbing mode corresponding to the attribute information by using first mapping relationship information in the mode discrimination model; inputting the grabbing mode into a process parameter generation model, determining grabbing process parameters corresponding to the grabbing mode by using second mapping relationship information in the process parameter generation model; inputting the multi-modal image, the state information and the grabbing process parameters into a pose solving network model, determining target action information corresponding to the multi-modal image, the state information and the grabbing process parameters by using third mapping relationship information in the pose solving network model; and calling a driver of the mechanical arm according to the target action information to execute the grabbing task. The grabbing task in different tasks or scenes is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an execution method and device of a grasping task, equipment, a storage medium and a product. BACKGROUND

[0002] The grasping task is one of the basic capabilities that an embodied intelligent system must have. The embodied intelligent system refers to an intelligent system that perceives and acts based on a physical body (such as a robot). The grasping task can include various complex tasks, such as making a bed, delivering fruits, etc.

[0003] In the prior art, the features meaningful to the grasping task, such as the edge features and corner point features of the object, are extracted from the state data of the object based on the pre-set feature extraction rules in a specific task or scene, and then the action information of the robot arm is determined according to the extracted features, and the grasping task is executed according to the action information. In the prior art, the embodied intelligent system can only perform the grasping task in a specific task or scene, and the generalization ability is poor. SUMMARY

[0004] The embodiments of the present application provide an execution method, device, equipment, storage medium and product of a grasping task, which can perform the grasping task in different tasks or scenes and improve the generalization ability.

[0005] In a first aspect, the embodiments of the present application provide an execution method of a grasping task, comprising:

[0006] obtaining multi-modal images, attribute information of an object, and state information of a robot arm;

[0007] inputting the attribute information into a mode discrimination model, determining a grasping mode corresponding to the attribute information by using first mapping relationship information in the mode discrimination model;

[0008] inputting the grasping mode into a process parameter generation model, determining grasping process parameters corresponding to the grasping mode by using second mapping relationship information in the process parameter generation model;

[0009] inputting the multi-modal images, the state information and the grasping process parameters into a pose solving network model, determining target action information corresponding to the multi-modal images, the state information and the grasping process parameters by using third mapping relationship information in the pose solving network model;

[0010] calling a driver of the robot arm according to the target action information to execute the grasping task.

[0011] In one possible implementation, the multi-modal images are obtained, comprising:

[0012] collecting initial images by using an embodied intelligent sensor;

[0013] spatially aligning the initial image to obtain an aligned image;

[0014] extracting data of different channels in the aligned image to obtain a multi-modal image.

[0015] In one possible implementation, the attribute information includes an article category and a plurality of article sub-attribute information; and obtaining the attribute information of the article includes:

[0016] inputting the multi-modal image into an image semantic segmentation model, and determining, by using fourth mapping relationship information in the image semantic segmentation model, an article category of the article corresponding to the multi-modal image, the article category including a category of each pixel in the multi-modal image;

[0017] determining, according to the category of each pixel in the multi-modal image, an occupied pixel of the article;

[0018] modifying pixel values of pixels other than the occupied pixel in the multi-modal image to a target value to obtain a target multi-modal image;

[0019] inputting the target multi-modal image into an attribute discrimination model, and determining, by using fifth mapping relationship information in the attribute discrimination model, a plurality of article sub-attribute information of the article corresponding to the target multi-modal image.

[0020] In one possible implementation, before inputting the attribute information into the mode discrimination model and determining, by using first mapping relationship information in the mode discrimination model, a grabbing mode corresponding to the attribute information, the method further includes:

[0021] obtaining an attribute information sample and an actual grabbing mode of the attribute information sample;

[0022] inputting the attribute information sample and the actual grabbing mode of the attribute information sample into an initial mode discrimination model, and determining, by using initial first mapping relationship information, a predicted grabbing mode corresponding to the attribute information sample;

[0023] determining a first loss value of the initial mode discrimination model according to the actual grabbing mode and the predicted grabbing mode;

[0024] in a case where the first loss value does not satisfy a first preset training stop condition, adjusting parameters of the initial mode discrimination model for updating the initial first mapping relationship information, the predicted grabbing mode and the first loss value;

[0025] in a case where the updated first loss value satisfies the first preset training stop condition, obtaining the mode discrimination model.

[0026] In a possible implementation, before inputting the grasping mode into the process parameter generation model and determining the grasping process parameter corresponding to the grasping mode by using the second mapping relationship information in the process parameter generation model, the method further includes:

[0027] Obtaining the grasping mode sample and the actual grasping process parameter corresponding to the grasping mode sample;

[0028] Inputting the grasping mode sample and the actual grasping process parameter corresponding to the grasping mode sample into the initial process parameter generation model, and determining the predicted grasping process parameter corresponding to the grasping mode sample by using the initial second mapping relationship information;

[0029] Determining the second loss value of the initial process parameter generation model according to the actual grasping process parameter and the predicted grasping process parameter;

[0030] In a case where the second loss value does not satisfy a second preset training stop condition, adjusting parameters of the initial process parameter generation model, to update the initial second mapping relationship information, the predicted grasping process parameter, and the second loss value;

[0031] In a case where the updated second loss value satisfies the second preset training stop condition, obtaining the process parameter generation model.

[0032] In a possible implementation, before inputting the multi-modal image, the state information, and the grasping process parameter into the pose solving network model and determining the target action information corresponding to the multi-modal image, the state information, and the grasping process parameter by using the third mapping relationship information in the pose solving network model, the method further includes:

[0033] Obtaining the multi-modal image sample, the state information sample of the robotic arm, and the grasping process parameter sample, and the actual action information corresponding to the multi-modal image sample, the state information sample, and the grasping process parameter sample;

[0034] Inputting the multi-modal image sample, the state information sample, and the grasping process parameter sample, and the actual action information corresponding to the multi-modal image sample, the state information sample, and the grasping process parameter sample into the initial pose solving network model, and determining the predicted action information corresponding to the multi-modal image sample, the state information sample, and the grasping process parameter sample by using the initial third mapping relationship information;

[0035] Determining the third loss value of the initial pose solving network model according to the actual action information and the predicted action information;

[0036] In a case where the third loss value does not satisfy a third preset training stop condition, adjusting parameters of the initial pose solving network model, to update the initial third mapping relationship information, the predicted action information, and the third loss value;

[0037] In a case where the updated third loss value satisfies a third preset training stop condition, the pose solving network model is obtained.

[0038] In one possible implementation, before inputting the multi-modal image into the image semantic segmentation model and determining the item category of the item corresponding to the multi-modal image by using the fourth mapping relationship information in the image semantic segmentation model, the method further includes:

[0039] Obtaining a multi-modal image sample and an actual item category corresponding to the multi-modal image;

[0040] Inputting the multi-modal image sample and the actual item category corresponding to the multi-modal image sample into an initial image semantic segmentation model, and determining a predicted item category of an item sample corresponding to the multi-modal image sample by using initial fourth mapping relationship information;

[0041] Determining a fourth loss value of the initial image semantic segmentation model according to the actual item category and the predicted item category;

[0042] In a case where the fourth loss value does not satisfy a fourth preset training stop condition, adjusting parameters of the initial image semantic segmentation model for updating the initial fourth mapping relationship information, the predicted item category and the fourth loss value;

[0043] In a case where an updated fourth loss value satisfies the fourth preset training stop condition, the image semantic segmentation model is obtained.

[0044] In one possible implementation, the predicted item category includes a category of each pixel in the multi-modal image sample;

[0045] Before inputting the target multi-modal image into the attribute discrimination model and determining a plurality of item sub-attribute information of the item corresponding to the target multi-modal image by using fifth mapping relationship information in the attribute discrimination model, the method further includes:

[0046] Determining a sample occupied pixel of the item sample according to the category of each pixel in the multi-modal image sample;

[0047] Modifying pixel values of pixels other than the sample occupied pixel in the multi-modal image sample to a target value to obtain a target multi-modal image sample;

[0048] Obtaining a plurality of actual item sub-attribute information corresponding to the target multi-modal image sample;

[0049] Inputting the target multi-modal image sample and the plurality of actual item sub-attribute information corresponding to the target multi-modal image sample into an initial attribute discrimination model, and determining a plurality of predicted item sub-attribute information corresponding to the target multi-modal image sample by using initial fifth mapping relationship information;

[0050] determine a fifth loss value of the initial attribute discrimination model according to the plurality of actual item sub-attribute information and the plurality of predicted item sub-attribute information;

[0051] In a case where the fifth loss value does not satisfy a fifth preset training stop condition, adjust parameters of the initial attribute discrimination model for updating the initial fifth mapping relationship information, the plurality of predicted item sub-attribute information and the fifth loss value;

[0052] In a case where the updated fifth loss value satisfies the fifth preset training stop condition, obtain the attribute discrimination model.

[0053] In a second aspect, an embodiment of the present application provides an execution device of a grabbing task, including:

[0054] An acquisition module is configured to acquire a multi-modal image, attribute information of an item and state information of a mechanical arm;

[0055] A determination module is configured to input the attribute information into a mode discrimination model, and determine a grabbing mode corresponding to the attribute information by using first mapping relationship information in the mode discrimination model;

[0056] The determination module is further configured to input the grabbing mode into a process parameter generation model, and determine grabbing process parameters corresponding to the grabbing mode by using second mapping relationship information in the process parameter generation model;

[0057] The determination module is further configured to input the multi-modal image, the state information and the grabbing process parameters into a pose solving network model, and determine target action information corresponding to the multi-modal image, the state information and the grabbing process parameters by using third mapping relationship information in the pose solving network model;

[0058] A calling module is configured to call a driver of the mechanical arm according to the target action information, so as to execute the grabbing task.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0060] A processor and a memory having computer program instructions stored therein;

[0061] The processor implements the execution method of the grabbing task of any one of the above aspects when executing the computer program instructions.

[0062] In a fourth aspect, an embodiment of the present application provides a computer storage medium, and the computer readable storage medium has computer program instructions stored therein. The computer program instructions are executed by a processor to implement the execution method of the grabbing task of any one of the above aspects.

[0063] In a fifth aspect, an embodiment of the present application provides a computer program product. Instructions in the computer program product, when executed by a processor of an electronic device, enable the electronic device to perform the execution method of the grabbing task according to any one of the above.

[0064] The execution method, device, equipment, storage medium and product of the grabbing task provided by the embodiments of the present application obtain the multi-modal image, the attribute information of the article and the state information of the mechanical arm; input the attribute information into the mode discrimination model, determine the grabbing mode corresponding to the attribute information by using the first mapping relationship information in the mode discrimination model; input the grabbing mode into the process parameter generation model, determine the grabbing process parameter corresponding to the grabbing mode by using the second mapping relationship information in the process parameter generation model; input the multi-modal image, the state information and the grabbing process parameter into the pose solving network model, determine the target action information corresponding to the multi-modal image, the state information and the grabbing process parameter by using the third mapping relationship information in the pose solving network model; and call the driver of the mechanical arm according to the target action information, so as to execute the grabbing task.

[0065] Firstly, the grabbing task is divided into one of multiple categories by determining the grabbing mode of the article, and the grabbing process parameter corresponding to the category is determined, so as to obtain the grabbing mode of the article; and then, the action information of the mechanical arm is determined based on the multi-modal image corresponding to the article, the state information of the mechanical arm and the grabbing mode of the article, and the driver of the mechanical arm is called according to the action information, so as to execute the grabbing task, which realizes the grabbing task in different tasks or scenes and improves the generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. For those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0067] Figure 1 is a structural schematic diagram of a grabbing task execution system provided by an embodiment of the present application;

[0068] Figure 2 is a flowchart of a grabbing task execution method provided by another embodiment of the present application;

[0069] Figure 3 is a flowchart of a grabbing task execution method provided by another embodiment of the present application;

[0070] Figure 4 is a flowchart of a grabbing task execution method provided by another embodiment of the present application;

[0071] Figure 5is a structural schematic diagram of an execution device for a grabbing task provided by another embodiment of the present application.

[0072] Figure 6 is a structural schematic diagram of an electronic device provided by another embodiment of the present application. DETAILED DESCRIPTION

[0073] The features and exemplary embodiments of various aspects of the present application will be described below in detail, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. For those skilled in the art, the present application can be implemented without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0074] It should be noted that, in this paper, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0075] The grabbing task is one of the basic capabilities that embodied intelligent systems must have. Among them, the embodied intelligent system refers to an intelligent system based on physical body (such as robot) for perception and action. The grabbing task can include various complex tasks, such as making a bed, delivering fruits, etc.

[0076] In the prior art, the features meaningful to the grabbing task are extracted from the state data of the object based on the pre-set feature extraction rules in a specific task or scene, such as the edge features and corner point features of the object, and then the action information of the mechanical arm is determined according to the extracted features, and the grabbing task is executed according to the action information. In the prior art, the embodied intelligent system can only perform the grabbing task in a specific task or scene, and has poor generalization ability.

[0077] To solve the problems in the prior art, the embodiments of the present application provide an execution method, device, equipment, storage medium and product of a grabbing task. The embodiments of the present application first divide the grabbing task into a category of multiple categories of tasks by determining the grabbing mode of an article, determine the grabbing process parameters corresponding to the category of tasks, and obtain the grabbing mode of the article; then determine the action information of the mechanical arm based on the multi-modal image corresponding to the article, the state information of the mechanical arm and the grabbing mode of the article, call the driver of the mechanical arm according to the action information, and execute the grabbing task, thereby realizing the grabbing task in different tasks or scenes and improving the generalization ability.

[0078] The embodiments of the present application can be applied to an execution system of a grabbing task. As shown in Figure 1 The execution system 100 of the grabbing task can include a multi-modal input module 110, an attribute discrimination module 120, a mode discrimination module 130 and a planning control module 140. The multi-modal input module 110 is configured to obtain multi-modal images; the attribute discrimination module 120 is configured to obtain attribute information of an article; the mode discrimination module 130 is configured to determine a grabbing mode corresponding to the attribute information; and the planning control module 140 is configured to determine action information of a mechanical arm and execute a grabbing task according to the action information.

[0079] The execution method of the grabbing task provided by the embodiments of the present application will be described in detail below.

[0080] As shown in Figure 2 The execution method of the grabbing task provided by the embodiments of the present application includes the following steps S210 to S250.

[0081] S210, obtaining multi-modal images, attribute information of an article and state information of a mechanical arm.

[0082] The article refers to an article to be grabbed. The multi-modal image is an image of the article to be grabbed. The state information of the mechanical arm is the current state information of the mechanical arm.

[0083] In some embodiments, the multi-modal input module simulates the way of human cognition of the environment, and through the input of multi-modal images of 7 channels of R, G, B, X, Y, Z and T corresponding to the red, green and blue (RGB) image, the depth image and the temperature image, the attribute discrimination module can obtain more attribute information about the grabbed article, thereby improving the robustness and generalization ability of the algorithm. Wherein, R, G, B, X, Y, Z and T represent the red, green and blue of each pixel, the x-direction coordinate, the y-direction coordinate, the z-direction coordinate and the temperature value in the current scene.

[0084] In some embodiments, the attribute discrimination module extracts features and fuses features of the images of the seven channels R, G, B, X, Y, Z, and T corresponding to the RGB image, the depth image, and the temperature image to obtain attribute information of the object. The attribute information can include a vector corresponding to a multi-dimensional feature of the object.

[0085] As an example, the attribute information of the object includes a feature vector corresponding to multiple dimensions of the object category, whether it is fragile, whether it is electrified, whether it is openable, whether it is high-temperature, whether it is liquid, whether it is flexible, and whether it is invertible.

[0086] In some embodiments, the state information can include, but is not limited to, joint angles, angular velocities, position coordinates, torque velocities, accelerations, and rotation amounts.

[0087] S220, input the attribute information into the mode discrimination model, and determine the grasping mode corresponding to the attribute information by using the first mapping relationship information in the mode discrimination model.

[0088] The mode discrimination model is pre-trained. The first mapping relationship information includes the relationship information between the preset attribute information and the preset grasping mode. The preset attribute information includes the attribute information of the object to be grasped, and the preset grasping mode includes the grasping mode of the object to be grasped.

[0089] In some embodiments, the attribute information of the object to be grasped is input into the mode discrimination model in the mode discrimination module. The mode discrimination module discriminates the grasping mode to be used for the object to be grasped according to the attribute information of the object obtained by the attribute discrimination module. The grasping mode can include, but is not limited to, a fragile rigid body mode, a fragile flexible body mode, a flexible body mode, a liquid mode, a rigid body combination mode, a liquid-rigid body combination mode, a dangerous object mode, and a normal mode. Each grasping mode corresponds to a different action state space. The action state space refers to a space composed of the execution state of the robot arm and the action. After determining the grasping mode, the planning control module uses the target action information corresponding to the grasping mode to control the grasping motion.

[0090] As an example, taking the liquid-rigid body combination mode as an example, the corresponding object to be grasped in this mode is generally an open cup or bowl containing liquid. For this type of grasping task, the object to be grasped is not allowed to be inverted during the grasping process. Therefore, the reward function of the planning control module can be set to fail when the object to be grasped is in an inverted state. In the liquid-rigid body combination mode, the reward value of the object to be grasped in the inverted state is 0, i.e., the reward value corresponding to the failure is the smallest.

[0091] The mode discrimination model can use a Transformer network structure. During training of the Transformer network structure, labeled data is used for training. For example, the object to be grasped is tofu: one piece of labeled training data is: the input attribute information includes tofu, fragile, not charged, not openable, normal temperature, not liquid, soft body, and can be inverted, and the output is fragile and soft body.

[0092] S230, input the grasping mode into the process parameter generation model, and determine the grasping process parameter corresponding to the grasping mode by using the second mapping relationship information in the process parameter generation model.

[0093] The process parameter generation model is pre-trained. The second mapping relationship information includes the relationship information between the preset grasping mode and the preset grasping process parameter. The preset grasping process parameter includes the grasping process parameter of the object to be grasped.

[0094] In some embodiments, the process parameter generation model is a front-end network model of the planning control module. The grasping mode of the object to be grasped is input into the process parameter generation model of the planning control module, and the grasping process parameter corresponding to the grasping mode of the object to be grasped is determined by using the second mapping relationship information in the process parameter generation model. The grasping process parameter can include but is not limited to the grasping mode, the grasping position, and the grasping force.

[0095] The grasping mode includes single-hand grasping and double-hand grasping, the grasping position refers to the position of the grasping point on the part of the object to be grasped, and the grasping force refers to the setting of the initial force and the step of increasing or decreasing during the grasping process.

[0096] In some embodiments, the grasping process parameter of the object to be grasped can be supervised. After determining the grasping process parameter corresponding to the grasping mode by using the second mapping relationship information in the process parameter generation model, the grasping process parameter is displayed for manual confirmation or machine automatic confirmation to confirm whether the grasping process parameter of the object to be grasped is reasonable. The specific process of machine automatic confirmation is not limited here. For example, machine automatic confirmation needs to set a large number of reasonable samples, compare the grasping process parameter samples in the reasonable samples with the determined grasping process parameter corresponding to the grasping mode, and confirm whether the grasping process parameter is reasonable.

[0097] S240, input the multi-modal image, the state information, and the grasping process parameter into the pose solving network model, and determine the target action information corresponding to the multi-modal image, the state information, and the grasping process parameter by using the third mapping relationship information in the pose solving network model.

[0098] The pose solving network model is pre-trained. The third mapping relationship information includes relationship information of the preset multi-modal image, preset state information, preset grabbing process parameter and preset action information. The preset multi-modal image includes a multi-modal image of the to-be-grabbed object, the preset state information includes current state information of the robot arm, and the preset action information includes target action information. The preset action information is action information of the robot arm.

[0099] In some embodiments, the multi-modal image, the state information and the grabbing process parameter are input into the pose solving network model of the planning control module, and the third mapping relationship information in the pose solving network model is used to determine the target action information corresponding to the multi-modal image, the state information and the grabbing process parameter.

[0100] The target action information may include, but is not limited to, a rotation amount, a rotation torque and a rotation direction of each joint of the robot arm at a time step in the grabbing process.

[0101] S250, calling a driver of the robot arm according to the target action information to execute the grabbing task.

[0102] In some embodiments, the planning control module generates a control instruction according to the target action information, and controls the execution unit to call the driver of the robot arm based on the control instruction to realize motor movement, so as to execute the grabbing task.

[0103] The embodiments of the present application first divide the grabbing task into a type of task in multiple types of tasks by determining the grabbing mode of the object, determine the grabbing process parameter corresponding to the type of task, and obtain the grabbing mode of the object. Then, based on the multi-modal image corresponding to the object, the state information of the robot arm and the grabbing mode of the object, the action information of the robot arm is determined, and the driver of the robot arm is called according to the action information to execute the grabbing task. The grabbing task in different tasks or scenes is realized, and the generalization ability is improved.

[0104] It can be understood that the embodiments of the present application introduce the grabbing mode, reduce the dimension of the solving process of the target action information with the grabbing mode, improve the convergence of the solving, make the solving more robust, and thus improve the generalization ability, so that the final grabbing mode is more in line with the imagination and requirements of the implementer.

[0105] Based on this, in some embodiments, as shown in FIG. 2, Figure 3 In the above S210, the multi-modal image is obtained, including S211 to S213.

[0106] S211, collecting an initial image by using an embodied intelligent sensor.

[0107] The embodied intelligent sensor is a sensor of an embodied intelligent system.

[0108] In some embodiments, the sensor of the embodied intelligent system is aimed at the to-be-grabbed object and collects an initial image by using the sensor. The sensor of the embodied intelligent system at least includes a Red Green Blue (RGB) camera, a laser radar or an infrared camera, and an infrared temperature measurement camera, and the collected initial image includes an RGB image, a depth image and a temperature image.

[0109] In some embodiments, the multi-modal input module can be used as an input module to obtain the RGB image collected by the RGB camera, the depth image collected by the depth camera or the laser radar, and the temperature image collected by the infrared temperature measurement camera.

[0110] In some embodiments, the depth image can also be replaced by a point cloud image. The depth image includes depth information, and the point cloud image includes point cloud information. The point cloud information can be obtained through the depth information of the depth image. The manner of obtaining the point cloud information is not specifically limited here.

[0111] S212, spatially aligning the initial image to obtain an aligned image.

[0112] The spatial alignment is a coordinate transformation and projection using a sensor conversion relationship, which aligns both the point cloud image and the temperature image to the RGB coordinate system and resolution. The coordinate transformation and projection are both described using the rules of the Cartesian coordinate system.

[0113] S213, extracting data of different channels in the aligned image to obtain a multi-modal image.

[0114] In some embodiments, the channels include R, G, B, X, Y, Z and T. The multi-modal input module is used to extract data of different channels in the aligned image to obtain a multi-modal image, which includes images of the R, G, B, X, Y, Z and T channels.

[0115] The embodiments of the present application extract data of different channels after spatially aligning the initial image collected by the sensor to obtain a multi-modal image, i.e., extract multiple information in the initial image, which can comprehensively utilize the advantages of various information to make the determination of target action information more accurate.

[0116] Based on this, in some embodiments, the attribute information includes an object category and a plurality of object sub-attribute information; as shown in Figure 4 In the above S210, the attribute information of the object is obtained, including S214 to S217.

[0117] S214, inputting the multi-modal image into an image semantic segmentation model, and determining an object category of the object corresponding to the multi-modal image by using fourth mapping relationship information in the image semantic segmentation model. The object category includes the category of each pixel in the multi-modal image.

[0118] The image semantic segmentation model is pre-trained. The fourth mapping relationship information includes relationship information of the preset multi-modal image and a preset article category. The preset multi-modal image includes a multi-modal image corresponding to the article to be grabbed, and the preset article category includes an article category corresponding to the article to be grabbed.

[0119] In some embodiments, the multi-modal input module is used to input the multi-modal image into the image semantic segmentation model of the attribute discrimination module, and the fourth mapping relationship information in the image semantic segmentation model is used to determine an article category of an article corresponding to the multi-modal image. The article category includes a category of each pixel in the multi-modal image.

[0120] S215, determining an occupied pixel of the article according to the category of each pixel in the multi-modal image.

[0121] The occupied pixel is a pixel occupied by the article to be grabbed.

[0122] S216, modifying pixel values of pixels other than the occupied pixel in the multi-modal image to a target value to obtain a target multi-modal image.

[0123] The target value is pre-set.

[0124] As an example, the target value is 0. The pixel values of the pixels other than the pixel occupied by the article to be grabbed in the target multi-modal image are all 0.

[0125] S217, inputting the target multi-modal image into the attribute discrimination model, and using the fifth mapping relationship information in the attribute discrimination model to determine a plurality of article sub-attribute information of the article corresponding to the target multi-modal image.

[0126] The attribute discrimination model is pre-trained. The fifth mapping relationship information includes relationship information of a preset target multi-modal image and a preset article sub-attribute information. The preset target multi-modal image includes a target multi-modal image of the article to be grabbed, and the preset article sub-attribute information includes a plurality of article sub-attribute information of the article to be grabbed. The preset article sub-attribute information includes a feature vector corresponding to a plurality of dimensional features such as whether it is fragile, whether it is connected to electricity, whether it can be opened, whether it is high temperature, whether it is liquid, whether it is flexible, and whether it can be inverted.

[0127] In some embodiments, the target multi-modal image is input into the attribute discrimination model of the attribute discrimination module, and the fifth mapping relationship information in the attribute discrimination model is used to determine a plurality of article sub-attribute information of the article corresponding to the target multi-modal image.

[0128] As an example, taking a "water cup" as an example, the multiple item sub-attribute information is not fragile, not electrically connected, not openable, not high temperature, not liquid, not flexible, and not invertible.

[0129] In some embodiments, the attribute discrimination model of the attribute discrimination module can perform feature extraction on the images of the seven channels of R, G, B, X, Y, Z, and T. The obtained item sub-attribute information already contains multiple item sub-attribute information. The attribute discrimination model of the attribute discrimination module can also send the images of the seven channels of R, G, B, X, Y, Z, and T into respective attribute discrimination models for feature extraction, and then fuse the obtained respective item sub-attribute information. The fusion methods include splicing and weighting. The attribute discrimination model can use a recurrent neural network (RNN) model, a convolutional neural network (CNN) model, or a Transformer network architecture.

[0130] The embodiments of the present application determine the multi-dimensional attribute information of the to-be-grabbed item, so that the embodied intelligent system can divide the grabbing task into several typical categories through attribute discrimination and mode discrimination when facing different environments and different to-be-grabbed items, provide more prior information for solving the grabbing process parameters, improve the accuracy of the grabbing mode discrimination, and further improve the accuracy of the target action information determination.

[0131] Based on this, in some embodiments, before S220, the method can further include:

[0132] obtaining attribute information samples and actual grabbing modes of the attribute information samples;

[0133] inputting the attribute information samples and the actual grabbing modes of the attribute information samples into an initial mode discrimination model, and determining predicted grabbing modes corresponding to the attribute information samples by using initial first mapping relationship information;

[0134] determining a first loss value of the initial mode discrimination model according to the actual grabbing modes and the predicted grabbing modes;

[0135] in a case where the first loss value does not satisfy a first preset training stop condition, adjusting parameters of the initial mode discrimination model for updating the initial first mapping relationship information, the predicted grabbing modes, and the first loss value;

[0136] in a case where the updated first loss value satisfies the first preset training stop condition, obtaining the mode discrimination model.

[0137] The initial mode discrimination model can be a Transformer network model. The first preset training stop condition is set in advance.

[0138] In some embodiments, the first preset training stop condition can include that the first loss value converges to a minimum.

[0139] In some embodiments, the actual grasping mode is manually labeled. The actual grasping mode can include, but is not limited to, a fragile rigid body mode, a fragile flexible body mode, a flexible body mode, a liquid mode, a rigid body combination mode, a liquid-rigid body combination mode, a dangerous object mode, and a normal mode.

[0140] As an example, taking a "water cup" as an example, the attribute information is: water cup, not fragile, not connected to electricity, not openable, not high temperature, not liquid, not flexible, and not invertible; and the corresponding actual grasping mode is a liquid-rigid body combination mode.

[0141] In some embodiments, the mode discrimination model includes an encoder network and a decoder network. The encoder network is composed of 6 identical layers, each layer including two sub-layers: a multi-head self-attention layer and a position-wise feed-forward neural network layer; after each sub-layer, a residual connection layer and a normalization layer are used to operate on the output result; the decoder network is composed of 6 identical layers, each layer including three sub-layers: a masked self-attention layer, an Encoder-Decoder attention layer, and a position-wise feed-forward neural network layer. Each sub-layer has a residual connection layer and a normalization layer to operate on the output result.

[0142] The embodiments of the present application reduce the dimension of the solution space by training the mode discrimination model separately, which not only improves the accuracy of the calculation result, but also saves the calculation resources.

[0143] Based on this, in some embodiments, before S230, the method can further include:

[0144] Obtaining a grasping mode sample and an actual grasping process parameter corresponding to the grasping mode sample;

[0145] Inputting the grasping mode sample and the actual grasping process parameter corresponding to the grasping mode sample into the initial process parameter generation model, and determining a predicted grasping process parameter corresponding to the grasping mode sample by using the initial second mapping relationship information;

[0146] Determining a second loss value of the initial process parameter generation model according to the actual grasping process parameter and the predicted grasping process parameter;

[0147] In a case where the second loss value does not satisfy the second preset training stop condition, the initial process parameter generation model is adjusted to generate parameters for updating the initial second mapping relationship information, the predicted grasping process parameter, and the second loss value.

[0148] In a case where the updated second loss value satisfies the second preset training stop condition, the process parameter generation model is obtained.

[0149] The initial process parameter generation model can be a Transformer network model. The second preset training stop condition is set in advance.

[0150] In some embodiments, the second preset training stop condition can include that the second loss value converges to a minimum.

[0151] In some embodiments, the second preset training stop condition can also include that a reward value exceeds a preset threshold. The actual grasping process parameter is used as part of a reward function, and the closer the predicted grasping process parameter is to the actual grasping process parameter, the higher the reward value of the reward function. In a case where the reward value exceeds the preset threshold, the process parameter generation model is obtained.

[0152] In some embodiments, the actual grasping process parameter is manually labeled. The actual grasping process parameter can include, but is not limited to, a grasping mode, a grasping position, and a grasping force.

[0153] As an example, taking a fragile flexible object as an example, a piece of labeled training data is: the input is a fragile flexible object, and the corresponding actual grasping process parameter is a grasping mode of two-hand grasping, a grasping position of bottom grasping, and a grasping force of gradually increasing from a minimum force until the object moves.

[0154] In some embodiments, the process parameter generation model includes an encoder network and a decoder network. The encoder network is composed of 6 identical layers, each layer including two sub-layers: a multi-head self-attention layer and a position-wise feed-forward neural network layer; after each sub-layer, a residual connection layer and a normalization layer are used to operate on the output result; the decoder network is composed of 6 identical layers, each layer including three sub-layers: a masking self-attention layer, an Encoder-Decoder attention layer, and a position-wise feed-forward neural network layer. Each sub-layer has a residual connection layer and a normalization layer to operate on the output result.

[0155] During the model training process, the input is text information of the grabbing mode, and the grabbing mode specifically includes a fragile rigid body mode, a fragile flexible body mode, a flexible body mode, a liquid mode, a rigid body combination mode, a liquid rigid body combination mode, a dangerous object mode, and a normal mode. The output is a grabbing process parameter, including a grabbing mode, a grabbing position, and a grabbing force. These parameters are described in a text manner. The grabbing mode includes single-hand grabbing and double-hand grabbing. The grabbing position refers to the position of the grabbing point on which part of the grabbed object. The grabbing force refers to the initial force setting and the step length of increase or decrease in the grabbing process.

[0156] The embodiment of the application reduces the dimension of the solution space by separately training the process parameter generation model, thereby improving the accuracy of the calculation result and saving the calculation resource.

[0157] Based on this, in some embodiments, before S240, the method can further include:

[0158] obtaining a multi-modal image sample, a state information sample of a mechanical arm, and a grabbing process parameter sample, and actual action information corresponding to the multi-modal image sample, the state information sample, and the grabbing process parameter sample;

[0159] inputting the multi-modal image sample, the state information sample, and the grabbing process parameter sample, and the actual action information corresponding to the multi-modal image sample, the state information sample, and the grabbing process parameter sample into an initial pose solving network model, determining predicted action information corresponding to the multi-modal image sample, the state information sample, and the grabbing process parameter sample by using initial third mapping relationship information;

[0160] determining a third loss value of the initial pose solving network model according to the actual action information and the predicted action information;

[0161] in a case where the third loss value does not satisfy a third preset training stop condition, adjusting a parameter of the initial pose solving network model, to be used for updating the initial third mapping relationship information, the predicted action information, and the third loss value;

[0162] in a case where the updated third loss value satisfies the third preset training stop condition, obtaining a pose solving network model.

[0163] The initial pose solving network model can be a deep Q network (Deep Q – Network, DQN) model. The third preset training stop condition is set in advance.

[0164] In some embodiments, the third preset training stop condition can include that the third loss value converges to a minimum.

[0165] In some embodiments, the pose solving network model mainly comprises a Q network, a target network, and an experience replay component. The Q network is an agent trained to generate optimal state-action values. The experience replay unit stores training data samples collected by interacting with the environment to train the Q network. The target network is identical to the Q network at the beginning.

[0166] Specifically, each storage unit of the experience replay component contains a four-tuple: (S, A, R, S’), where S is the state information of the current state (including a 7-channel image, the grasping process parameters, and the current state information of the robot arm), A is the action taken (the rotation amount, the rotation torque, and the rotation direction of each joint), R is the reward value obtained, and S’ is the state information of the next state after the action is executed. The state information can include, but is not limited to, joint angle, angular velocity, position coordinates, torque velocity, acceleration, and rotation amount, etc.

[0167] First, an initial DQN network architecture is built, which includes an input layer, several hidden layers, and an output layer. The number of neurons in the input layer is determined according to the dimension of the input data (i.e., the dimension of the concatenated 7-channel data, process parameters, and current state information of the robot arm); the hidden layers can be composed of fully connected layers, using appropriate activation functions (such as ReLU: f ( x )=max(0, x ) to increase the non-linear representation ability of the network; the number of neurons in the output layer is determined according to the output requirements, i.e., the sum of the number of neurons corresponding to the rotation amount, rotation torque, and rotation direction of each joint. For example, assuming that the robot arm has n joints, the rotation amount and rotation torque each have n outputs, and the rotation direction is divided into m cases, then the rotation direction output is n × m , so the total number of output layer neurons is 2 n + n × m .

[0168] The weights and biases of the initial DQN network architecture are randomly initialized. A target network identical to the Q network in the initial DQN network structure is built, and the initial parameters of the Q network are copied to the target network. The target network is used to calculate the target Q value to stabilize the training process.

[0169] Then, a batch of training data samples are randomly extracted from the experience replay component, which can break the temporal correlation between data and improve the stability and efficiency of training.

[0170] extracted current state information S is input into the DQN network, and the Q values of taking each possible action in the current state are calculated through forward propagation, i.e. Q S , a θ , where a represents all possible actions, θ is a parameter of the DQN network. For the next state S' in the extracted sample, it is input into the target network, and the Q values of taking all possible actions in the S' state are calculated Q ′( S ’, a θ ′), where θ ′ is a parameter of the target network. Then the action with the maximum Q value in each next state is selected. The target Q value is calculated (such as the Bellman equation). The difference between the Q value predicted by the DQN network and the target Q value is measured using a loss function such as Mean Squared Error (MSE). Every certain training step, the parameters of the DQN network are copied to the target network, i.e. θ ′← θ , to keep the target network parameters updated in time while providing a relatively stable target for training.

[0171] Finally, the extraction of the training data sample and the training process are repeated until the difference between the output results of the Q network and the target network converges to a minimum, and the pose solving network model is obtained.

[0172] Embodiments of the present application train the pose solving network model separately, reduce the dimension of the solution space, improve the accuracy of the calculation results, and save computing resources.

[0173] Based on this, in some embodiments, before S214, the method can further include:

[0174] obtaining a multi-modal image sample and an actual article category corresponding to the multi-modal image;

[0175] inputting the multi-modal image sample and the actual article category corresponding to the multi-modal image sample into an initial image semantic segmentation model, determining a predicted article category of an article sample corresponding to the multi-modal image sample using the initial fourth mapping relationship information;

[0176] determining a fourth loss value of the initial image semantic segmentation model according to the actual article category and the predicted article category;

[0177] ​​​In a case where the fourth loss value does not satisfy the fourth preset training stop condition, parameters of the initial image semantic segmentation model are adjusted for updating the initial fourth mapping relationship information, the predicted object category, and the fourth loss value.

[0178] In a case where the updated fourth loss value satisfies the fourth preset training stop condition, the image semantic segmentation model is obtained.

[0179] The initial image semantic segmentation model is a SegNet network model, which is a convolutional neural network model used for semantic segmentation tasks. The fourth preset training stop condition is set in advance. The actual object category is manually labeled.

[0180] In some embodiments, the fourth preset training stop condition can include that the fourth loss value converges to a minimum.

[0181] In some embodiments, the multi-modal image sample includes an image with 7 channels of R, G, B, X, Y, Z, and T. The multi-modal image sample is subjected to feature extraction and semantic segmentation by using the initial image semantic segmentation model to obtain semantic information of the object to be grasped. The semantic information represents the object category. For example, the semantic information is “water cup”.

[0182] Specifically, the image semantic segmentation model includes an encoder network and a decoder network. The encoder network part is responsible for extracting features of an image and generating a feature map, which is composed of 13 convolutional layers; the decoder network part is mainly responsible for restoring the feature map to the same spatial resolution as the original input image and generating a segmentation result. A single decoder network component is as follows:

[0183] An up-sampling layer is used to restore the feature map to a higher spatial resolution;

[0184] A filter bank is composed of multiple convolutional layers that extract and integrate features of the feature map after the up-sampling layer;

[0185] A BN layer performs batch normalization operation;

[0186] A nonlinear layer adopts a ReLU activation function that performs nonlinear operation;

[0187] A softmax layer (softmax classifier) is a classifier for k classes, which is used to predict the class of each pixel. It is in the last layer of the decoder and will output the predicted object category.

[0188] As an example, a cross-entropy loss function is used to determine the fourth loss value of the initial image semantic segmentation model. The cross-entropy loss function is shown in formula (1):

[0189] (1)

[0190] where N is the number of samples, and M is the number of classes; y ic is an indicator function (0 or 1) that takes the value 1 if the true class of sample i is equal to c and 0 otherwise; p ic is the predicted probability that sample i belongs to class c.

[0191] Embodiments of the present application reduce the dimension of the solution space by training the image semantic segmentation model separately, which not only improves the accuracy of the calculation result, but also saves computing resources.

[0192] Based on this, in some embodiments, the predicted item class includes the class of each pixel in the multi-modal image sample;

[0193] Before S217, the method can further include:

[0194] According to the class of each pixel in the multi-modal image sample, a sample-occupied pixel of the item sample is determined;

[0195] The pixel values of the pixels in the multi-modal image sample, except for the sample-occupied pixel, are modified to a target value to obtain a target multi-modal image sample;

[0196] A plurality of actual item sub-attribute information corresponding to the target multi-modal image sample is obtained;

[0197] The target multi-modal image sample and the plurality of actual item sub-attribute information corresponding to the target multi-modal image sample are input into the initial attribute discrimination model, and the initial fifth mapping relationship information is used to determine a plurality of predicted item sub-attribute information corresponding to the target multi-modal image sample;

[0198] According to the plurality of actual item sub-attribute information and the plurality of predicted item sub-attribute information, a fifth loss value of the initial attribute discrimination model is determined;

[0199] In a case where the fifth loss value does not satisfy a fifth preset training stop condition, the parameters of the initial attribute discrimination model are adjusted for updating the initial fifth mapping relationship information, the plurality of predicted item sub-attribute information, and the fifth loss value;

[0200] In a case where the updated fifth loss value satisfies the fifth preset training stop condition, an attribute discrimination model is obtained.

[0201] The initial attribute discrimination model includes a CNN model. The fifth preset training stop condition is set in advance.

[0202] In some embodiments, the fifth preset training stop condition can include that the fifth loss value converges to a minimum.

[0203] In some embodiments, among the 7-channel multi-modal images of R, G, B, X, Y, Z, T, parts other than the pixels occupied by the to-be-grabbed object are set to 0, and then input to the attribute discrimination model for feature extraction, and then the feature values are classified to obtain multiple predicted object sub-attribute information.

[0204] The attribute discrimination model includes a convolutional layer network, a pooling layer, and a fully connected layer. The fully connected layer uses softmax as an activation function.

[0205] The embodiments of the present application reduce the dimension of the solution space by training the attribute discrimination model separately, which not only improves the accuracy of the calculation result, but also saves computing resources.

[0206] In the embodiments provided in the present application, the robot can understand the grabbing process of different objects as a person does. When a person performs a grabbing task, the person will fully consider the shape, temperature, and fragility of the to-be-grabbed object, and then decide the grabbing method, such as whether to grab with one hand or two hands, how much force to use to grab, and where to grab, and then adjust according to the feedback in the actual grabbing process. The embodiments provided in the present application abstract each step and connect them in series, so as to obtain a grabbing process and a grabbing method that conform to human thinking. The embodiments of the present application provide a unified algorithm framework for grabbing tasks in different environments and objects, and provide a general and efficient grabbing algorithm framework for different types of objects, which greatly improves the robustness and convergence of the algorithm. It can be understood that, in the training process, the mode discrimination module can effectively limit the dimension of the solution space under a certain type of grabbing task, thereby improving the training efficiency and convergence of the training.

[0207] The embodiments provided in the present application introduce the attribute discrimination module and the mode discrimination module. Such a design has two advantages. First, the entire process can be effectively supervised. If the attribute discrimination and the grabbing mode discrimination of an object are wrong, the grabbing process parameter calculation of the object is likely to be wrong. Through supervision of the process, the occurrence of incorrect execution can be effectively reduced. Second, through attribute discrimination and grabbing mode discrimination, the dimensionality of the solution space of the grabbing process parameters is reduced, and the convergence and robustness of the solving process are improved.

[0208] In the solving process of the grabbing process parameters, the embodiments provided in the present application fully consider the attribute information of the grabbed object, and then design effective training parameters, so that the grabbing process parameter solving converges quickly and is more in line with the original intention of the designer in designing the grabbing method.

[0209] Based on the execution method of the grabbing task provided in the above embodiments, the present application also provides a specific implementation manner of an execution device of the grabbing task. Please refer to the following embodiments.

[0210] With reference to Figure 5 The execution device 300 for a grabbing task provided in the embodiments of the present application comprises:

[0211] The acquisition module 310 is configured to acquire a multi-modal image, attribute information of an article, and state information of a mechanical arm.

[0212] The determination module 320 is configured to input the attribute information into a mode discrimination model, and determine a grabbing mode corresponding to the attribute information by using first mapping relationship information in the mode discrimination model.

[0213] The determination module 320 is further configured to input the grabbing mode into a process parameter generation model, and determine grabbing process parameters corresponding to the grabbing mode by using second mapping relationship information in the process parameter generation model.

[0214] The determination module 320 is further configured to input the multi-modal image, the state information, and the grabbing process parameters into a pose solving network model, and determine target action information corresponding to the multi-modal image, the state information, and the grabbing process parameters by using third mapping relationship information in the pose solving network model.

[0215] The calling module 330 is configured to call a driver of the mechanical arm according to the target action information, so as to execute the grabbing task.

[0216] Based on this, in some embodiments, the acquisition module 310 can be specifically configured to:

[0217] acquire an initial image by using an embodied intelligent sensor;

[0218] perform spatial alignment on the initial image to obtain an aligned image;

[0219] extract data in different channels of the aligned image to obtain the multi-modal image.

[0220] Based on this, in some embodiments, the attribute information comprises an article category and a plurality of article sub-attribute information; and the acquisition module 310 can be specifically configured to:

[0221] input the multi-modal image into an image semantic segmentation model, and determine an article category corresponding to the multi-modal image by using fourth mapping relationship information in the image semantic segmentation model, wherein the article category comprises a category of each pixel in the multi-modal image;

[0222] determine an occupied pixel of the article according to the category of each pixel in the multi-modal image;

[0223] modify pixel values of pixels other than the occupied pixel in the multi-modal image to a target value to obtain a target multi-modal image;

[0224] The target multi-modal image is input into the attribute discrimination model, and the fifth mapping relationship information in the attribute discrimination model is used to determine the multiple item sub-attribute information of the item corresponding to the target multi-modal image.

[0225] Based on this, in some embodiments, the apparatus 300 can further include:

[0226] The acquisition module 310 is further configured to, before inputting the attribute information into the mode discrimination model and using the first mapping relationship information in the mode discrimination model to determine the grabbing mode corresponding to the attribute information, acquire the attribute information sample and the actual grabbing mode of the attribute information sample.

[0227] The determination module 320 is further configured to input the attribute information sample and the actual grabbing mode of the attribute information sample into the initial mode discrimination model, and use the initial first mapping relationship information to determine the predicted grabbing mode corresponding to the attribute information sample.

[0228] The determination module 320 is further configured to determine the first loss value of the initial mode discrimination model according to the actual grabbing mode and the predicted grabbing mode.

[0229] The adjustment module is configured to, in the case that the first loss value does not satisfy the first preset training stop condition, adjust the parameters of the initial mode discrimination model for updating the initial first mapping relationship information, the predicted grabbing mode and the first loss value.

[0230] The determination module 320 is further configured to, in the case that the updated first loss value satisfies the first preset training stop condition, obtain the mode discrimination model.

[0231] Based on this, in some embodiments, the apparatus 300 can further include:

[0232] The acquisition module 310 is further configured to, before inputting the grabbing mode into the process parameter generation model and using the second mapping relationship information in the process parameter generation model to determine the grabbing process parameter corresponding to the grabbing mode, acquire the grabbing mode sample and the actual grabbing process parameter corresponding to the grabbing mode sample.

[0233] The determination module 320 is further configured to input the grabbing mode sample and the actual grabbing process parameter corresponding to the grabbing mode sample into the initial process parameter generation model, and use the initial second mapping relationship information to determine the predicted grabbing process parameter corresponding to the grabbing mode sample.

[0234] The determination module 320 is further configured to determine the second loss value of the initial process parameter generation model according to the actual grabbing process parameter and the predicted grabbing process parameter.

[0235] The adjusting module is configured to adjust the parameters of the initial process parameter generation model for updating the initial second mapping relationship information, the predicted grasping process parameter, and the second loss value, in a case where the second loss value does not satisfy the second preset training stop condition.

[0236] The determining module 320 is further configured to obtain the process parameter generation model in a case where the updated second loss value satisfies the second preset training stop condition.

[0237] Based on this, in some embodiments, the apparatus 300 can further include:

[0238] The obtaining module 310 is further configured to obtain a multi-modal image sample, a state information sample of the robot arm, and a grasping process parameter sample, and actual action information corresponding to the multi-modal image sample, the state information sample, and the grasping process parameter sample, before inputting the multi-modal image, the state information, and the grasping process parameter into the pose solving network model and determining the target action information corresponding to the multi-modal image, the state information, and the grasping process parameter by using the third mapping relationship information in the pose solving network model.

[0239] The determining module 320 is further configured to input the multi-modal image sample, the state information sample, and the grasping process parameter sample, and the actual action information corresponding to the multi-modal image sample, the state information sample, and the grasping process parameter sample into the initial pose solving network model, and determine predicted action information corresponding to the multi-modal image sample, the state information sample, and the grasping process parameter sample by using the initial third mapping relationship information.

[0240] The determining module 320 is further configured to determine a third loss value of the initial pose solving network model according to the actual action information and the predicted action information.

[0241] The adjusting module is configured to adjust the parameters of the initial pose solving network model for updating the initial third mapping relationship information, the predicted action information, and the third loss value, in a case where the third loss value does not satisfy a third preset training stop condition.

[0242] The determining module 320 is further configured to obtain the pose solving network model in a case where the updated third loss value satisfies the third preset training stop condition.

[0243] Based on this, in some embodiments, the apparatus 300 can further include:

[0244] The obtaining module 310 is further configured to obtain a multi-modal image sample and actual object categories corresponding to the multi-modal image, before inputting the multi-modal image into the image semantic segmentation model and determining object categories of an object corresponding to the multi-modal image by using fourth mapping relationship information in the image semantic segmentation model.

[0245] The determining module 320 is further configured to input the multi-modal image sample and the actual article category corresponding to the multi-modal image sample into the initial image semantic segmentation model, and determine a predicted article category of the article sample corresponding to the multi-modal image sample by using the initial fourth mapping relationship information.

[0246] The determining module 320 is further configured to determine a fourth loss value of the initial image semantic segmentation model according to the actual article category and the predicted article category.

[0247] The adjusting module is configured to adjust parameters of the initial image semantic segmentation model for updating the initial fourth mapping relationship information, the predicted article category and the fourth loss value, in a case where the fourth loss value does not satisfy a fourth preset training stop condition.

[0248] The determining module 320 is further configured to obtain the image semantic segmentation model in a case where the updated fourth loss value satisfies the fourth preset training stop condition.

[0249] Based on this, in some embodiments, the predicted article category includes a category of each pixel in the multi-modal image sample; the apparatus 300 can further include:

[0250] The determining module 320 is further configured to, before determining the plurality of article sub-attribute information of the article corresponding to the target multi-modal image by using the fifth mapping relationship information in the attribute discrimination model, determine a sample occupied pixel of the article sample according to the category of each pixel in the multi-modal image sample.

[0251] The modifying module is configured to modify pixel values of pixels in the multi-modal image sample other than the sample occupied pixel to a target value, to obtain a target multi-modal image sample.

[0252] The obtaining module 310 is further configured to obtain a plurality of actual article sub-attribute information corresponding to the target multi-modal image sample.

[0253] The determining module 320 is further configured to input the target multi-modal image sample and the plurality of actual article sub-attribute information corresponding to the target multi-modal image sample into the initial attribute discrimination model, and determine a plurality of predicted article sub-attribute information corresponding to the target multi-modal image sample by using the initial fifth mapping relationship information.

[0254] The determining module 320 is further configured to determine a fifth loss value of the initial attribute discrimination model according to the plurality of actual article sub-attribute information and the plurality of predicted article sub-attribute information.

[0255] The adjusting module is configured to adjust parameters of the initial attribute discrimination model for updating the initial fifth mapping relationship information, the plurality of predicted article sub-attribute information and the fifth loss value, in a case where the fifth loss value does not satisfy a fifth preset training stop condition.

[0256] The determining module 320 is further configured to obtain the attribute discrimination model when the updated fifth loss value satisfies a fifth preset training stop condition.

[0257] The various modules of the execution device of the crawling task provided in the embodiments of the present application can realize the functions of the various steps of the execution method of the crawling task provided above and achieve the corresponding technical effects. For brevity, the details are not described here.

[0258] Based on the same inventive concept, the embodiments of the present application also provide an electronic device.

[0259] Figure 6 A hardware structure schematic diagram of the electronic device provided in the embodiments of the present application is shown.

[0260] The electronic device can include a processor 401 and a memory 402 storing computer program instructions.

[0261] Specifically, the processor 401 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0262] The memory 402 can include a mass storage for data or instructions. By way of example and not limitation, the memory 402 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. The memory 402 can include removable or non-removable (or fixed) media, where appropriate. The memory 402 can be integral to the integrated gateway disaster recovery device, or external to the integrated gateway disaster recovery device, where appropriate. In particular embodiments, the memory 402 is non-volatile, solid-state memory.

[0263] The memory can include read only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions that, when executed (e.g., by one or more processors), are operable to perform the operations described with reference to the methods according to an aspect of the present disclosure.

[0264] The processor 401 implements the execution method of any one of the above-described embodiments of the grabbing task by reading and executing the computer program instructions stored in the memory 402.

[0265] In one example, the electronic device can further include a communication interface 403 and a bus 410. Wherein, as shown in the figure, the processor 401, the memory 402, the communication interface 403 are connected through the bus 410 and complete the communication between each other. Figure 6

[0266] The communication interface 403 is mainly used to realize the communication between each module, device, unit and / or equipment in the embodiments of the present application.

[0267] ​Bus 410 includes a hardware, software, or both, that couples electronic devices to each other. By way of example, and not limitation, the bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand™ interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association local (VESA) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 410 can include one or more buses. Although the example describes and shows a particular bus, this application contemplates any suitable bus or interconnect. The electronic device can perform the execution method of the grasping task in the embodiments of the application, thereby realizing the execution method of the grasping task described above.

[0268] In addition, in combination with the execution method of the grasping task in the above embodiments, the embodiments of the application can provide a computer storage medium to realize. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by the processor to realize any one of the execution methods of the grasping task in the above embodiments.

[0269] The application also provides a computer program product, the instructions in the computer program product are executed by the processor of the electronic device, so that the electronic device executes each process of the execution method embodiment of any one of the above grasping tasks.

[0270] It is to be understood that the application is not limited to particular configurations and processes described herein and shown in the drawings. For simplicity, detailed descriptions of known methods and apparatuses are omitted so as not to obscure the disclosure. In the above-described embodiments, several specific steps are described as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated herein, but can include any number of additional steps or changes to the described steps, or can be performed in any order, or in parallel, depending on the circumstances.

[0271] The functional blocks shown in the structural block diagrams described above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, and the like. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of the machine-readable medium include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact discs (CD-ROM), optical disks, hard disks, optical fiber media, radio frequency (RF) links, and the like. The code segments can be downloaded via a computer network such as the Internet, an intranet, and the like.

[0272] It is also to be understood that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0273] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0274] The above is merely specific implementation of the present application, and those skilled in the art can clearly understand the specific working process of the system, module and unit described above for the convenience and brevity of description, which can refer to the corresponding process in the foregoing method embodiments, and will not be described herein. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A method for executing a data crawling task, characterized in that, include: The system acquires multimodal images, attribute information of the object, and state information of the robotic arm, wherein the object is the object to be grasped, and the multimodal image is an image of the object to be grasped. The attribute information is input into the pattern discrimination model, and the first mapping relationship information in the pattern discrimination model is used to determine the grasping mode corresponding to the attribute information. The grasping mode is the grasping mode of the item to be grasped. The crawling mode is input into the process parameter generation model, and the crawling process parameters corresponding to the crawling mode are determined using the second mapping relationship information in the process parameter generation model. The multimodal image, the state information, and the grasping process parameters are input into the pose solving network model. Using the third mapping relationship information in the pose solving network model, the target action information corresponding to the multimodal image, the state information, and the grasping process parameters is determined. The target action information is the target action information for the robotic arm to grasp the object to be grasped. According to the target motion information, the actuator of the robotic arm is invoked to perform the grasping task.

2. The method for executing a crawling task according to claim 1, characterized in that, The acquisition of multimodal images includes: Initial images are captured using embodied smart sensors; Spatially align the initial image to obtain an aligned image; Data from different channels in the aligned image are extracted to obtain a multimodal image.

3. The method for executing a crawling task according to claim 1 or 2, characterized in that, The attribute information includes the item category and multiple item sub-attribute information; obtaining the item's attribute information includes: The multimodal image is input into an image semantic segmentation model. Using the fourth mapping relationship information in the image semantic segmentation model, the item category corresponding to the multimodal image is determined. The item category includes the category of each pixel in the multimodal image. The number of pixels occupied by the item is determined based on the category of each pixel in the multimodal image; The pixel values ​​of all pixels in the multimodal image except for the occupied pixels are modified to the target values ​​to obtain the target multimodal image; The target multimodal image is input into the attribute discrimination model, and the fifth mapping relationship information in the attribute discrimination model is used to determine the multiple item sub-attribute information of the item corresponding to the target multimodal image.

4. The method for executing a crawling task according to claim 1, characterized in that, Before inputting the attribute information into the pattern discrimination model and determining the crawling pattern corresponding to the attribute information using the first mapping relationship information in the pattern discrimination model, the method further includes: Obtain attribute information samples and the actual crawling mode of the attribute information samples; The attribute information sample and the actual crawling mode of the attribute information sample are input into the initial mode discrimination model. The predicted crawling mode corresponding to the attribute information sample is determined by using the initial first mapping relationship information. Based on the actual crawling pattern and the predicted crawling pattern, determine the first loss value of the initial pattern discrimination model; If the first loss value does not meet the first preset training stopping condition, the parameters of the initial pattern discrimination model are adjusted to update the initial first mapping relationship information, the predicted grasping mode, and the first loss value. The pattern discrimination model is obtained when the updated first loss value satisfies the first preset training stopping condition.

5. The method for executing a crawling task according to claim 1, characterized in that, Before inputting the crawling pattern into the process parameter generation model and determining the crawling process parameters corresponding to the crawling pattern using the second mapping relationship information in the process parameter generation model, the method further includes: Obtain the crawling pattern sample and the actual crawling process parameters corresponding to the crawling pattern sample; The crawling pattern sample and the actual crawling process parameters corresponding to the crawling pattern sample are input into the initial process parameter generation model. Using the initial second mapping relationship information, the predicted crawling process parameters corresponding to the crawling pattern sample are determined. Based on the actual crawling process parameters and the predicted crawling process parameters, determine the second loss value of the initial process parameter generation model; If the second loss value does not meet the second preset training stopping condition, the parameters of the initial process parameter generation model are adjusted to update the initial second mapping relationship information, the prediction and grasping process parameters, and the second loss value. The process parameter generation model is obtained when the updated second loss value satisfies the second preset training stopping condition.

6. The method for executing a crawling task according to claim 1, characterized in that, Before inputting the multimodal image, the state information, and the grasping process parameters into the pose solving network model, and using the third mapping relationship information in the pose solving network model to determine the target action information corresponding to the multimodal image, the state information, and the grasping process parameters, the method further includes: Acquire multimodal image samples, state information samples of the robotic arm, and grasping process parameter samples, as well as the actual action information corresponding to the multimodal image samples, the state information samples, and the grasping process parameter samples; The multimodal image samples, the state information samples, and the grasping process parameter samples, as well as the actual action information corresponding to the multimodal image samples, the state information samples, and the grasping process parameter samples, are input into the initial pose solving network model. Using the initial third mapping relationship information, the predicted action information corresponding to the multimodal image samples, the state information samples, and the grasping process parameter samples is determined. Based on the actual action information and the predicted action information, determine the third loss value of the initial pose solving network model; If the third loss value does not meet the third preset training stopping condition, the parameters of the initial pose solving network model are adjusted to update the initial third mapping relationship information, the predicted action information, and the third loss value. The pose solving network model is obtained when the updated third loss value satisfies the third preset training stopping condition.

7. The method for executing a crawling task according to claim 3, characterized in that, Before inputting the multimodal image into an image semantic segmentation model and determining the item category corresponding to the multimodal image using the fourth mapping relationship information in the image semantic segmentation model, the method further includes: Obtain multimodal image samples and the corresponding actual item categories of the multimodal images; The multimodal image samples and the actual item categories corresponding to the multimodal image samples are input into the initial image semantic segmentation model. Using the initial fourth mapping relationship information, the predicted item category of the item sample corresponding to the multimodal image sample is determined. Based on the actual item category and the predicted item category, determine the fourth loss value of the initial image semantic segmentation model; If the fourth loss value does not meet the fourth preset training stopping condition, the parameters of the initial image semantic segmentation model are adjusted to update the initial fourth mapping relationship information, the predicted item category, and the fourth loss value. The image semantic segmentation model is obtained when the updated fourth loss value satisfies the fourth preset training stopping condition.

8. The method for executing a crawling task according to claim 7, characterized in that, The predicted item category includes the category of each pixel in the multimodal image sample; Before inputting the target multimodal image into the attribute discrimination model and using the fifth mapping relationship information in the attribute discrimination model to determine the multiple item sub-attribute information of the item corresponding to the target multimodal image, the method further includes: Based on the category of each pixel in the multimodal image sample, determine the number of pixels occupied by the item sample; In the multimodal image sample, the pixel values ​​of the pixels other than those occupied by the sample are modified to the target values ​​to obtain the target multimodal image sample; Obtain multiple sub-attribute information of actual objects corresponding to the target multimodal image sample; The target multimodal image sample and the sub-attribute information of multiple actual items corresponding to the target multimodal image sample are input into the initial attribute discrimination model. Using the initial fifth mapping relationship information, multiple predicted item sub-attribute information corresponding to the target multimodal image sample are determined. Based on the multiple actual item sub-attribute information and the multiple predicted item sub-attribute information, the fifth loss value of the initial attribute discrimination model is determined; If the fifth loss value does not meet the fifth preset training stopping condition, the parameters of the initial attribute discrimination model are adjusted to update the initial fifth mapping relationship information, the multiple predicted item sub-attribute information, and the fifth loss value. The attribute discrimination model is obtained when the updated fifth loss value satisfies the fifth preset training stopping condition.

9. An execution device for a grasping task, characterized in that, include: The acquisition module is used to acquire multimodal images, attribute information of the object, and state information of the robotic arm, wherein the object is the object to be grasped, and the multimodal image is an image of the object to be grasped; The determination module is used to input the attribute information into the pattern discrimination model, and use the first mapping relationship information in the pattern discrimination model to determine the grasping mode corresponding to the attribute information, wherein the grasping mode is the grasping mode of the item to be grasped; The determining module is further configured to input the crawling mode into the process parameter generation model, and use the second mapping relationship information in the process parameter generation model to determine the crawling process parameters corresponding to the crawling mode; The determining module is further configured to input the multimodal image, the state information and the grasping process parameters into the pose solving network model, and use the third mapping relationship information in the pose solving network model to determine the target action information corresponding to the multimodal image, the state information and the grasping process parameters, wherein the target action information is the target action information of the robotic arm grasping the object to be grasped; The calling module is used to call the driver of the robotic arm according to the target motion information to perform the grasping task.

10. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the method for executing the grabbing task as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the method for executing a crawling task as described in any one of claims 1-8.

12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the execution method of the grasping task as described in any one of claims 1-8.

Images