Method, apparatus, robot, electronic device, and storage medium for handling a cargo

Abstract

The application provides a method and device for carrying goods, a robot, electronic equipment and a storage medium, and relates to the technical fields of artificial intelligence, intelligent logistics, industrial robots and the like. The method for carrying goods comprises: acquiring a target image related to a carrying task, the carrying task indicating carrying of goods with target inventory unit information by a mechanical arm, and the target image representing at least one target good with target inventory unit information; detecting the target image based on a reference image feature related to the target inventory unit information, determining a grasping pose parameter and a motion control parameter, the grasping pose parameter comprising target grasping position information representing a target surface position that the mechanical arm should contact when performing a grasping action, and the motion control parameter being used to control a motion state of the mechanical arm when performing the grasping action on the target good; and controlling the mechanical arm to perform the carrying task by grasping the target good based on the grasping pose parameter and the motion control parameter.

Description

Technical Field

[0001] This application relates to the technical fields of artificial intelligence, smart logistics, and industrial robots, and more specifically, to a method, apparatus, robot, electronic device, and storage medium for handling goods. Background Technology

[0002] With the rapid development of technology, intelligent equipment such as robots can be used in warehousing and logistics operations to complete tasks such as cargo handling. For example, by sending grasping task instructions to a robot via a server, the robot can grasp the goods according to the received grasping task instructions and move the goods to a designated location.

[0003] In realizing the concept of this application, the inventors discovered at least the following problems in the related technology: when robots are handling goods, it is difficult to accurately grasp the goods, and the goods are easily damaged due to collisions, falling, or other reasons. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, robot, electronic device, and storage medium for handling goods.

[0005] One aspect of this application provides a method for handling goods, comprising: acquiring a target image related to a handling task, the handling task indicating the handling of goods having target inventory unit information by using a robotic arm, the target image representing at least one target item having the target inventory unit information; detecting the target image based on reference image features related to the target inventory unit information, determining grasping pose parameters and motion control parameters, the grasping pose parameters including target grasping position information, the target grasping position information representing the position where the robotic arm should contact a target surface on the target item when performing a grasping action, the motion control parameters being used to control the motion state of the robotic arm performing the grasping action on the target item; and controlling the robotic arm to perform the handling task by grasping the target item based on the grasping pose parameters and the motion control parameters.

[0006] According to an embodiment of this application, the above-mentioned detection of the target image based on reference image features related to the target inventory unit information includes: performing target detection on the target image based on the reference image features to obtain a plurality of candidate image regions, wherein the candidate image regions represent candidate goods having the target inventory unit information; detecting the candidate image regions to obtain a cargo surface sub-region representing the cargo surface of the candidate goods, wherein the cargo surface sub-region includes candidate gripping position information corresponding to the cargo surface of the candidate goods; determining candidate surface image blocks from the cargo surface sub-region based on the candidate gripping position information; and determining the target gripping position information from the plurality of candidate gripping position information based on the evaluation results of each of the plurality of candidate surface image blocks.

[0007] According to an embodiment of this application, determining the target grasping position information from multiple candidate grasping position information based on the evaluation results of each of the multiple candidate surface image blocks includes: fusing multiple evaluation results related to the candidate surface image blocks to obtain a target evaluation result; and determining the target grasping position information from the multiple candidate grasping position information based on the target evaluation results corresponding to each of the multiple candidate surface image blocks, wherein the multiple evaluation results include at least two of the following: a first evaluation result, characterizing the positional relationship between the surface of the candidate cargo and the robotic arm; a second evaluation result, characterizing the size comparison relationship between the first surface dimension of the candidate surface image block in a preset coordinate system and the working dimension of the suction cup component on the robotic arm in the preset coordinate system, wherein the first surface dimension represents the distance between the candidate grasping position information and the boundary of the candidate surface image block; and a third evaluation result, characterizing the area comparison relationship between the surface area of ​​the candidate surface image block in the preset coordinate system and the working area of ​​the suction cup component.

[0008] According to an embodiment of this application, determining the target grasping position information from the multiple candidate grasping position information based on the target evaluation results corresponding to each of the multiple candidate surface image blocks includes: processing the multiple target evaluation results based on a decision model to obtain the target grasping position information.

[0009] According to an embodiment of this application, the above-mentioned detection of the target image based on reference image features related to the target inventory unit information further includes: determining cargo attributes corresponding to the target grasping position information based on the target inventory unit information, wherein the cargo attributes characterize attribute information related to the target surface position of the target cargo; performing motion state decision based on the cargo attributes, and outputting motion control parameters matching the cargo attributes.

[0010] According to an embodiment of this application, the above-mentioned target detection of the target image based on the reference image features to obtain multiple candidate image regions includes: performing feature matching between the reference image features and multiple initial image regions in the target image to obtain a matching result, wherein the initial image regions are determined by target detection of the target image; and determining multiple candidate image regions from the multiple initial image regions based on the matching result.

[0011] According to an embodiment of this application, the method further includes: in response to a detection result obtained by detecting the target image based on the reference image features indicating that a grasping anomaly condition is met, sending a target prompting instruction to a target terminal communicatively connected to the robotic arm, wherein the grasping anomaly condition indicates at least one of the following: the uncovered surface area of ​​the target cargo does not match the working area of ​​the suction cup of the robotic arm's suction cup component; the space occupied by the robotic arm moving along the movement path indicated by the grasping pose parameters at least partially overlaps with the space occupied by obstacles around the target cargo, wherein the detection result includes the uncovered surface area and the grasping pose parameters.

[0012] According to an embodiment of this application, the target terminal is a remote control terminal for remotely controlling the robotic arm. The remote control terminal is used to send remote control commands to the robotic arm to drive the robotic arm to perform a grasping action on the target goods.

[0013] According to an embodiment of this application, the target image is determined based on the following operations: based on the received handling task, target storage location information is determined, which is related to the storage location in the shelf used to place the target goods; based on the target storage location information, the robotic arm is controlled to move to the acquisition position corresponding to the storage location, and the image acquisition device installed on the robotic arm is controlled to acquire an image of the storage location to obtain the target image.

[0014] Another aspect of this application provides an apparatus for handling goods, comprising: an acquisition module for acquiring a target image related to a handling task, the handling task indicating the handling of goods having target inventory unit information by using a robotic arm, the target image representing at least one target item having the target inventory unit information; a detection module for detecting the target image based on reference image features related to the target inventory unit information, determining grasping pose parameters and motion control parameters, the grasping pose parameters including target grasping position information, the target grasping position information representing the position where the robotic arm should contact a target surface on the target item when performing a grasping action, the motion control parameters being used to control the motion state of the robotic arm performing the grasping action on the target item; and a control module for controlling the robotic arm to perform the handling task by grasping the target item based on the grasping pose parameters and the motion control parameters.

[0015] Another aspect of this application provides a robot, including: a robotic arm; and a processor configured to perform the above-described method of transporting goods.

[0016] Another aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method described above.

[0017] Another aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the above-described method.

[0018] Another aspect of this application provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0019] According to embodiments of this application, by acquiring target images related to the handling task and using reference image features related to target inventory unit information to detect the target images, the target grasping position information and motion control parameters can be determined. This enables the robotic arm to grasp target goods with target inventory unit information as indicated by the handling task more accurately, avoiding grasping errors. Simultaneously, motion control parameters determined based on the product attributes corresponding to the target inventory unit information can be used to control the robotic arm's grasping action. This ensures that the robotic arm's movement state when grasping the target goods matches the product attributes of the target goods, preventing damage to fragile or deformable items and improving the efficiency and safety of the goods handling task. Attached Figure Description

[0020] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0021] Figure 1 An exemplary system architecture for a method and apparatus for transporting goods, according to embodiments of this application, is shown.

[0022] Figure 2 A flowchart of a method for transporting goods according to an embodiment of this application is shown.

[0023] Figure 3 A rendering of a target image according to an embodiment of this application is shown.

[0024] Figure 4 A schematic diagram of a robotic arm handling target goods according to an embodiment of this application is shown.

[0025] Figure 5 A rendering of a candidate image region according to an embodiment of this application is shown.

[0026] Figure 6 A rendering of a sub-region of the cargo surface according to an embodiment of this application is shown.

[0027] Figure 7 A schematic diagram illustrating target grabbing location information according to another embodiment of this application is shown.

[0028] Figure 8 A schematic diagram of a robot according to an embodiment of this application is shown.

[0029] Figure 9 A block diagram of a cargo handling apparatus according to an embodiment of this application is shown.

[0030] Figure 10 A block diagram of an electronic device suitable for implementing the above-described method of transporting goods, according to an embodiment of this application, is shown. Detailed Implementation

[0031] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0034] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0035] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0036] In the embodiments of this application, the user's authorization or consent was obtained before obtaining or collecting the user's personal information.

[0037] The inventors discovered that in logistics operations, due to the large variety of goods and the irregular storage patterns, robots have difficulty accurately identifying the goods that need to be moved as indicated by the logistics task. This can easily lead to collisions with the goods being moved or other goods during the handling process, as well as abnormal situations such as goods slipping or being damaged by the pressure of the gripping parts.

[0038] Embodiments of this application provide a method, apparatus, robot, electronic device, and storage medium for handling goods. The method includes: acquiring a target image related to a handling task, the handling task indicating the use of a robotic arm to handle goods having target inventory unit information, the target image representing at least one target item having target inventory unit information; detecting the target image based on reference image features related to the target inventory unit information, determining grasping pose parameters and motion control parameters, the grasping pose parameters including target grasping position information, the target grasping position information representing the position on a target surface on the target item that the robotic arm should contact when performing a grasping action, and the motion control parameters used to control the motion state of the robotic arm performing the grasping action on the target item; and controlling the robotic arm to perform the handling task by grasping the target item based on the grasping pose parameters and motion control parameters.

[0039] Figure 1 An exemplary system architecture for a method and apparatus for transporting goods, applicable according to embodiments of this application, is shown. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this application, in order to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0040] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105, and a robot 106. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0041] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0042] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0043] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0044] The robot may have a communication module and a robotic arm. The communication module is used to interact with any one of the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105 via network 104. The robot controls the movement of the robotic arm according to the control information received by the communication module.

[0045] It should be noted that the method for transporting goods provided in this application embodiment can generally be executed by server 105. Correspondingly, the device for transporting goods provided in this application embodiment can generally be installed in server 105. The method for transporting goods provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the device for transporting goods provided in this application embodiment can also be installed in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the method for transporting goods provided in this application embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the cargo handling device provided in this application embodiment may also be installed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0046] It should be understood that Figure 1 The number of terminal devices, networks, servers, and robots shown is merely illustrative. Any number of terminal devices, networks, servers, and robots can be included depending on implementation needs.

[0047] Figure 2 A flowchart of a method for transporting goods according to an embodiment of this application is shown.

[0048] like Figure 2 As shown, the method for handling goods includes operations S210 to S230.

[0049] In operation S210, acquire the target image related to the transport task.

[0050] In operation S220, the target image is detected based on the reference image features related to the target inventory unit information to determine the grasping pose parameters and motion control parameters.

[0051] When operating S230, the robotic arm is controlled to perform handling tasks by grasping target goods based on the grasping pose parameters and motion control parameters.

[0052] It should be noted that the execution entity of the cargo handling method provided in this application embodiment can be a server connected in communication with the robot, or it can also be the robot's processor. The robot is a device with a robotic arm that can handle cargo using suction cups, gripping components, or other devices for grasping cargo.

[0053] In this embodiment, the handling task instructs the use of a robotic arm to move goods with target stock keeping unit (SKU) information. A stock keeping unit (SKU) is an identifier used to identify goods. For example, different SKUs represent diverse attributes such as the color, brand, and memory capacity of a mobile phone. As the smallest management unit of goods, the SKU indicates the diverse attributes of the goods. Multiple items with the same SKU information can be understood as multiple identical goods.

[0054] For example, a handling task could be instructed to "move the A brand a1 model, blue, 128G smartphone from storage location aa on the shelf to the temporary storage area".

[0055] A target image represents at least one target item having target inventory unit information. The target image can be an image determined by photographing multiple items containing the same target inventory unit information. In one example, the target image records multiple items with target inventory unit information, and may also include items with other inventory unit information.

[0056] In embodiments of this application, reference image features represent the visual semantic information of goods containing target inventory unit information. For example, a reference image is determined by photographing the mobile phone goods containing target inventory unit information, and the visual semantics of the reference image are extracted using a convolutional neural network to obtain reference image features.

[0057] In embodiments of this application, the grasping pose parameters include target grasping position information, which characterizes the position where the robotic arm should contact the target surface on the target cargo when performing the grasping action. For example, the target cargo has a cube-like outline, and the target image represents the three exposed surfaces within the outline of the target cargo. The grasping position information indicates the position of the target surface with the largest area among the three surfaces in a preset coordinate system. The preset coordinate system can be a geodetic coordinate system or a robotic arm coordinate system.

[0058] It should be noted that the grasping pose parameters may also include parameters representing the position and posture of the gripping component of the robotic arm during movement, used to control the degree of position and posture changes of the gripping component during movement. For example, the grasping pose parameters include the robotic arm's motion trajectory composed of control information corresponding to 6 degrees of freedom, and the grasping position information can be the control information corresponding to the end point of the robotic arm's motion trajectory.

[0059] By performing coordinate transformation based on the coordinate transformation parameters between the image coordinate system, the geodetic coordinate system, and the robotic arm coordinate system used for the target image, the grasping pose parameters in the preset coordinate system can be determined.

[0060] Motion control parameters are used to control the motion state of the robotic arm when performing a grasping action on the target goods. Motion control parameters can include, for example, the speed, acceleration, and rotational angular velocity of the robotic arm during its movement. These parameters can be determined based on the product attributes corresponding to the target inventory unit information, ensuring that the motion state represented by the motion control parameters is compatible with the product attributes of the target goods, such as their fragility and deformability.

[0061] Target image detection based on reference image features can utilize a trained deep learning model to process the reference image features and the target image. This allows for accurate identification of the surface contour and storage location of target goods containing target inventory unit information within the target image, based on the visual semantics of the reference image features. This ensures that the output grasping pose parameters match the storage location and surface position of the target goods. Furthermore, the deep learning model can make decisions based on the product attributes corresponding to the target inventory unit information. This allows for the output of motion control parameters based on product attributes such as the fragility, deformability, and surface roughness of the target goods. This ensures that the motion state of the robotic arm grasping the target goods is accurately adapted to the product attributes of the target goods, avoiding excessive collisions, compression, or damage caused by grasping unsuitable surfaces, thus improving the efficiency of the handling task.

[0062] In some embodiments, the target image may be determined by image acquisition performed by a monitoring device for acquiring images of the cargo storage area.

[0063] In some embodiments, the target image can also be determined based on the following operations: determining the target storage location information according to the received handling task; controlling the robotic arm to move to the acquisition position corresponding to the storage location according to the target storage location information, and controlling the image acquisition device installed on the robotic arm to acquire the image of the storage location to obtain the target image.

[0064] The target storage location information is related to the storage location on the shelf used to place the target goods. For example, the target storage location information can be the coordinates, identification information, etc. of the target storage location. The target storage location is used to store multiple goods with target inventory unit information.

[0065] Based on the target storage location information, the robotic arm is controlled to move to a position directly facing the target storage location, ensuring that the image acquisition device of the robotic arm is aligned with the target storage location. By sending control signals to the image acquisition device installed on the robotic arm, image acquisition of the target storage location can be achieved, thus obtaining the target image.

[0066] Figure 3 A rendering of a target image according to an embodiment of this application is shown. Figure 4 A schematic diagram of a robotic arm handling target goods according to an embodiment of this application is shown.

[0067] like Figure 3 As shown, the target image is a color image captured by a camera on a robotic arm moved to the acquisition position, showing the target storage location. The target storage location is a compartment on a shelf, used to store goods with target inventory unit information as indicated by the handling task. "11223" displayed in the target image is the storage location identifier corresponding to the target storage location. Image area 320 shows multiple goods with target inventory unit information stored in the target storage location. "11222" is a storage location used to store goods with other inventory unit information.

[0068] Combination Figure 3 and Figure 4As shown, the gripping component of the robotic arm is a suction cup component 311. It performs gripping actions on the target goods and carries out the transport task based on gripping posture parameters and motion control parameters. By controlling the suction cup component 311 to contact the top surface of the target goods according to the target gripping position information, the robotic arm can maintain the target goods with their top surface facing upwards throughout the transport process. This avoids the suction cup component 311 disengaging from the side surface of the target goods during transport, preventing the top surface of the target goods from flipping over during transport. This reduces the risk of damage to the target goods and improves the efficiency and stability of the transport task.

[0069] It is understandable that the target grasping position information obtained by detecting the target image based on the features of the reference image is... Figure 4 The position coordinates on the top surface of the target cargo correspond to the position coordinates. Therefore, the method provided in this application embodiment can be used to grasp the target cargo more accurately, and the posture of the target cargo during the handling task can be matched with the fragile and deformable properties of the product, thereby improving the efficiency of the handling task.

[0070] In some embodiments, detecting a target image based on reference image features related to target inventory unit information includes: performing target detection on the target image based on the reference image features to obtain multiple candidate image regions; detecting the candidate image regions to obtain cargo surface sub-regions representing cargo surfaces of candidate goods; determining candidate surface image blocks from the cargo surface sub-regions based on candidate grabbing position information; and determining target grabbing position information from the multiple candidate grabbing position information based on the evaluation results of each of the multiple candidate surface image blocks.

[0071] In the embodiments of this application, the reference image features are obtained by extracting visual semantic features from a reference image, and the reference image represents goods with target inventory unit information.

[0072] In some embodiments, the reference image features represent reference images acquired from multiple perspectives of goods containing target inventory unit information. Target detection based on the reference image features may further include detection based on both the reference image features and goods description features. The goods description features include text feature vectors determined by extracting text features from the product attribute information of the goods containing the target inventory unit information, thereby enabling target image detection based on multimodal feature information.

[0073] In embodiments of this application, candidate image regions represent candidate goods with target inventory unit information. In one example, multiple target detection boxes are obtained by processing the target image based on a trained deep learning model. The similarity between the reference image features and the image regions enclosed by the target detection boxes is calculated to obtain multiple candidate detection boxes that match the reference image features. The image regions enclosed by the multiple candidate detection boxes are the candidate image regions.

[0074] The candidate image region displays at least one surface of the candidate cargo; for example, the candidate image region displays three adjacent cargo surfaces of a cubic outline of the candidate cargo. The cargo surface sub-region includes candidate gripping location information corresponding to the cargo surfaces of the candidate cargo. The candidate gripping location information can, for example, be represented as arbitrary coordinate positions within the candidate image region.

[0075] Figure 5 A rendering of a candidate image region according to an embodiment of this application is shown. Figure 6 A rendering of a sub-region of the cargo surface according to an embodiment of this application is shown.

[0076] like Figure 5 As shown, by using an object detection model to process the target image, multiple object detection boxes are obtained in the target image. Each object detection box includes information about the inventory unit of the goods and a confidence level. By matching the image regions within the object detection boxes with features of a reference image, candidate image regions with high visual semantic similarity to the features of the reference image are obtained.

[0077] Multiple candidate image regions represent candidate goods corresponding to the target inventory unit information. By segmenting the candidate image regions, the resulting goods surface sub-regions represent the surface of the candidate goods displayed in the candidate image regions. For example... Figure 6 As shown, the candidate grab location information can be represented as the coordinate position of the circular point displayed on the sub-region of the cargo surface. Figure 6 The circular dots shown can be used as markers for cargo surface sub-regions to indicate the geometric center of candidate cargo surface sub-regions.

[0078] It should be noted that, Figure 5 The target detection box shown is a multi-colored box displayed in the target image. The text or characters near the target detection box have been desensitized to avoid revealing detailed information about the goods on the shelf. Figure 6 The text or characters shown are also blurred to desensitize the specific cargo information represented by the candidate image area.

[0079] In some embodiments, candidate surface image blocks are determined by using the distance between the candidate grasping location information and the edge pixels of the cargo surface represented by the cargo surface sub-region. Each candidate surface image block represents a range of size parameters such as the width and length of the cargo surface of a candidate cargo. This allows for the evaluation of the size of multiple candidate surface image blocks, with the evaluation result representing the area of ​​the candidate surface image block. The candidate surface image block with the largest size area is then determined as the target surface image block, and the candidate grasping location information corresponding to the target surface image block is determined as the target grasping location information.

[0080] Therefore, the suction cup component of the robotic arm can be used to grasp the target cargo surface corresponding to the target surface image block according to the target grasping position information, so as to avoid the problem of air leakage of the suction cup component due to the surface of the cargo being adsorbed not meeting the requirements during the process of the robotic arm grasping the target cargo, thereby improving the stability and execution efficiency of the target cargo handling during the handling task.

[0081] In some embodiments, multiple candidate surface image blocks can be evaluated based on product attributes corresponding to the target inventory unit information. For example, product attributes may indicate that the top cargo surface has the first priority, the rectangular cargo surface adjacent to the top cargo surface has the second priority, and the square cargo surface adjacent to the top cargo surface has the third priority. This allows for priority evaluation of multiple candidate surface image blocks, yielding a priority evaluation result. Based on the priority evaluation result, the target surface image block representing the top cargo surface and the target gripping position information representing the top cargo surface are determined. Therefore, based on gripping pose parameters containing the target gripping position information, the robotic arm can be controlled to grip the top cargo surface of the target cargo, avoiding rotation of the target cargo and potential damage.

[0082] In some embodiments, the target grasping location information can also be determined based on multiple types of evaluation results corresponding to candidate surface image patches. For example, multiple evaluation results can be processed based on a trained machine learning model such as a decision tree model to obtain the target grasping location information.

[0083] In some embodiments, the target grasping position information is processed based on a trained trajectory planning model, and the output is a grasping motion trajectory that represents the grasping pose parameters. The trajectory points in the grasping motion trajectory indicate the position and attitude parameters of each component of the robotic arm during the grasping action.

[0084] In one example, a trained end-to-end model is used to process the target image. Taking input data such as reference image features and task information for the handling task, the output is grasping pose parameters and motion control parameters.

[0085] The target image in the input data is a single or multiple frame color depth image captured by a camera mounted on the robotic arm at the target storage location. Task information for the handling task includes target storage location information, shelf code, and target inventory unit information. Additionally, the input data includes sensor parameters for the robotic arm, including camera intrinsic parameters, extrinsic parameters for conversion between the camera and the robotic arm base, and geometric model parameters of the shelf. Furthermore, the input data includes component parameters for the robotic arm's end effector suction cup, such as suction cup diameter, effective sealing radius, allowable tilt angle, and minimum adsorption area.

[0086] The end-to-end model processes input data based on preset similarity threshold Ts, grasping score threshold Tg, visibility threshold, and other constraint prompts. The output data includes grasping pose parameters such as the coordinate position, rotation angle, and pitch angle of the robotic arm in the corresponding robotic arm coordinate system. It can also include motion control parameters such as motion direction, speed, and acceleration based on motion trajectory points. Thus, the robotic arm can be controlled to perform grasping actions and carry out handling tasks based on the grasping pose parameters and motion control parameters, thereby improving the efficiency of handling tasks.

[0087] In one example, target detection of a target image based on reference image features to obtain multiple candidate image regions may include: performing feature matching between the reference image features and multiple initial image regions in the target image to obtain matching results; and determining multiple candidate image regions from the multiple initial image regions based on the matching results.

[0088] The initial image region is determined by target detection in the initial image. The initial image region represents the multiple goods stored in the target storage location.

[0089] The image acquisition device installed on the robotic arm acquires images of the identification codes near the target storage location and identifies the target storage location information to determine that the target image can represent multiple goods corresponding to the target inventory unit information.

[0090] In one example, the robot moves to the location corresponding to the target storage location based on the received handling task. It verifies the consistency between the identified storage location identifier and the target storage location information for the handling task by identifying the identifier. If the identifier matches, the target storage location is identified. If they don't match or the identifier is not identified, a second image capture of the storage location can be triggered, or a prompt message can be sent to a designated terminal for manual confirmation. This allows for more accurate identification of the target storage location, avoiding the processing of images unrelated to the target inventory unit information, thus improving the accuracy of identifying the target goods and reducing the risk of mishandling goods.

[0091] In the offline phase, image segmentation is performed on the reference image corresponding to the target inventory unit information to obtain an image that retains the visual semantics of the main cargo, thereby reducing the visual semantic interference of the background on the reference image features and improving the reliability of the similarity ranking. A visual semantic extraction network is then used to extract visual features from the segmented image to obtain the reference image features. For example, reference image features can be extracted based on the image branch of a vision-language encoder.

[0092] The initial image features corresponding to the initial image regions are compared with the features of the reference image to calculate a similarity score. Based on the ranking of multiple similarity scores, the top K initial image regions are determined as candidate image regions. Alternatively, a similarity threshold Ts can be set to filter out multiple candidate image regions that meet the similarity threshold.

[0093] Multiple candidate image regions are processed using a trained depth estimation model to obtain depth estimation information for each region. This depth estimation information is then used to correct the depth detection information of the depth camera mounted on the robotic arm, resulting in target depth information. Based on the target depth information and pixel detection information such as normal vectors corresponding to each pixel in the candidate image regions, pixels are clustered to obtain candidate cargo surface sub-regions representing the cargo surface on the candidate cargo. The position of the geometric center pixel of each candidate cargo surface sub-region is used as the candidate grasping position information.

[0094] In some examples, depth estimation information can be depth distance reconstructed from binocular vision images, or it can be depth estimation information determined by other methods such as monocular images. Alternatively, depth estimation information can be obtained by jointly estimating shelf geometry prior parameters, camera calibration information, and historical depth samples.

[0095] Figure 7 A schematic diagram illustrating target grabbing location information according to another embodiment of this application is shown.

[0096] like Figure 7 As shown, the target image represents three candidate goods with target inventory unit information. The image area of ​​the surface of the candidate goods can be a sub-region of the candidate goods surface, and the candidate grab position information can be the coordinate position of the marker point displayed on the sub-region of the candidate goods surface.

[0097] The area size of the candidate surface image patch is determined by the distance between the marker point and the boundary of the candidate surface sub-region. The evaluation result is obtained by evaluating the area size of the candidate surface image patch, and the target grabbing position information is determined as the coordinate position corresponding to the marker point 331.

[0098] In some embodiments, determining the target grasping location information from multiple candidate grasping location information based on the evaluation results for each of the multiple candidate surface image patches may include: fusing multiple evaluation results related to the candidate surface image patches to obtain a target evaluation result; and determining the target grasping location information from the multiple candidate grasping location information based on the target evaluation results corresponding to each of the multiple candidate surface image patches.

[0099] Among them, multiple evaluation results include at least two of the first evaluation result, the second evaluation result, and the third evaluation result.

[0100] The first evaluation result characterizes the positional relationship between the surface of the candidate cargo and the robotic arm.

[0101] For example, the first evaluation result is the depth information corresponding to the candidate surface image patch, or, for another example, the first evaluation result is determined based on the spacing d, where the spacing d is the Euclidean distance between the coordinates of the candidate grasping position information in the preset coordinate system and the coordinates of the image acquisition device of the robotic arm in the preset coordinate system. The spacing score is obtained by quantifying and scoring the spacing d. distance As the first evaluation result, the spacing score is inversely proportional to the spacing d. This allows for the determination of a higher spacing score based on the nearest candidate grasping position, thus preventing the robotic arm from extending too far and colliding with other goods.

[0102] The second evaluation result characterizes the dimensional comparison between the first surface dimension of the candidate surface image patch in the preset coordinate system and the working dimension of the suction cup component on the robotic arm in the preset coordinate system. The preset coordinate system can be any one of the image coordinate system, camera coordinate system, and geodetic coordinate system. The first surface dimension represents the distance between the candidate grasping position information and the boundary of the candidate surface image patch.

[0103] For example, the working dimension of a suction cup can be the working radius of the suction cup component, and the first surface dimension represents... Figure 7 The distance between marker point 331 and candidate cargo surface 330 in the preset coordinate system is shown. The first surface size is obtained by calculating the distance between the candidate grab position information corresponding to marker point 331 and the boundary pixels of the candidate cargo surface sub-region in the target image, and performing coordinate system transformation.

[0104] First surface dimension d edge-m Smaller than the working radius r of the suction cup component cup In the case where the robotic arm's suction cup grips the candidate cargo surface corresponding to the candidate surface image patch, there is a defect that makes it prone to air leakage. The second score corresponding to the second evaluation result is... suction =0, which allows filtering out the candidate grasping position information corresponding to the candidate surface image patch.

[0105] The third evaluation result characterizes the comparison between the surface area of ​​the candidate surface image patch on the cargo in the preset coordinate system and the working area of ​​the suction cup component. The working area of ​​the suction cup component represents the area covered by the suction cup component in the preset coordinate system that can be covered by the suction cup component to perform the gripping action. The working area of ​​the suction cup component is calculated based on the working radius of the suction cup component. The cargo surface area represents the visible area of ​​the candidate cargo surface that is not obscured by other objects.

[0106] The third score corresponding to the third evaluation result can be expressed as follows: if the surface area of ​​the cargo is smaller than the working area of ​​the suction cup component, the third score is 0, and the candidate surface image block is eliminated. The third score can also be determined by quantifying the surface area of ​​the cargo. visibility Third score visibility This indicates whether the surface area of ​​the unobstructed candidate goods among the multiple goods placed in the target storage location can meet the working area requirements of the suction cup component.

[0107] In some embodiments, the evaluation results also include obstacle evaluation results. For example, if no obstacle is detected above a candidate surface image patch, the obstacle evaluation result corresponds to an obstacle score of Score. clearance =0, indicating that an obstacle is detected above the candidate surface image patch, and the obstacle evaluation result corresponds to the obstacle score of Score. clearance =1.

[0108] In some embodiments, determining the target grasping location information from the multiple candidate grasping location information based on the target evaluation results corresponding to each of the multiple candidate surface image patches may include fusing the scores corresponding to the multiple evaluation results to obtain a target score, and determining the highest-ranked target grasping location information based on the target score corresponding to each of the multiple candidate grasping location information.

[0109] For example, the target score is determined based on the following formula (1).

[0110] (1);

[0111] Where S1 is the obstacle score. clearance S2 is the second score. suction S3 is the third score. visibility S4 is the interval score. distance .

[0112] In some embodiments, the target grasping location information can also be determined based on the target score and similarity score of the candidate grasping location information, so as to achieve joint gating of recognition confidence and grasping action execution stability to obtain the target grasping location information.

[0113] For all candidate capture location information, determine the similarity and target score of the corresponding target detection boxes, based on the similarity threshold T. s , and confidence threshold T g The system filters out bounding boxes with similarity and confidence thresholds. Based on the target score corresponding to the candidate capture location information of each bounding box, the target capture location information is determined.

[0114] Table 1

[0115]

[0116] In some embodiments, determining the target grasping location information from multiple candidate grasping location information based on the target evaluation results corresponding to each of the multiple candidate surface image blocks may further include: processing multiple target evaluation results based on a decision model to obtain the target grasping location information.

[0117] The decision model can include any type of model such as decision tree model and neural network model. The embodiments of this application do not limit the specific model type of the decision model.

[0118] By using a trained decision model to process the evaluation results of multiple targets, the decision-making process of filtering multiple candidate grabbing location information can be executed more accurately, improving decision-making speed and execution efficiency of handling tasks.

[0119] In some embodiments, the target score can be obtained by weighted fusion of multiple evaluation results of candidate crawling location information, and then the target crawling location information can be determined from multiple candidate crawling location information by joint determination based on the target score and similarity score.

[0120] In some embodiments, detecting the target image based on reference image features related to the target inventory unit information further includes: determining the cargo attributes corresponding to the target grasping location information based on the target inventory unit information; performing motion state decisions based on the cargo attributes; and outputting motion control parameters that match the cargo attributes.

[0121] For example, cargo attributes represent attribute information related to the target surface location of the target cargo. For instance, cargo attributes indicate information such as the fragility and deformability of the target cargo at the target surface location. Therefore, a trained deep learning model can process cargo attributes to make motion state decisions, outputting motion control parameters such as velocity, acceleration, and direction that are adapted to the fragility and deformability of the target surface location. This allows the robotic arm to grasp the target cargo based on contact force adapted to the target surface location, improving the efficiency and safety of cargo handling.

[0122] In some embodiments, the method for handling goods further includes: in response to a detection result obtained by detecting a target image based on reference image features indicating that the grasping anomaly condition is met, sending a target prompting instruction to a target terminal that is communicatively connected to the robotic arm.

[0123] Among them, the captured abnormal condition represents at least one of the first abnormal condition and the second abnormal condition:

[0124] The first anomaly condition indicates that the uncovered surface area of ​​the target cargo does not match the working area of ​​the suction cup component of the robotic arm. For example, the area of ​​a sub-region of the candidate cargo surface in the preset coordinate system is smaller than the working area of ​​the suction cup component.

[0125] The second anomaly condition indicates that the space occupied by the robotic arm, which moves along a path indicated by the grasping pose parameters, at least partially overlaps with the space occupied by obstacles around the target cargo.

[0126] The space occupied by the robot arm represents the area of ​​space in a preset coordinate system occupied by the robot arm moving according to the grasping pose parameters. The position coordinates indicated by the grasping pose parameters and the contour dimensions of any component in the robot arm can determine the space occupied by the robot arm.

[0127] The space occupied by obstacles can be determined based on the target detection bounding box associated with the target cargo. When the space occupied by the moving object and the space occupied by the obstacle at least partially overlap, it can be determined that a collision will occur when the robotic arm grasps the target cargo. Therefore, a target cueing command can be sent to prompt human intervention, thereby improving the efficiency of handling abnormal situations during the handling process.

[0128] It should be understood that the detection results include the uncovered surface area and the grasping pose parameters.

[0129] In some embodiments, the target terminal is a remote control terminal for remotely controlling a robotic arm. The remote control terminal is used to send remote control commands to the robotic arm to drive the robotic arm to perform a grasping action on the target goods.

[0130] Staff can remotely control the robotic arm to grab target goods based on the prompts, so as to quickly move target goods that are difficult to grab accurately and improve handling efficiency.

[0131] In some embodiments, a re-acquisition command can be sent to the robotic arm to control the camera mounted on the robotic arm to move to a new viewing angle and acquire a new target image from the target storage location. Alternatively, the image quality of the target image can be improved by correcting the image resolution or performing local magnification. This allows the current automatic grasping process to be terminated and a rollback process to be entered when the confidence level corresponding to the target detection box is insufficient, thereby reducing the risk of false grasping.

[0132] In some embodiments, local image regions can be processed using various algorithms such as genetic algorithms and local search algorithms to obtain multiple candidate capture location information.

[0133] Based on the cargo handling method provided in the above embodiments, embodiments of this application also provide a robot. The robot can be used to perform the cargo handling method provided in the above embodiments.

[0134] Figure 8 A schematic diagram of a robot according to an embodiment of this application is shown.

[0135] like Figure 8 As shown, robot 800 includes: robotic arm 810 and processor 820.

[0136] The processor is configured to perform the cargo handling method provided in the above embodiments.

[0137] For example, the end effector of the robotic arm is a suction cup component.

[0138] Figure 9 A block diagram of a cargo handling apparatus according to an embodiment of this application is shown.

[0139] like Figure 9 As shown, the cargo handling device 900 includes an acquisition module 910, a detection module 920, and a control module 930.

[0140] The acquisition module 910 is used to acquire a target image related to a handling task, which indicates that a robotic arm is used to handle goods with target inventory unit information, and the target image represents at least one target goods with target inventory unit information.

[0141] The detection module 920 is used to detect the target image based on the reference image features related to the target inventory unit information, and determine the grasping pose parameters and motion control parameters. The grasping pose parameters include the target grasping position information, which represents the position of the target surface on the target goods that the robotic arm should contact when performing the grasping action. The motion control parameters are used to control the motion state of the robotic arm when performing the grasping action on the target goods.

[0142] The control module 930 is used to control the robotic arm to perform handling tasks by grasping target goods based on the grasping pose parameters and motion control parameters.

[0143] According to an embodiment of this application, the detection module includes: a first obtaining unit, a detection unit, a first determining unit, and a second determining unit.

[0144] The first acquisition unit is used to perform target detection on the target image based on the features of the reference image to obtain multiple candidate image regions, which represent candidate goods with target inventory unit information.

[0145] The detection unit is used to detect candidate image regions and obtain cargo surface sub-regions that characterize the cargo surface of candidate cargo. The cargo surface sub-regions include candidate grab position information corresponding to the cargo surface of the candidate cargo.

[0146] The first determining unit is used to determine candidate surface image blocks from the cargo surface sub-region based on candidate grabbing location information.

[0147] The second determining unit is used to determine the target grasping position information from multiple candidate grasping position information based on the evaluation results of each of the multiple candidate surface image patches.

[0148] According to an embodiment of this application, the second determining unit includes a fusion subunit and a first determining subunit.

[0149] The fusion subunit is used to fuse multiple evaluation results associated with candidate surface image patches to obtain the target evaluation result.

[0150] The first determining subunit is used to determine the target grasping position information from multiple candidate grasping position information based on the target evaluation results corresponding to each of the multiple candidate surface image blocks. The multiple evaluation results include at least two of the following: a first evaluation result, which characterizes the positional relationship between the surface of the candidate cargo and the robotic arm; a second evaluation result, which characterizes the size comparison relationship between the first surface dimension of the candidate surface image block in the preset coordinate system and the working dimension of the suction cup component on the robotic arm in the preset coordinate system, wherein the first surface dimension represents the distance between the candidate grasping position information and the boundary of the candidate surface image block; and a third evaluation result, which characterizes the area comparison relationship between the surface area of ​​the candidate surface image block in the preset coordinate system and the working area of ​​the suction cup component.

[0151] According to an embodiment of this application, the first determining subunit is configured to: process multiple target evaluation results based on a decision model to obtain target grasping location information.

[0152] According to an embodiment of this application, the detection module further includes: a cargo attribute determination unit and a decision unit.

[0153] The cargo attribute determination unit is used to determine the cargo attributes corresponding to the target grabbing location information based on the target inventory unit information. The cargo attributes represent attribute information related to the target surface location of the target cargo.

[0154] The decision-making unit is used to make motion state decisions based on cargo attributes and output motion control parameters that match the cargo attributes.

[0155] According to an embodiment of this application, the first obtaining unit includes: a matching subunit and a second determining subunit.

[0156] The matching subunit is used to perform feature matching between the reference image features and multiple initial image regions in the target image to obtain the matching result. The initial image regions are determined by target detection in the target image.

[0157] The second determining subunit is used to determine multiple candidate image regions from multiple initial image regions based on the matching results.

[0158] According to an embodiment of this application, the device for transporting goods further includes a sending module.

[0159] The sending module is used to send a target prompt instruction to the target terminal connected to the robotic arm in response to a detection result obtained by detecting the target image based on the features of the reference image, which indicates that the grasping anomaly condition is met. The grasping anomaly condition indicates at least one of the following: the uncovered surface area of ​​the target cargo does not match the working area of ​​the suction cup of the robotic arm's suction cup component; the space occupied by the robotic arm moving according to the moving path indicated by the grasping pose parameters at least partially overlaps with the space occupied by obstacles around the target cargo. The detection result includes the uncovered surface area and the grasping pose parameters.

[0160] According to an embodiment of this application, the target terminal is a remote control terminal for remotely controlling a robotic arm. The remote control terminal is used to send remote control commands to the robotic arm to drive the robotic arm to perform a grasping action on the target goods.

[0161] According to an embodiment of this application, the target image is determined based on the following operations: determining target storage location information based on the received handling task, the target storage location information being related to the storage location in the shelf used to place the target goods; controlling the robotic arm to move to the acquisition position corresponding to the storage location based on the target storage location information, and controlling the image acquisition device installed on the robotic arm to acquire images of the storage location to obtain the target image.

[0162] Any one or more of the modules, units, and subunits according to the embodiments of this application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, units, and subunits according to the embodiments of this application can be implemented by splitting them into multiple modules. Any one or more of the modules, units, and subunits according to the embodiments of this application can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by hardware or firmware in any other reasonable manner by integrating or packaging circuits, or implemented by any one of software, hardware, and firmware, or by any appropriate combination of any of these. Alternatively, one or more of the modules, units, and subunits according to the embodiments of this application can be at least partially implemented as computer program modules, which can perform corresponding functions when the computer program module is run.

[0163] For example, any plurality of the acquisition module 910, detection module 920, and control module 930 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this application, at least one of the acquisition module 910, detection module 920, and control module 930 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 910, the detection module 920, and the control module 930 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0164] It should be noted that the device part for transporting goods in the embodiments of this application corresponds to the method part for transporting goods in the embodiments of this application. The description of the device part for transporting goods is specifically referred to in the method part for transporting goods, and will not be repeated here.

[0165] Figure 10 A block diagram of an electronic device suitable for implementing the above-described method of transporting goods, according to an embodiment of this application, is shown. Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0166] like Figure 10 As shown, an electronic device 1000 according to an embodiment of this application includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0167] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the aforementioned programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in the aforementioned one or more memories.

[0168] According to embodiments of this application, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the input / output (I / O) interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0169] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0170] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0171] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0172] For example, according to embodiments of this application, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0173] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the method for transporting goods provided in the embodiments of this application.

[0174] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc. described above can be implemented by computer program modules.

[0175] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0176] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0177] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0178] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. This application does not depart from its scope, and those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for transporting goods, characterized in that, include: Acquire a target image related to a handling task, the handling task indicating the use of a robotic arm to handle goods having target inventory unit information, the target image representing at least one target item having the target inventory unit information; The target image is detected based on reference image features related to the target inventory unit information to determine grasping pose parameters and motion control parameters. The grasping pose parameters include target grasping position information, which represents the position of the target surface on the target goods that the robotic arm should contact when performing the grasping action. The motion control parameters are used to control the motion state of the robotic arm when performing the grasping action on the target goods. Based on the grasping pose parameters and the motion control parameters, the robotic arm is controlled to grasp the target cargo to perform the handling task.

2. The method according to claim 1, characterized in that, The detection of the target image based on reference image features related to the target inventory unit information includes: Based on the features of the reference image, target detection is performed on the target image to obtain multiple candidate image regions, and the candidate image regions represent candidate goods with the target inventory unit information; The candidate image region is detected to obtain a cargo surface sub-region that represents the cargo surface of the candidate cargo. The cargo surface sub-region includes candidate gripping position information corresponding to the cargo surface of the candidate cargo. Based on the candidate grab location information, candidate surface image blocks are determined from the cargo surface sub-region; and The target grasping position information is determined from the candidate grasping position information based on the evaluation results of each of the candidate surface image patches.

3. The method according to claim 2, characterized in that, The step of determining the target grasping position information from the multiple candidate grasping position information based on the evaluation results of each of the multiple candidate surface image patches includes: Multiple evaluation results related to the candidate surface image patch are fused to obtain the target evaluation result; Based on the target evaluation results corresponding to each of the multiple candidate surface image patches, the target grasping position information is determined from the multiple candidate grasping position information, wherein the multiple evaluation results include at least two of the following: The first evaluation result characterizes the positional relationship between the surface of the candidate cargo and the robotic arm; The second evaluation result characterizes the size comparison relationship between the first surface size of the candidate surface image block in the preset coordinate system and the working size of the suction cup component on the robotic arm in the preset coordinate system. The first surface size represents the distance between the candidate grasping position information and the boundary of the candidate surface image block. The third evaluation result characterizes the area comparison relationship between the cargo surface area of ​​the candidate surface image block in the preset coordinate system and the working area of ​​the suction cup component.

4. The method according to claim 3, characterized in that, The step of determining the target grasping position information from the plurality of candidate grasping position information based on the target evaluation results corresponding to each of the plurality of candidate surface image patches includes: The target capture location information is obtained by processing multiple target evaluation results based on the decision model.

5. The method according to claim 2, characterized in that, The detection of the target image based on reference image features related to the target inventory unit information further includes: Based on the target inventory unit information, the cargo attributes corresponding to the target grabbing location information are determined, and the cargo attributes represent attribute information related to the target surface location of the target cargo. Based on the cargo attributes, a motion state decision is made, and motion control parameters matching the cargo attributes are output.

6. The method according to claim 2, characterized in that, The target detection based on the reference image features yields multiple candidate image regions, including: Based on the features of the reference image, feature matching is performed with multiple initial image regions in the target image to obtain a matching result. The initial image regions are determined by target detection in the target image. Based on the matching results, multiple candidate image regions are determined from multiple initial image regions.

7. The method according to claim 1, characterized in that, The method further includes: In response to a detection result obtained by detecting the target image based on the features of the reference image indicating that a grasping anomaly condition is met, a target prompting command is sent to the target terminal communicatively connected to the robotic arm, wherein the grasping anomaly condition indicates at least one of the following: The uncovered surface area of ​​the target cargo does not match the working area of ​​the suction cup of the robotic arm's suction cup component; The space occupied by the robotic arm, which moves along a path indicated by the grasping pose parameters, at least partially overlaps with the space occupied by obstacles around the target cargo. The detection result includes the uncovered surface area and the grasping pose parameters.

8. The method according to claim 7, characterized in that, The target terminal is a remote control terminal used to remotely control the robotic arm. The remote control terminal is used to send remote control commands to the robotic arm to drive the robotic arm to perform a grasping action on the target goods.

9. The method according to claim 1, characterized in that, The target image is determined based on the following operations: Based on the received handling task, target storage location information is determined, which is related to the storage location in the shelf used to place the target goods; Based on the target storage location information, the robotic arm is controlled to move to the acquisition position corresponding to the storage location, and the image acquisition device installed on the robotic arm is controlled to acquire the image of the storage location to obtain the target image.

10. A device for transporting goods, characterized in that, include: An acquisition module is used to acquire a target image related to a handling task, the handling task indicating the use of a robotic arm to handle goods having target inventory unit information, the target image representing at least one target item having the target inventory unit information; The detection module is used to detect the target image based on reference image features related to the target inventory unit information, and determine the grasping pose parameters and motion control parameters. The grasping pose parameters include target grasping position information, which represents the position of the target surface on the target goods that the robotic arm should contact when performing the grasping action. The motion control parameters are used to control the motion state of the robotic arm when performing the grasping action on the target goods. as well as The control module is used to control the robotic arm to perform the handling task by grasping the target cargo based on the grasping pose parameters and the motion control parameters.

11. A robot, characterized in that, include: robotic arm; A processor configured to perform the method of any one of claims 1 to 9.

12. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 9.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images