Robot grabbing positioning method and system

Abstract

The invention discloses a robot grabbing positioning method and system, and the method comprises the steps: obtaining a depth image of a to-be-grabbed region, carrying out the interception of a target region corresponding to at least one marking region and a transition region in the depth image, and obtaining a target image region corresponding to at least one to-be-grabbed object; acquiring three-dimensional point cloud data corresponding to the target image area; and inputting the three-dimensional point cloud data into a pre-trained grabbing strategy prediction model to obtain an optimal grabbing strategy of the at least one to-be-grabbed object, so that the mechanical arm grabs the at least one to-be-grabbed object according to the optimal grabbing strategy. The problem that when multiple target objects are stacked and placed, an existing target object grabbing method cannot accurately recognize the specific positions of the target objects stacked and placed in a to-be-grabbed area, and consequently the target objects cannot be accurately grabbed due to the fact that the spatial positions of the target objects are located wrongly is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of positioning, and in particular relates to a robot grasping positioning method and system. Background Art

[0002] With the continuous development of automatic control technology, automatic grasping equipment such as manipulators and grasping robots are widely used in industrial assembly lines. People can use grasping equipment to automate operations such as sorting, assembling and transferring of target objects, thereby improving the efficiency of industrial production.

[0003] However, during the operation of the loading and unloading robot, low grasping efficiency may occur due to problems such as inaccurate identification of the workpiece. Summary of the invention

[0004] The present invention provides a robot grasping and positioning method and system, which are used to solve the technical problem of low grasping efficiency caused by inaccurate identification of workpieces.

[0005] In a first aspect, the present invention provides a robot grasping and positioning method for a mobile robot having a grasping mechanical arm, the method comprising:

[0006] Acquire a visible light image of the area to be grasped, and recognize the visible light image based on a pre-trained image recognition model, wherein the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image;

[0007] According to the position information, at least one object to be grasped in the visible light image is marked with a marking frame corresponding to the type information to obtain at least one marked area, and a transition area connected to the at least one marked area is selected by sliding on an edge of the at least one marked area based on a preset sliding window;

[0008] Acquire a depth image of the area to be grasped, and intercept a target area in the depth image corresponding to the at least one marked area and the transition area to obtain a target image area corresponding to at least one object to be grasped;

[0009] Acquire three-dimensional point cloud data corresponding to the target image area;

[0010] The three-dimensional point cloud data is input into a pre-trained grasping strategy prediction model to obtain an optimal grasping strategy for the at least one object to be grasped, so that the robotic arm grasps the at least one object to be grasped according to the optimal grasping strategy, wherein the optimal grasping strategy includes optimal grasping position information and optimal grasping posture information.

[0011] In a second aspect, the present invention provides a robot grasping and positioning system for a mobile robot having a grasping mechanical arm, the system comprising:

[0012] a recognition module configured to obtain a visible light image of the area to be grasped, and recognize the visible light image based on a pre-trained image recognition model, wherein the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image;

[0013] a marking module configured to mark at least one object to be grasped in the visible light image according to the position information using a marking frame corresponding to the type information to obtain at least one marked area, and to slide on the edge of the at least one marked area based on a preset sliding window to select a transition area connected to the at least one marked area;

[0014] A capture module is configured to obtain a depth image of the area to be captured, and to capture a target area in the depth image corresponding to the at least one marked area and the transition area, so as to obtain a target image area corresponding to at least one object to be captured;

[0015] An acquisition module, configured to acquire three-dimensional point cloud data corresponding to the target image area;

[0016] The grasping module is configured to input the three-dimensional point cloud data into a pre-trained grasping strategy prediction model to obtain an optimal grasping strategy for the at least one object to be grasped, so that the robotic arm grasps the at least one object to be grasped according to the optimal grasping strategy, wherein the optimal grasping strategy includes optimal grasping position information and optimal grasping posture information.

[0017] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the robot grasping and positioning method of any embodiment of the present invention.

[0018] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor is caused to execute the steps of the robot grasping and positioning method of any embodiment of the present invention.

[0019] The robot grasping and positioning method and system of the present application, by intercepting the target area corresponding to at least one marked area and the transition area in the depth image, obtains the target image area corresponding to at least one object to be grasped, which can avoid the disadvantage of too large recognition amount during global recognition, and can effectively improve the detection and recognition capabilities of the target, and obtains three-dimensional point cloud data corresponding to the target image area, and inputs the three-dimensional point cloud data into a pre-trained grasping strategy prediction model to obtain the best grasping strategy for at least one object to be grasped, so that the robot arm can grasp at least one object to be grasped according to the best grasping strategy, so as to solve the problem that when multiple targets are stacked, the existing target grasping method cannot accurately identify the specific positions of the stacked targets in the area to be grasped, and the target cannot be accurately grasped due to the wrong spatial position positioning of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a robot grasping and positioning method provided by one embodiment of the present invention;

[0022] Figure 2 A structural block diagram of a robot grasping and positioning system provided by one embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] See also Figure 1 , which shows a flow chart of a robot grasping and positioning method of the present application.

[0026] like Figure 1 As shown, the robot grasping and positioning method specifically includes the following steps:

[0027] Step S101, obtaining a visible light image of the area to be grasped, and recognizing the visible light image based on a pre-trained image recognition model, wherein the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image.

[0028] In this step, the convolutional neural network is iteratively trained according to the visible light image samples to obtain an image recognition model, wherein the convolutional neural network includes a convolution layer, an activation layer, a pooling layer, and a fully connected layer. Afterwards, a visible light image of the area to be grasped is obtained, and the visible light image is recognized based on the pre-trained image recognition model, and the image recognition model outputs the type information and position information of at least one object to be grasped in the visible light image.

[0029] Step S102: Based on the position information, at least one object to be grasped in the visible light image is marked with a marking frame corresponding to the type information to obtain at least one marked area, and a preset sliding window is used to slide on the edge of the at least one marked area to select a transition area connected to the at least one marked area.

[0030] In this step, contour information of at least one object to be grasped is obtained according to the type information; at least one object to be grasped in the visible light image is located according to the position information, and the located at least one object to be grasped is framed according to the contour information, so as to obtain at least one marked area containing only the object to be grasped.

[0031] Step S103 , acquiring a depth image of the area to be grasped, and intercepting a target area in the depth image corresponding to the at least one marked area and the transition area to obtain a target image area corresponding to at least one object to be grasped.

[0032] In this step, the target size of the visible light image is obtained, and the size of the depth image is adjusted to be consistent with the target size; based on the SIFT algorithm, the feature points in the depth image are matched with the feature points of the target transition area in the visible light image, and the depth image is intercepted according to the matching results to obtain a target image area corresponding to at least one object to be grasped.

[0033] It should be noted that since the transition area is small, the SIFT algorithm can be used to achieve scale-invariant feature point detection, appropriately reduce the neighborhood range of feature points in the SIFT algorithm, increase the density of feature points to make up for the lack of quantity, and also reduce the descriptor scale accordingly to ensure consistency. Ratio screening and geometric verification are added during matching to improve accuracy, and a relatively adaptive ratio threshold is used instead of a fixed threshold to make the matching more stable. The screening results are then sorted by the matching score, and the subset with the highest confidence is selected. Finally, the RANSAC algorithm with a large number of internal points is used for model fitting to obtain a more accurate image transformation estimate and complete the accurate registration of small images to scene images. This improvement in parameter tuning and matching strategy can effectively improve the extraction and matching effects of SIFT feature points in small images.

[0034] Step S104, obtaining three-dimensional point cloud data corresponding to the target image area.

[0035] In this step, the intrinsic and extrinsic parameters of the acquisition device are obtained; based on the intrinsic and extrinsic parameters, the image coordinates of the target image area are converted to the world coordinate system to obtain the three-dimensional point cloud data corresponding to the target image area.

[0036] Specifically, the camera coordinates of the target grasping area are calculated using the image coordinates and the internal parameters; and the world coordinates of the target grasping area are calculated using the camera coordinates and the external parameters.

[0037] Optionally, the internal parameters include the focal length of the acquisition device, the position of the principal point, and the ratio of the pixel to the real environment size. Optionally, the external parameters include a translation matrix and a rotation matrix; wherein the translation matrix represents the translation amount in the x, y, and z directions, and the rotation angle represents the rotation angle around the x, y, and z axes.

[0038] Step S105, inputting the three-dimensional point cloud data into a pre-trained grasping strategy prediction model to obtain an optimal grasping strategy for the at least one object to be grasped, so that the robotic arm grasps the at least one object to be grasped according to the optimal grasping strategy, wherein the optimal grasping strategy includes optimal grasping position information and optimal grasping posture information.

[0039] In this step, the grasping strategy prediction model is used to predict the probability of successful grasping when using multiple grasping postures to grasp different positions on at least one object to be grasped, and the optimal grasping strategy is the grasping strategy corresponding to the highest probability of successfully grasping at least one object to be grasped; the strategy prediction model includes a probability prediction sub-model, and the probability prediction sub-model is used to predict the probability of successful grasping when using each grasping strategy to grasp the at least one object to be grasped.

[0040] In summary, the method of the present application, by intercepting the target area corresponding to at least one annotated area and a transition area in the depth image, obtains a target image area corresponding to at least one object to be grasped, which can avoid the disadvantage of excessive recognition amount during global recognition, and can effectively improve the detection and recognition capabilities of the target, and obtain three-dimensional point cloud data corresponding to the target image area, and input the three-dimensional point cloud data into a pre-trained grasping strategy prediction model to obtain the best grasping strategy for at least one object to be grasped, so that the robotic arm can grasp at least one object to be grasped according to the best grasping strategy, thereby solving the problem that when multiple targets are stacked, the existing target grasping method cannot accurately identify the specific positions of the stacked targets in the area to be grasped, and the target cannot be accurately grasped due to the incorrect spatial positioning of the target.

[0041] See also Figure 2 , which shows a structural block diagram of a robot grasping and positioning system of the present application.

[0042] like Figure 2 As shown, the robot grasping and positioning system 200 includes a recognition module 210 , a labeling module 220 , a interception module 230 , an acquisition module 240 and a grasping module 250 .

[0043] The recognition module 210 is configured to obtain a visible light image of the area to be grasped, recognize the visible light image based on a pre-trained image recognition model, and the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image;

[0044] The marking module 220 is configured to mark at least one object to be grasped in the visible light image according to the position information using a marking frame corresponding to the type information to obtain at least one marked area, and slide on the edge of the at least one marked area based on a preset sliding window to select a transition area connected to the at least one marked area;

[0045] The interception module 230 is configured to obtain a depth image of the area to be captured, intercept a target area corresponding to the at least one marked area and the transition area in the depth image, and obtain a target image area corresponding to at least one object to be captured;

[0046] An acquisition module 240 is configured to acquire three-dimensional point cloud data corresponding to the target image area;

[0047] The grasping module 250 is configured to input the three-dimensional point cloud data into a pre-trained grasping strategy prediction model to obtain an optimal grasping strategy for the at least one object to be grasped, so that the robotic arm grasps the at least one object to be grasped according to the optimal grasping strategy, wherein the optimal grasping strategy includes optimal grasping position information and optimal grasping posture information.

[0048] It should be understood that Figure 2 Modules and references documented in Figure 1 Therefore, the operations and features described above for the method and the corresponding technical effects are also applicable to Figure 2 The modules in it will not be described in detail here.

[0049] In some other embodiments, the embodiments of the present invention further provide a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor is caused to execute the robot grasping and positioning method in any of the above method embodiments;

[0050] As an implementation mode, the computer-readable storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:

[0051] Acquire a visible light image of the area to be grasped, and recognize the visible light image based on a pre-trained image recognition model, wherein the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image;

[0052] According to the position information, at least one object to be grasped in the visible light image is marked with a marking frame corresponding to the type information to obtain at least one marked area, and a transition area connected to the at least one marked area is selected by sliding on an edge of the at least one marked area based on a preset sliding window;

[0053] Acquire a depth image of the area to be grasped, and intercept a target area in the depth image corresponding to the at least one marked area and the transition area to obtain a target image area corresponding to at least one object to be grasped;

[0054] Acquire three-dimensional point cloud data corresponding to the target image area;

[0055] The three-dimensional point cloud data is input into a pre-trained grasping strategy prediction model to obtain an optimal grasping strategy for the at least one object to be grasped, so that the robotic arm grasps the at least one object to be grasped according to the optimal grasping strategy, wherein the optimal grasping strategy includes optimal grasping position information and optimal grasping posture information.

[0056] The computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the robot grasping and positioning system, etc. In addition, the computer-readable storage medium may include a high-speed random access memory, and may also include a memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the robot grasping and positioning system via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0057] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the device includes: a processor 310 and a memory 320. The electronic device may also include: an input device 330 and an output device 340. The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Figure 3 The example of connecting via bus is taken. The memory 320 is the computer-readable storage medium mentioned above. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the robot grasping and positioning method of the above method embodiment is realized. The input device 330 can receive input digital or character information, and generate key signal input related to user settings and function control of the robot grasping and positioning system. The output device 340 may include display devices such as display screens.

[0058] The electronic device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0059] As an implementation mode, the electronic device is applied to a robot grasping and positioning system, and is used for a client, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can:

[0060] Acquire a visible light image of the area to be grasped, and recognize the visible light image based on a pre-trained image recognition model, wherein the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image;

[0061] According to the position information, at least one object to be grasped in the visible light image is marked with a marking frame corresponding to the type information to obtain at least one marked area, and a transition area connected to the at least one marked area is selected by sliding on an edge of the at least one marked area based on a preset sliding window;

[0062] Acquire a depth image of the area to be grasped, and intercept a target area in the depth image corresponding to the at least one marked area and the transition area to obtain a target image area corresponding to at least one object to be grasped;

[0063] Acquire three-dimensional point cloud data corresponding to the target image area;

[0064] The three-dimensional point cloud data is input into a pre-trained grasping strategy prediction model to obtain an optimal grasping strategy for the at least one object to be grasped, so that the robotic arm grasps the at least one object to be grasped according to the optimal grasping strategy, wherein the optimal grasping strategy includes optimal grasping position information and optimal grasping posture information.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot grasping and positioning method for a mobile robot having a grasping mechanical arm, characterized in that the method include: Acquire a visible light image of the area to be grasped, and recognize the visible light image based on a pre-trained image recognition model, wherein the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image; According to the position information, at least one object to be grasped in the visible light image is marked with a marking frame corresponding to the type information to obtain at least one marked area, and a transition area connected to the at least one marked area is selected by sliding on an edge of the at least one marked area based on a preset sliding window; Acquire a depth image of the area to be grasped, and intercept a target area in the depth image corresponding to the at least one marked area and the transition area to obtain a target image area corresponding to at least one object to be grasped; Acquire three-dimensional point cloud data corresponding to the target image area; The three-dimensional point cloud data is input into a pre-trained grasping strategy prediction model to obtain an optimal grasping strategy for the at least one object to be grasped, so that the robotic arm grasps the at least one object to be grasped according to the optimal grasping strategy, wherein the optimal grasping strategy includes optimal grasping position information and optimal grasping posture information.

2. A robot grasping and positioning method according to claim 1, characterized in that: Before the visible light image is recognized based on a pre-trained image recognition model and the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image, the method includes: The convolutional neural network is iteratively trained according to the visible light image samples to obtain the image recognition model, wherein the convolutional neural network includes a convolution layer, an activation layer, a pooling layer and a fully connected layer.

3. A robot grasping and positioning method according to claim 1, characterized in that: The step of labeling at least one object to be grasped in the visible light image according to the position information using a labeling frame corresponding to the type information to obtain at least one labeling area includes: Acquire contour information of the at least one object to be grasped according to the type information; At least one object to be grasped in the visible light image is positioned according to the position information, and the positioned at least one object to be grasped is framed according to the contour information, so as to obtain the at least one marked area containing only the object to be grasped.

4. A robot grasping and positioning method according to claim 1, characterized in that: The step of intercepting the target area corresponding to the at least one marked area and the transition area in the depth image to obtain the target image area corresponding to at least one object to be grasped comprises: Acquire a target size of the visible light image, and adjust the size of the depth image to be consistent with the target size; Based on the SIFT algorithm, feature points in the depth image are matched with feature points in the target transition area in the visible light image, and the depth image is intercepted according to the matching result to obtain a target image area corresponding to at least one object to be grasped.

5. A robot grasping and positioning method according to claim 1, characterized in that: The depth image is collected by an acquisition device; and obtaining the three-dimensional point cloud data corresponding to the target image area includes: Obtaining internal and external parameters of the acquisition device; Based on the intrinsic parameters and the extrinsic parameters, the image coordinates of the target image area are converted into a world coordinate system to obtain three-dimensional point cloud data corresponding to the target image area.

6. A robot grasping and positioning method according to claim 1, characterized in that: in, The grasping strategy prediction model is used to predict the probability of successful grasping when grasping different positions on the at least one object to be grasped using multiple grasping postures, and the optimal grasping strategy is the grasping strategy corresponding to the highest probability of successful grasping of the at least one object to be grasped; The strategy prediction model includes a probability prediction sub-model, and the probability prediction sub-model is used to predict the probability of successful grasping when grasping the at least one object to be grasped using each grasping strategy.

7. A robot grasping and positioning system for a mobile robot having a grasping mechanical arm, characterized in that: The system includes: a recognition module configured to obtain a visible light image of the area to be grasped, and recognize the visible light image based on a pre-trained image recognition model, wherein the image recognition model outputs type information and position information of at least one object to be grasped in the visible light image; a marking module configured to mark at least one object to be grasped in the visible light image according to the position information using a marking frame corresponding to the type information to obtain at least one marked area, and to slide on the edge of the at least one marked area based on a preset sliding window to select a transition area connected to the at least one marked area; A capture module is configured to obtain a depth image of the area to be captured, and to capture a target area in the depth image corresponding to the at least one marked area and the transition area, so as to obtain a target image area corresponding to at least one object to be captured; An acquisition module, configured to acquire three-dimensional point cloud data corresponding to the target image area; The grasping module is configured to input the three-dimensional point cloud data into a pre-trained grasping strategy prediction model to obtain an optimal grasping strategy for the at least one object to be grasped, so that the robotic arm grasps the at least one object to be grasped according to the optimal grasping strategy, wherein the optimal grasping strategy includes optimal grasping position information and optimal grasping posture information.

8. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images