Structured light-based industrial robot positioning method and device, controller, medium

Abstract

The application relates to a kind of industrial robot positioning method and device based on structured light, controller, medium, the method comprises: obtaining the two-dimensional image information of preset area, and the pixel region where target object is located is segmented from the two-dimensional image information of preset area;According to the mapping relationship between the two-dimensional image information and three-dimensional data of pre-set, the pixel region is converted into the three-dimensional data corresponding to the target object;According to the pose of the target object of three-dimensional data positioning target object is grasped to the target object.The application can realize the accurate and rapid scanning of the three-dimensional profile of target object in complex environment based on the visual guidance of industrial robot recognition positioning, and through the one-to-one mapping of three-dimensional data and two-dimensional image data, the pose of the target object is positioned efficiently and accurately, and the industrial robot is guided to grasp target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot vision, and in particular to an industrial robot positioning method and device based on structured light, a controller and a medium. BACKGROUND

[0002] With the rapid development of industrial automation, the application of industrial robots is becoming more and more common. For most industrial robot application scenarios in the prior art, manual teaching or offline programming is usually required to plan the working path of the robot in advance. This highly structured working mode strictly limits the flexibility and intelligence of the use of industrial robots.

[0003] The existing industrial robot positioning method can be divided into two categories according to the type of data: (1) recognizing the target by comparing with the template from the two-dimensional image, and then extracting the three-dimensional data from the image area of the recognized target or calculating the pose of the target by using the distance sensor to obtain the local plane of the target. This method is heavily dependent on the quality of the captured image, but the light in the industrial production environment is complex, so this method is difficult to adapt to the actual production. (2) Comparing the three-dimensional data directly with the CAD model. This method no longer depends on the quality of the obtained two-dimensional image, but for the case where multiple workpieces are stacked together, it is easy to cause ambiguity in registration, thereby affecting the stability of the template comparison.

[0004] In summary, the existing industrial robot positioning method is difficult to use in complex and variable industrial application environments and cannot meet the needs of flexible production. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an industrial robot positioning method and device based on structured light, a controller and a medium. The industrial robot recognition and positioning based on visual guidance can realize accurate and rapid scanning of the three-dimensional profile of the target object in a complex environment, and through the one-to-one mapping and analysis of three-dimensional data and two-dimensional image data, the pose of the target object can be positioned efficiently and accurately, and the industrial robot can be guided to grasp the target object.

[0006] In order to solve the above technical problems, according to the first embodiment of the present application, an industrial robot positioning method based on structured light is provided, comprising the following steps:

[0007] Obtaining two-dimensional image information of a preset area, and segmenting a pixel area where a target object is located from the two-dimensional image information of the preset area;

[0008] According to the mapping relationship between the preset two-dimensional image information and the three-dimensional data, converting the pixel area into three-dimensional data corresponding to the target object;

[0009] Positioning a pose of the target object according to the three-dimensional data of the target object so as to grasp the target object.

[0010] Further, segmenting a pixel region where the target object is located from the two-dimensional image information of the preset region includes the following steps:

[0011] Segmenting the pixel region where the target object is located from the two-dimensional image information by AI segmentation or image recognition.

[0012] Further, the method further includes: constructing a mapping relationship between the preset two-dimensional image information and the three-dimensional data, specifically including the following steps:

[0013] Using a structured light vision sensor to acquire the two-dimensional image information and the three-dimensional data of the target object, and forming a one-to-one mapping between the two-dimensional image information and the three-dimensional data, wherein the structured light vision sensor includes a light source and at least one camera.

[0014] Further, the using the structured light vision sensor to acquire the two-dimensional image information of the target object includes the following steps:

[0015] Projecting an image with a brightness exceeding a preset brightness to acquire two-dimensional image information after lighting.

[0016] Further, the light source is a grating projection device, and the using the structured light vision sensor to acquire the three-dimensional data of the target object includes the following steps:

[0017] Calibrating an internal parameter of each of the cameras and an external parameter between each of the cameras and the grating projection device;

[0018] The grating projection device projects an image and generates a grating sinusoidal image;

[0019] The camera captures the projected image to form a camera image;

[0020] Obtaining a principal value of a phase of the grating sinusoidal image;

[0021] Obtaining grating sinusoidal image coordinates according to the principal value of the phase and the camera image coordinates;

[0022] Correcting the camera image coordinates and the grating sinusoidal image coordinates;

[0023] Obtaining three-dimensional coordinates of the target object according to the corrected camera image coordinates, grating sinusoidal image coordinates, and the internal parameter and the external parameter.

[0024] Further, the positioning a pose of the target object according to the three-dimensional data of the target object so as to grasp the target object includes the following steps:

[0025] According to the three-dimensional data of the target object and a preset CAD model of the target object, the pose of the target object is located;

[0026] The pose of the target object is converted to a mechanical arm coordinate system to grasp the target object.

[0027] Further, the method further comprises: during the grasping of the target object, judging whether the grasping clamp interferes with the target object and the material frame where the target object is located, and if interference occurs, readjusting the position of the clamp and then grasping again.

[0028] Further, judging whether the grasping clamp interferes with the target object and the material frame where the target object is located comprises the following steps:

[0029] Collecting a three-dimensional image of the grasping clamp;

[0030] The three-dimensional image of the grasping clamp is divided into a plurality of cuboids, and each cuboid is used to describe the structural features of each part of the grasping clamp;

[0031] Judging whether each cuboid has an intersection point with each plane of the material frame, and if there is an intersection point, the cuboid interferes with the material frame;

[0032] By analyzing the pose of the target object, it is judged whether each cuboid has an intersection point with the target object, and if there is an intersection point, the cuboid interferes with the target object.

[0033] According to the second embodiment of the present application, an industrial robot positioning device based on structured light is provided, comprising:

[0034] An acquisition module is configured to acquire two-dimensional image information of a preset area, and segment a pixel area where a target object is located from the two-dimensional image information of the preset area;

[0035] A conversion module is configured to convert the pixel area into three-dimensional data corresponding to the target object according to a mapping relationship between the preset two-dimensional image information and the three-dimensional data;

[0036] A positioning module is configured to locate the pose of the target object according to the three-dimensional data of the target object to grasp the target object.

[0037] Further, the acquisition module is further configured to:

[0038] The two-dimensional image information is segmented into a pixel area where the target object is located through AI segmentation or image recognition.

[0039] Further, the apparatus further comprises a constructing module configured to construct a mapping relationship between the preset two-dimensional image information and the three-dimensional data.

[0040] The constructing module comprises:

[0041] a first obtaining unit configured to obtain two-dimensional image information of a target object by using a structured light vision sensor;

[0042] a second obtaining unit configured to obtain three-dimensional data of the target object by using the structured light vision sensor;

[0043] a mapping unit configured to form a one-to-one mapping between the two-dimensional image information and the three-dimensional data.

[0044] The structured light vision sensor comprises a light source and at least one camera.

[0045] Further, the first obtaining unit is specifically configured to project an image with a brightness exceeding a preset brightness to obtain two-dimensional image information after lighting.

[0046] Further, the light source is a grating projection device, and the second obtaining unit is specifically configured to obtain three-dimensional data of the target object by using the structured light vision sensor.

[0047] The second obtaining unit comprises:

[0048] a calibration sub-unit configured to calibrate an internal parameter of each camera and an external parameter between each camera and the grating projection device;

[0049] a first image obtaining sub-unit configured to project an image by using the grating projection device and generate a grating sinusoidal image;

[0050] a second image obtaining sub-unit configured to capture the projected image by using the camera to form a camera image;

[0051] a phase principal value obtaining sub-unit configured to obtain a phase principal value of the grating sinusoidal image;

[0052] a first coordinate obtaining sub-unit configured to obtain grating sinusoidal image coordinates according to the phase principal value and the camera image coordinates;

[0053] a correction sub-unit configured to correct the camera image coordinates and the grating sinusoidal image coordinates;

[0054] a second coordinate obtaining sub-unit configured to obtain three-dimensional coordinates of the target object according to the corrected camera image coordinates, the grating sinusoidal image coordinates, and the internal parameter and the external parameter.

[0055] Further, the positioning module comprises:

[0056] A pose acquisition unit is configured to locate a pose of the target object according to the three-dimensional data of the target object and a preset CAD model of the target object.

[0057] A coordinate conversion unit is configured to convert the pose of the target object to a robot coordinate system for grasping the target object.

[0058] Further, the device further comprises a detection module configured to determine whether the grasping clamp interferes with the target object and the frame during grasping the target object, and if interference occurs, the position of the clamp is readjusted for grasping again.

[0059] Further, the detection module comprises:

[0060] An image acquisition unit is configured to acquire a three-dimensional image of the grasping clamp.

[0061] A division unit is configured to divide the three-dimensional image of the grasping clamp into a plurality of cuboids, and each cuboid is configured to describe the structural features of each part of the grasping clamp.

[0062] A first determination unit is configured to determine whether each cuboid has an intersection with each plane of the frame, and if there is an intersection, the cuboid interferes with the frame.

[0063] A second determination unit is configured to determine whether each cuboid has an intersection with the target object by analyzing the pose of the target object, and if there is an intersection, the cuboid interferes with the target object.

[0064] According to the third embodiment of the present application, a controller is provided, which comprises a memory and a processor, the memory stores a computer program, and the program can implement the steps of the method when executed by the processor.

[0065] According to the fourth embodiment of the present application, a computer readable storage medium is provided for storing a computer program, and the program implements the steps of the method when executed by a computer or a processor.

[0066] Compared with the prior art, the present application has obvious advantages and beneficial effects. By the above technical scheme, the industrial robot positioning method and device based on structured light, the controller and the medium can achieve considerable technical progress and practicality, and have wide industrial utilization value, and at least have the following advantages:

[0067] The application is based on visual guidance of an industrial robot to recognize and position, can realize accurate and rapid scanning of a three-dimensional profile of a target object in a complex environment, and through one-to-one mapping and analysis of three-dimensional data and two-dimensional image data, the pose of the target object is positioned efficiently and accurately, the industrial robot is guided to grasp the target object, the complex and changeable industrial application environment is adapted, and the production efficiency of industrial assembly, feeding and discharging in the complex environment is improved; in addition, in the process of grasping the target object, whether the grasping clamp interferes with the target object and the material frame where the target object is located is analyzed, and corresponding adjustment is made, so that the industrial production of the industrial robot is more flexible and intelligent.

[0068] The above description is only a summary of the technical solutions of the application, in order to enable the technical means of the application to be implemented according to the content of the specification, and in order to enable the above and other purposes, characteristics and advantages of the application to be more obvious and easy to understand, the following preferred embodiments are described in detail, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 The schematic diagram of the industrial robot positioning method based on structured light provided by an embodiment of the application is shown.

[0070] Figure 2 The schematic diagram of the two camera space coordinate systems of an embodiment of the application is shown.

[0071] Figure 3 The schematic diagram of the industrial robot positioning device based on structured light provided by an embodiment of the application is shown.

[0072] SYMBOL DESCRIPTION

[0073] 1: acquisition module 2: conversion module

[0074] 3: positioning module DETAILED DESCRIPTION

[0075] In order to further illustrate the technical means and effects taken by the application to achieve the predetermined invention purpose, the specific embodiments of the industrial robot positioning method and device based on structured light, controller and medium according to the application and their effects are described in detail as follows.

[0076] The embodiment of the application provides an industrial robot positioning method based on structured light, as shown in Figure 1 The method comprises the following steps:

[0077] Step S1, acquiring two-dimensional image information of a preset area, and segmenting a pixel area where a target object is located from the two-dimensional image information of the preset area;

[0078] As an example, the step S1 further comprises the following steps:

[0079] The two-dimensional image information is segmented by AI (Artificial Intelligence) segmentation or image recognition to segment the pixel region where the target object is located. It should be noted that the AI segmentation or image recognition technology is an existing image processing technology, which will not be described here. The two-dimensional image information can include one or more target objects, and each target object can be separated by the AI segmentation or image recognition technology. In industrial applications, as an example, the preset region is the region inside the frame, and the target object is the object to be grabbed inside the frame. The object inside the frame can be called a workpiece, and the target object is a target workpiece.

[0080] Step S2, according to the mapping relationship between the preset two-dimensional image information and the three-dimensional data, the pixel region is converted into the three-dimensional data corresponding to the target object AI segmentation or image recognition can separate each target object corresponding to the three-dimensional data.

[0081] The method further comprises step S20, constructing the mapping relationship between the preset two-dimensional image information and the three-dimensional data, specifically comprising: using a structured light vision sensor to obtain two-dimensional image information and three-dimensional data of the target object, and forming a one-to-one mapping between the two-dimensional image information and the three-dimensional data. Wherein, the structured light vision sensor comprises a light source and at least one camera. As an example, the light source is a grating projection device. The following embodiments are described by taking the grating projection device as an example, but it can be understood that other light sources that can be applied to the embodiments of the present application to obtain two-dimensional image information and three-dimensional data can also be applicable. Using multiple cameras and grating projection devices in combination can reduce the blind area caused by the installation angle limitation of each camera and grating projection device, and improve the positioning accuracy. In the embodiments of the present application, two cameras are taken as an example to illustrate the structured light vision sensor.

[0082] Step S20, using a structured light vision sensor, can simultaneously obtain two-dimensional image information and three-dimensional data of the target, and form a one-to-one mapping between the two, so that when positioning the target object, both two-dimensional image information and three-dimensional data of the target object can be used, thereby adapting to complex environments and improving the robustness of target object positioning, making the industrial robot positioning more flexible and intelligent.

[0083] In step S20, the two-dimensional image information of the target object obtained by the structured light vision sensor comprises the following steps:

[0084] Step S200, projecting an image with brightness exceeding a preset brightness to obtain a two-dimensional image information after lighting, and using binocular vision to cooperate with a grating structured light to capture a high-quality two-dimensional image by itself lighting, wherein the preset brightness is set according to the projection requirement, and as an example, the image with brightness exceeding the preset brightness can be a white image, a blue image, etc.

[0085] In step S20, the three-dimensional data of the target object is acquired by using the structured light vision sensor, including the following steps.

[0086] Step S201, calibrating the internal parameters of each camera and the external parameters between each camera and the grating projection device;

[0087] Step S202, the grating projection device projects an image and generates a grating sinusoidal image;

[0088] The gray scale distribution of the grating sinusoidal image is:

[0089] I(u, v) = a + b * cos(θ(u, v)), θ(u, v) = φ(u, v) + α

[0090] Where (u, v) represents the coordinates of the pixel unit of the projection plane, I(u, v) is the gray value of the (u, v) point, a and b are the direct current fundamental component (light intensity background value) and amplitude (modulated light intensity) of the sinusoidal grating respectively, θ(u, v) is the grating phase corresponding to I(u, v), φ is the main value to be solved, and α is the phase displacement.

[0091] Step S203, the camera captures the projection image to form a camera image;

[0092] Step S204, acquiring the main value of the phase of the grating sinusoidal image;

[0093] The main value of the phase is the relative phase value, and as an example, the standard four-step phase shift method can be used to calculate the main value of the phase of the grating image, but it can be understood that other methods for calculating the main value of the phase are also applicable.

[0094] A phase main value image is calculated by using four grating images with the same frequency, and the light intensity expression of the four grating images is:

[0095] I i (u, v) = a + b * cos(θ i (u, v)) θ i (u, v) = φ(u, v) + π / 2*i, i ∈ {0, 1, 2, 3}

[0096] Then the main value of the phase of the grating image is:

[0097] φ(u, v) = atan2(I3 - I1, I0 - I2)

[0098] Step S205, obtaining the grating sinusoidal image coordinates according to the phase principal value and the camera image coordinates;

[0099] According to the absolute phase value of one direction, a vertical line or a horizontal line in the projection image can be determined; and according to the absolute phase values of the horizontal and vertical directions, a point in the projection image can be determined.

[0100] Through the point in the camera image of the vertical and horizontal grating fringes, the point in the camera image is set as (uc, vc), a corresponding point in the projection image, i.e. the grating sinusoidal image coordinate point, can be determined, and the grating sinusoidal image coordinate point is set as (up, vp). The point coordinates can be calculated by the following formula.

[0101]

[0102] wherein Φv and Φh are the absolute phase values of the vertical and horizontal directions at the point (uc, vc) in the grating image, Nv and Nh are the grating fringe numbers in the vertical and horizontal camera grating images, and H and W are the vertical and horizontal resolution of the camera grating image.

[0103] Step S206, correcting the camera image coordinates and the grating sinusoidal image coordinates;

[0104] As an example, the system distortion parameters calibrated in advance are used to correct (uc, vc) and (up, vp), and the second-order radial distortion correction formula is as follows:

[0105] u - = u + (u - u0) [k1(x 2 +y 2 ) + k2(x 2 +y 2 ) 2 ]

[0106] v - = v + (v - v0) [k1(x 2 +y 2 ) + k2(x 2 +y 2 ) 2 ]

[0107] Step S207, obtaining the three-dimensional coordinates of the target object according to the corrected camera image coordinates, the grating sinusoidal image coordinates, and the intrinsic parameters and extrinsic parameters.

[0108] According to the corrected camera image coordinates (uc, vc) and the grating sinusoidal image coordinates (up, vp), and the intrinsic and extrinsic parameters obtained by the camera calibration, a line between the camera optical center and the image point is calculated, and then the corresponding three-dimensional coordinates are calculated by using the above information and the principle of triangulation.

[0109]

[0110] wherein Sc and Sp are the scale factors of the camera and the optical machine respectively, Ac and Ap are the intrinsic parameter matrices, [Rc tc] and [Rptp] are the extrinsic parameter matrices, and Xw, Yw, Zw, 1 are the spatial vectors.

[0111] The following describes in detail the calculation of the corresponding three-dimensional coordinates by the principle of triangulation:

[0112] Figure 2 wherein 01-xyz and 02-xyz are the spatial coordinate systems of the two cameras respectively; P1 and P2 are a pair of homonymous points; S1 and S2 are the center positions of the camera lenses; and w is a point in the real space. P1 and S1 determine a straight line in the space, and P2 and S2 determine another straight line, which intersect at W in the space.

[0113] Spatial straight line: after the camera captures an image, a point on the camera CCD and the center of the camera lens can determine a straight line, such as Figure 2 The coordinates of the two points are in the camera coordinate system, and the spatial straight line equation composed of the two points is:

[0114]

[0115]

[0116] wherein X, Y, and Z are the three-dimensional coordinates of the target point, which are unknown numbers; x, y, and f are the image point coordinates, which are known quantities (obtained by analyzing the image); Xs, Ys, and Zs are the lens center coordinates, which are known quantities (obtained in the camera calibration process); a i , b i , c i (i = 1, 2, 3) are the coordinate system transformation parameters, which are known quantities (obtained in the camera calibration process). One image can form one straight line equation, and two images can form two straight line equations, a total of four equations, while the unknown numbers in the formula are only three (three-dimensional point coordinates X, Y, and Z), so the three unknown numbers can be calculated.

[0117] Step S3: positioning the pose of the target object according to the three-dimensional data of the target object to grasp the target object.

[0118] As an example, the step S3 includes the following steps:

[0119] Step S31, positioning the pose of the target object according to the three-dimensional data of the target object and the preset CAD model of the target object;

[0120] Each target object has a corresponding CAD model, and is preset. By comparing the three-dimensional data of the target object and the corresponding CAD model, the pose of the target object can be positioned.

[0121] Step S32, converting the pose of the target object to the coordinate system of the manipulator to grasp the target object.

[0122] The method corresponds to a system which can include an industrial robot, a host computer and a three-dimensional vision sensor, and the three-dimensional vision sensor is a structured light vision sensor. Compared with the existing method which simply relies on two-dimensional images and three-dimensional data, the method can not only solve the problem that it is difficult to calculate the three-dimensional coordinates of the target object by simply relying on two-dimensional image analysis, but also avoid the problem that it is difficult to effectively process the segmentation of the target object when it is closely arranged by simply relying on three-dimensional data. The method can realize accurate and rapid scanning of the three-dimensional profile of the target object in a complex environment by executing steps S1-S3, and efficiently and accurately position the pose of the target object through the mapping relationship between the three-dimensional data and the two-dimensional image, thereby guiding the industrial robot to grasp the workpiece.

[0123] In the application of structured light guiding industrial robot identification and positioning, the analysis of the interference between the clamp and the material frame and the clamp and the workpiece during the grasping process is an important research content and a problem to be solved. Due to the diversity of the pose of the workpiece in the material frame, the design of the clamp is irregular. Based on this, the method further includes step S4, judging whether the grasping clamp interferes with the target object and the material frame where the target object is located during grasping the target object. If interference occurs, the position of the clamp is adjusted again for grasping.

[0124] In step S4, judging whether the grasping clamp interferes with the target object and the material frame where the target object is located includes the following steps:

[0125] Step S41, acquiring a three-dimensional image of the grasping clamp;

[0126] Step S42, dividing the three-dimensional image of the grasping clamp into a plurality of cuboids, each of which is used to describe the structural features of each part of the grasping clamp;

[0127] Step S43, judging whether each of the cuboids has a point of intersection with each plane of the material frame. If there is, the cuboid interferes with the material frame.

[0128] Step S44: By analyzing the pose of the target object, determine whether each cuboid intersects with the target object. If so, the cuboid interferes with the target object.

[0129] Step S4 enables the industrial robot to avoid interference between the fixture, the material frame, and the workpiece during the workpiece gripping process, while ensuring that the workpiece is gripped to the maximum extent.

[0130] This invention also provides an industrial robot positioning device based on structured light, such as... Figure 3 As shown, the device includes an acquisition module 1, a conversion module 2, and a positioning module 3. The acquisition module 1 acquires two-dimensional image information of a preset area and segments the pixel region containing the target object from the two-dimensional image information of the preset area. The conversion module 2 converts the pixel region into three-dimensional data corresponding to the target object based on a preset mapping relationship between two-dimensional image information and three-dimensional data. The positioning module 3 locates the pose of the target object based on the three-dimensional data of the target object, thereby grasping the target object. Compared with existing methods that rely solely on two-dimensional images and three-dimensional data, the device described in this embodiment not only solves the problem of difficulty in calculating the three-dimensional coordinates of the target object by relying solely on two-dimensional image analysis, but also avoids the problem of ineffective segmentation when target objects are closely arranged by relying solely on three-dimensional data. The device can achieve accurate and rapid scanning of the three-dimensional contour of the target object in complex environments, and efficiently and accurately locates the pose of the target object through the mapping relationship between three-dimensional data and two-dimensional images, thereby guiding an industrial robot to grasp the workpiece.

[0131] As an example, module 1 is further configured to segment the pixel region where the target object is located by using AI segmentation or image recognition to segment the two-dimensional image information. It should be noted that AI segmentation or image recognition technology is an existing image processing technique and will not be elaborated upon here. The two-dimensional image information may include one or more target objects, and the pixel region corresponding to each target object can be separated using AI segmentation or image recognition technology. In industrial applications, as an example, the preset area is the area within a material frame, the target object is the object to be grasped within the material frame, the object within the material frame can be called a workpiece, and the target object is the target workpiece.

[0132] The device also includes a construction module for constructing the mapping relationship between the preset two-dimensional image information and the three-dimensional data.

[0133] The construction module comprises a first acquisition unit, a second acquisition unit and a mapping unit, wherein the first acquisition unit is configured to acquire two-dimensional image information of a target object by using a structured light vision sensor; the second acquisition unit is configured to acquire three-dimensional data by using the structured light vision sensor; and the mapping unit is configured to form one-to-one mapping between the two-dimensional image information and the three-dimensional data, so that both the two-dimensional image information and the three-dimensional data can be used when the target object is positioned, thereby adapting to a complex environment, improving the robustness of target object positioning, and making the industrial robot positioning more flexible and intelligent. The structured light vision sensor comprises a light source and at least one camera. As an example, the light source is a grating projection device. The following embodiments are described by taking the grating projection device as an example, but it can be understood that other light sources applicable to the embodiments of the present application to acquire two-dimensional image information and three-dimensional data can also be applicable. The use of a combination of multiple cameras and grating projection devices can reduce the blind area caused by the installation angle limitation of each camera and grating projection device, and improve the positioning accuracy. In the embodiments of the present application, two cameras are taken as an example for description.

[0134] As an example, the first acquisition unit is specifically configured to project an image with a brightness exceeding a preset brightness, acquire two-dimensional image information after lighting, and use binocular vision combined with grating structured light to capture high-quality two-dimensional images by lighting itself. The preset brightness is set according to the projection requirement. As an example, the image with a brightness exceeding the preset brightness can be a white image, a blue image, etc. The second acquisition unit is specifically configured to acquire three-dimensional data of the target object by using the structured light vision sensor. The second acquisition unit comprises a calibration subunit, a first image acquisition subunit, a second image acquisition subunit, a phase main value acquisition subunit, a first coordinate acquisition subunit, a correction subunit and a second coordinate acquisition subunit, wherein,

[0135] The calibration subunit is configured to calibrate the internal parameters of each camera and the external parameters between each camera and the grating projection device;

[0136] The first image acquisition subunit is configured to project an image by the grating projection device and generate a grating sinusoidal image;

[0137] The gray scale distribution of the grating sinusoidal image is:

[0138] I(u, v) = a + b * cos(θ(u, v)), θ(u, v) = φ(u, v) + α

[0139] Wherein, (u, v) represents the coordinates of the projection plane pixel unit, I (u, v) is the gray value of (u, v) point, a and b are the direct current basic wave component (light intensity background value) and amplitude (modulated light intensity) of the sinusoidal grating respectively, θ (u, v) is the grating phase corresponding to I (u, v), φ is the phase principal value to be solved, and α is the phase displacement.

[0140] The second image acquisition subunit is configured to capture the camera image by shooting the projection image through the camera.

[0141] The phase principal value acquisition subunit is configured to acquire the phase principal value of the grating sinusoidal image.

[0142] One phase principal value image is calculated by using four grating images with the same frequency, and the light intensity expression of the four grating images is as follows:

[0143] I i (u, v) = a + b * cos (θ i (u, v)), θ i (u, v) = φ (u, v) + π / 2 * i, i ∈ {0, 1, 2, 3}

[0144] Then the phase principal value of the grating image is as follows:

[0145] φ (u, v) = atan2 (I3-I1, I0-I2)

[0146] The first coordinate acquisition subunit is configured to acquire the grating sinusoidal image coordinates according to the phase principal value and the camera image coordinates.

[0147] According to the absolute phase value in one direction, a vertical line or a horizontal line in the projection image can be determined; then according to the absolute phase values in the horizontal and vertical directions, a point in the projection image can be determined.

[0148] Through the point in the camera image of the vertical and horizontal grating stripes, the point in the camera image is set as (uc, vc), a corresponding point in the projection image, that is, the grating sinusoidal image coordinate point, can be determined, and the grating sinusoidal image coordinate point is set as (up, vp). The point coordinates can be calculated by the following formula.

[0149]

[0150] Wherein, Φv and Φh are the absolute phase values of the vertical and horizontal directions of the (uc, vc) point in the grating image respectively, Nv and Nh are the grating stripe numbers in the vertical and horizontal camera grating images respectively, and H and W are the vertical and horizontal resolution of the camera grating image respectively.

[0151] The correction subunit is configured to correct the camera image coordinates and the grating sinusoidal image coordinates.

[0152] As an example, the pre-calibrated system distortion parameters are used to correct (uc, vc) and (up, vp), and the second-order radial distortion correction formula is as follows:

[0153] u - = u + (u - u0) [k1(x 2 +y 2 ) + k2(x 2 +y 2 ) 2 ]

[0154] v - = v + (v - v0) [k1(x 2 +y 2 ) + k2(x 2 +y 2 ) 2 ]

[0155] The second coordinate acquisition subunit is configured to acquire three-dimensional coordinates of the target object according to the corrected camera image coordinates, the grating sinusoidal image coordinates, and the intrinsic parameters and the extrinsic parameters.

[0156] According to the corrected camera image coordinates (uc, vc) and the grating sinusoidal image coordinates (up, vp), and the pre-calibrated intrinsic parameters and extrinsic parameters, a line between the camera optical center and the image point is calculated, and then the corresponding three-dimensional coordinates are calculated by using the above information and the triangulation principle.

[0157]

[0158] wherein Sc and Sp are scale factors of the camera and the optical machine respectively, Ac and Ap are intrinsic parameter matrices, [Rc tc] and [Rptp] are extrinsic parameter matrices, and Xw, Yw, Zw, 1 are space vectors.

[0159] The calculation of the corresponding three-dimensional coordinates by the triangulation principle is described in detail as follows:

[0160] Figure 2 wherein 01-xyz and 02-xyz are two camera space coordinate systems respectively; P1 and P2 are a pair of homonymous points; S1 and S2 are the center positions of the camera lenses; and w is a point in the real space. P1 and S1 determine a straight line in the space, and P2 and S2 determine another straight line, which intersect at W in the space.

[0161] Space straight line: after the camera captures an image, a point on the camera CCD and the camera lens center can determine a straight line, such as Figure 2 The coordinates of the two points are in the camera coordinate system, and the space straight line equation composed of the two points is:

[0162]

[0163]

[0164] wherein X, Y, Z are three-dimensional coordinates of the target point, are unknowns; x, y, f are image point coordinates, are known quantities (obtained by analyzing the image); Xs, Ys, Zs are lens center coordinates, are known quantities (obtained in the camera calibration process); a i , b i , c i (i = 1, 2, 3) are coordinate system transformation parameters, are known quantities (obtained in the camera calibration process). One image can form a linear equation, two images can form two linear equations, a total of 4 equations, while the unknowns in the formula are only three (three-dimensional point coordinates X, Y, Z), therefore, the three unknowns can be calculated.

[0165] As an example, the positioning module 3 includes a pose acquisition unit and a coordinate conversion unit, wherein the pose acquisition unit is configured to locate the pose of the target object according to the three-dimensional data of the target object and the pre-set CAD model of the target object; the coordinate conversion unit is configured to convert the pose of the target object to the mechanical hand coordinate system to grasp the target object. Each target object has its corresponding CAD model, which is pre-set. By comparing the three-dimensional data of the target object and its corresponding CAD model, the pose of the target object can be located.

[0166] In the application of structure light guided industrial robot recognition and positioning, the analysis of the interference between the clamp and the material frame and the clamp and the workpiece during the grasping process is an important research content and a problem to be solved. Due to the diversity of the pose of the workpiece in the material frame, the irregularity of the clamp design is caused. Based on this, the device further includes a detection module configured to determine whether the grasping clamp interferes with the target object and the material frame where the target object is located during the grasping process of the target object. If interference occurs, the position of the clamp is adjusted again for grasping.

[0167] The detection module comprises an image acquisition unit, a division unit, a first judgment unit and a second judgment unit, wherein the image acquisition unit is configured to acquire the stereoscopic image of the grabbing fixture; the division unit is configured to divide the stereoscopic image of the grabbing fixture into a plurality of cuboids, each of which is configured to describe the structural features of each part of the grabbing fixture; the first judgment unit is configured to determine whether each of the cuboids has an intersection with each plane of the material frame, and if so, the cuboid interferes with the material frame; and the second judgment unit is configured to determine whether each of the cuboids has an intersection with the target object by analyzing the pose of the target object, and if so, the cuboid interferes with the target object. Through the detection module, the industrial robot can not only avoid interference between the fixture, the material frame and the workpiece during the grabbing process of the workpiece, but also ensure that the workpiece is grabbed to the greatest extent.

[0168] The method and device of the present application can realize accurate and rapid scanning of the three-dimensional profile of the target object in a complex environment, and accurately and efficiently locate the pose of the target object through one-to-one mapping and analysis of three-dimensional data and two-dimensional image data, so as to guide the industrial robot to grab the target object, adapt to complex and variable industrial application environment, and further improve the production efficiency of industrial assembly, feeding and discharging in a complex environment. In addition, during the grabbing of the target object, whether the grabbing fixture interferes with the target object and the material frame where the target object is located is analyzed, and corresponding adjustment is made, so that the industrial production using the industrial robot is more flexible and intelligent.

[0169] The embodiment of the present application also provides a controller comprising a memory and a processor, wherein the memory stores a computer program, and the program can realize the steps of the structure light-based industrial robot positioning method when executed by the processor.

[0170] The embodiment of the present application also provides a computer readable storage medium for storing a computer program, and the program realizes the steps of the structure light-based industrial robot positioning method when executed by a computer or a processor.

[0171] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the present application, and any simple modification, equivalent change and modification of the above embodiment based on the technical essence of the present application are still within the scope of the technical solution of the present application.

Claims

1. A structured light based industrial robot positioning method, characterized in that, The method comprises the following steps: acquiring two-dimensional image information of a preset area in an industrial production environment, and segmenting a pixel area where a target object is located from the two-dimensional image information of the preset area; The industrial production environment at least has varying light; According to a preset mapping relationship between the two-dimensional image information and the three-dimensional data, the pixel area is converted into three-dimensional data corresponding to the target object; A structured light vision sensor is used to acquire two-dimensional image information and three-dimensional data of the target object, and a one-to-one mapping is formed between the two-dimensional image information and the three-dimensional data, wherein the structured light vision sensor comprises a light source and at least one camera; the structured light vision sensor is used to acquire two-dimensional image information of the target object, which comprises projecting an image with a brightness higher than a preset brightness to acquire two-dimensional image information after lighting; The pose of the target object is located according to the three-dimensional data of the target object, so that the target object is grasped; the pose of the target object is located according to the three-dimensional data of the target object and a preset CAD model of the target object; the pose of the target object is converted into a mechanical hand coordinate system to grasp the target object; During the grasping of the target object, it is judged whether the grasping clamp interferes with the target object and a material frame where the target object is located, if interference occurs, the position of the clamp is readjusted and the grasping is performed again; wherein the judgment of whether the grasping clamp interferes with the target object and the material frame where the target object is located comprises the following steps: collecting a three-dimensional image of the grasping clamp; dividing the three-dimensional image of the grasping clamp into a plurality of cuboids, each of which is used to describe the structural features of each part of the grasping clamp; it is judged whether each of the cuboids has an intersection point with each plane of the material frame, if there is an intersection point, the cuboid interferes with the material frame; by analyzing the pose of the target object, it is judged whether each of the cuboids has an intersection point with the target object, if there is an intersection point, the cuboid interferes with the target object; The light source is a grating projection device, and the structured light vision sensor is used to acquire three-dimensional data of the target object, which comprises the following steps: Calibrating the internal parameters of each camera and the external parameters between each camera and the grating projection device; The grating projection device projects an image and generates a grating sinusoidal image; the gray scale distribution of the grating sinusoidal image is I(u,v)=a+b*cos(θ(u,v)), θ(u,v)=φ(u,v)+α; wherein (u,v) represents the coordinates of the pixel unit of the projection plane, I(u,v) is the gray scale value of the (u,v) point, a and b are the direct current fundamental component (light intensity background value) and amplitude (modulated light intensity) of the sinusoidal grating respectively, θ(u,v) is the grating phase corresponding to I(u,v), φ is the principal value of the to-be-solved phase, and α is the phase displacement; The camera shoots the projection image to form a camera image; The phase principal value of the grating sinusoidal image is obtained; a phase principal value image is calculated by using four grating images with the same frequency, and the intensity expression of the four grating images is as follows: i (u,v) = a + b*cos(theta i (u,v)), theta i (u,v) = phi(u,v) + pi / 2*i, i belongs to {0,1,2,3}; and the phase principal value of the grating image is phi(u,v) = atan2(I3-I1,I0-I2). According to the phase principal value and the camera image coordinates, the grating sinusoidal image coordinates (up,vp) are acquired: Wherein, Φv and Φh are absolute phase values of the (uc, vc) point in the vertical and horizontal direction of the grating image, Nv and Nh are the number of grating fringes in the vertical and horizontal camera grating image, H and W are the vertical and horizontal resolution of the camera grating image, respectively; Correct the camera image coordinates and the grating sinusoidal image coordinates; correct (uc, vc) and (up, vp) using the system distortion parameters calibrated in advance by the following formula: u = u + (u - u0)[k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 ]; v = v + (v - v0)[k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 ]; According to the corrected camera image coordinates, grating sinusoidal image coordinates and the internal and external parameters, obtain the three-dimensional coordinates of the target object: Wherein, Sc and Sp are the scale factors of the camera and the light machine, Ac and Ap are the internal parameter matrix, [Rctc] and [Rptp] are the external parameter matrix, Xw, Yw, Zw, 1 are the space vector.

2. The structured light-based industrial robot positioning method according to claim 1, wherein, The pixel region where the target object is located is segmented from the two-dimensional image information of the preset area, including the following steps: The pixel region where the target object is located is segmented from the two-dimensional image information by AI segmentation or image recognition.

3. A structured light based industrial robot positioning device, characterized in that, It includes: The acquisition module is used for acquiring two-dimensional image information of a preset area in an industrial production environment, and segmenting a pixel region where a target object is located from the two-dimensional image information of the preset area; The industrial production environment at least has changing light; The conversion module is used for converting the pixel region into three-dimensional data corresponding to the target object according to a preset mapping relationship between the two-dimensional image information and the three-dimensional data; The positioning module is used for positioning the pose of the target object according to the three-dimensional data of the target object so as to grasp the target object; The positioning module includes: a pose acquisition unit, which is used for positioning the pose of the target object according to the three-dimensional data of the target object and a preset CAD model of the target object; and a coordinate conversion unit, which is used for converting the pose of the target object to a mechanical hand coordinate system to grasp the target object; The detection module is used for judging whether the grasping clamp interferes with the target object and the material frame where the target object is located during grasping the target object, and if interference occurs, the position of the clamp is adjusted again for grasping; wherein, the detection module includes: an image acquisition unit, which is used for acquiring a three-dimensional image of the grasping clamp; a division unit, which is used for dividing the three-dimensional image of the grasping clamp into a plurality of cuboids, each of which is used to describe the structural features of each part of the grasping clamp; a first judgment unit, which is used for judging whether each cuboid has an intersection point with each plane of the material frame, and if there is an intersection point, the cuboid interferes with the material frame; and a second judgment unit, which is used for judging whether each cuboid has an intersection point with the target object by analyzing the pose of the target object, and if there is an intersection point, the cuboid interferes with the target object. The device also comprises a construction module for constructing a mapping relationship between the preset two-dimensional image information and three-dimensional data; the construction module comprises: a first acquisition unit for acquiring two-dimensional image information of a target object using a structured light vision sensor; a second acquisition unit for acquiring three-dimensional data using a structured light vision sensor; a mapping unit for forming a one-to-one mapping between the two-dimensional image information and the three-dimensional data; wherein the structured light vision sensor comprises a light source and at least one camera; the first acquisition unit is specifically configured to project an image with a luminance higher than a preset luminance to acquire two-dimensional image information after lighting; The light source is a grating projection device, and the second acquisition unit is specifically configured to acquire three-dimensional data of a target object using a structured light vision sensor; the second acquisition unit comprises: a calibration subunit for calibrating the internal parameters of each camera and the external parameters between each camera and the grating projection device; a first image acquisition subunit for projecting an image by the grating projection device and generating a grating sinusoidal image; the grating sinusoidal image has a gray scale distribution of I(u,v)=a+b*cos(θ(u,v)), θ(u,v)=φ(u,v)+α; wherein (u,v) represents the coordinates of a pixel unit of a projection plane, I(u,v) is the gray scale value of the (u,v) point, a and b are respectively a direct current fundamental wave component (light intensity background value) and an amplitude (modulated light intensity) of the sinusoidal grating, θ(u,v) is a grating phase corresponding to I(u,v), φ is a main value to be solved, and α is a phase shift; a second image acquisition subunit for capturing the projection image by the camera to form a camera image; A phase principal value obtaining subunit is configured to obtain a phase principal value of the grating sinusoidal image; one phase principal value image is calculated by using four grating images with the same frequency, and the light intensity expression of the four grating images is as follows: i (u,v) = a + b*cos(theta i (u,v)), theta i (u,v) = phi(u,v) + pi / 2*i, i element of {0,1,2,3}; and the phase principal value of the grating image is phi(u,v) = atan2(I3-I1,I0-I2). a first coordinate acquisition subunit for acquiring grating sinusoidal image coordinates (up,vp) according to the main value and the camera image coordinates: wherein Φv and Φh are respectively absolute phase values of a (uc,vc) point in a vertical direction and a horizontal direction of a grating image, Nv and Nh are respectively the number of grating stripes in a vertical camera grating image and a horizontal camera grating image, and H and W are respectively the resolution of the vertical camera grating image and the horizontal camera grating image; a correction subunit for correcting the camera image coordinates and the grating sinusoidal image coordinates; the (uc,vc) and (up,vp) are corrected by using system distortion parameters calibrated in advance according to the following formula: u = u + (u - u0)[k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 ]; v = v + (v - v0)[k1(x 2 +y 2 )+k2(x 2 +y 2 ) 2 ]; a second coordinate acquisition subunit for acquiring three-dimensional coordinates of a target object according to the corrected camera image coordinates, the grating sinusoidal image coordinates, and the internal parameters and the external parameters: wherein Sc and Sp are respectively scale factors of the camera and the light machine, Ac and Ap are internal parameter matrices, [Rctc] and [Rptp] are external parameter matrices, and Xw,Yw,Zw,1 are space vectors.

4. The structured light-based industrial robot positioning device according to claim 3, wherein the acquisition module is further configured to: segment a pixel region where the target object is located by AI segmentation or image recognition on the two-dimensional image information.

5. A controller comprising a memory and a processor, the controller characterized by: The memory stores a computer program which, when executed by the processor, is able to implement the steps of the method according to any one of claims 1 to 2.

6. A computer readable storage medium for storing a computer program, characterized in that: The program, when executed by a computer or processor, implements the steps of the method according to any one of claims 1 to 2.

Images