Rectangular label-based visual positioning method

Abstract

The application discloses a visual positioning method based on a rectangular label, which comprises the following steps: A, a robot pre-processes an image collected by a camera of the robot, and searches for a graphic attribute of the rectangular label and a vertex of the rectangular label in the pre-processed image; B, according to the graphic attribute of the rectangular label and the vertex of the rectangular label, the distance between the rectangular label and the camera, the deviation angle of the rectangular label relative to the camera, and the inclination angle of the plane where the rectangular label is located are calculated by using a monocular distance measuring principle, so that in the case that the plane where the rectangular label is located is not parallel to the pinhole plane of the camera, the distance between the rectangular label and the camera is the distance from the camera to the side of the rectangular label which is parallel to the target intersection line; and then the distance between the rectangular label and the camera and the deviation angle of the rectangular label relative to the camera are set as the visual positioning result of the robot to the device to be positioned or the rectangular label.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of monocular vision pose measurement, in particular to a vision positioning method based on a rectangular label. BACKGROUND

[0002] Vision ranging is one of the important technologies in the field of robots, and has wide applications in vision positioning, target tracking, vision obstacle avoidance, etc. Commonly used vision ranging methods include monocular ranging. Monocular ranging has simple structure and fast operation speed, thus having broad application prospects.

[0003] The monocular ranging technology disclosed in the prior art generally simplifies the monocular vision system into a camera projection model, establishes a ranging model through geometric relationship derivation, obtains the conversion relationship between the image coordinates and the world coordinate system, and finally realizes the calculation of the distance of the obstacle through geometric relationship (including the pinhole imaging model and the geometric proportional relationship of similar triangles). However, this scheme needs manual alignment of the camera optical axis, and the focal length of the monocular camera is fixed. When the robot collects image information of the same device component from different angles at different times, the angle change of the line connecting the two actual position points collected by the monocular camera relative to the imaging plane of the monocular camera will cause ranging errors. SUMMARY

[0004] The present application proposes a vision positioning method based on a rectangular label, and the specific technical solutions are as follows:

[0005] The vision positioning method based on a rectangular label includes the following steps: step A, the robot pre-processes the image collected by the camera, and then searches for the graphical attributes of the rectangular label and the vertices of the rectangular label in the pre-processed image; step B, according to the graphical attributes of the rectangular label and the vertices of the rectangular label, the distance between the rectangular label and the camera, the deviation angle of the rectangular label relative to the camera, and the inclination angle of the plane where the rectangular label is located are calculated by using the monocular ranging principle, so that in the case that the plane where the rectangular label is located is not parallel to the pinhole plane of the camera, the distance between the rectangular label and the camera is the distance from the camera to the side of the rectangular label that is parallel to the target intersection line; then the distance between the rectangular label and the camera, and the deviation angle of the rectangular label relative to the camera are set as the vision positioning result of the robot to the device to be positioned or the rectangular label; wherein the intersection line of the plane where the rectangular label is located and the pinhole plane of the camera is denoted as the target intersection line; wherein the rectangular label is arranged on the surface of the device to be positioned.

[0006] Compared with the prior art, the robot uses the identified vertexes of the rectangular label to calculate the distance and angle information of the rectangular label relative to the same camera, and then predicts the relative position relationship between the rectangular label and the same camera in the case of suppressing camera distortion, thereby providing more accurate positioning information for robot navigation back to the device to be positioned. Moreover, instead of staying at the same position for too much time to collect multiple frames of images, the distance and angle information of the identified rectangular label is calculated within the same frame of image collected by using monocular ranging principle, which takes into account the influence of camera ranging error and reduces the real-time positioning calculation amount.

[0007] Further, the visual positioning method further comprises: determining the case of distortion of the rectangular label in the camera based on the inclination angle of the plane where the rectangular label is located, so that when the distance between two adjacent vertexes in the edge line of the rectangular label is less than a preset distortion error value, the line connecting the two adjacent vertexes in the edge line of the rectangular label is configured not to be distorted in the camera.

[0008] Further, in the case that the plane where the rectangular label is located is parallel to the pinhole plane of the camera, the robot determines that any edge of the rectangular label is not distorted in the camera. Therefore, the distance between the rectangular label and the camera can be directly calculated by calling the pinhole imaging model, saving the participation of distortion parameters, and ensuring and improving the camera ranging efficiency and accuracy without going through distortion parameters or the distortion formula commonly used in the prior art.

[0009] Further, in the step A, in the process of searching for the graphical attributes of the rectangular label and the vertexes of the rectangular label in the preprocessed image, the robot detects the number of edges surrounding the closed graph; wherein the graphical attributes of the rectangular label include the edge line features of the closed graph, and the edge line features of the closed graph include the number of edges surrounding the closed graph and the number of vertexes of the closed graph; in the edge line of the closed graph, the robot identifies the line connecting two adjacent vertexes as an edge surrounding the closed graph; the robot counts the number of edges of a single closed graph in the preprocessed image, and when the robot counts that the number of edges surrounding the closed graph is four and there are two groups of mutually perpendicular edges, the currently detected closed graph is identified as a rectangular label; wherein each group of edges is provided with two mutually parallel edges. Thus, the shape of the rectangular label and the distribution characteristics of the internal pixel points are analyzed in combination with the graphical attributes, and labels of different shapes are distinguished.

[0010] Further, the method for calculating the distance between the rectangular label and the camera by using the pinhole imaging model comprises: the robot sets two vertexes in a rectangular label which are distributed along a target straight line direction as two adjacent target detection points in a to-be-positioned device plane; the line connecting the two adjacent target detection points is parallel to the target straight line direction; the to-be-positioned device plane is used to represent the surface of the to-be-positioned device; then among the two adjacent target detection points, the robot records one of the target detection points as a first detection point, and records the distance d1 from the camera to the to-be-measured edge where the first detection point is located; the robot records the other target detection point as a second detection point, and records the distance d2 from the camera to the to-be-measured edge where the second detection point is located; wherein the edge in the rectangular label which is perpendicular to the target straight line direction is the to-be-measured edge of the rectangular label; the distance between the rectangular label and the camera in step B comprises the distance from the camera to the to-be-measured edge where the first detection point is located and the distance from the camera to the to-be-measured edge where the second detection point is located; the lens focal length f of the camera, the edge length w of the to-be-measured edge and the pixel width p of the to-be-measured edge formed in the imaging plane of the camera are obtained in advance; the distance between the to-be-measured edge and the camera is calculated by the following formula:

[0011]

[0012] If one end point of the to-be-measured edge is the first detection point, the to-be-measured edge is the to-be-measured edge where the first detection point is located, and then d is set to be equal to the distance d1 from the to-be-measured edge where the first detection point is located to the camera; if one end point of the to-be-measured edge is the second detection point, the to-be-measured edge is the to-be-measured edge where the second detection point is located, and then d is set to be equal to the distance d2 from the to-be-measured edge where the second detection point is located to the camera; wherein when the to-be-positioned device plane is not parallel to the pinhole plane of the camera, d1 is not equal to d2; wherein when the to-be-positioned device plane is not parallel to the pinhole plane of the camera, the intersection line of the to-be-positioned device plane and the pinhole plane of the camera is perpendicular to the target straight line direction; the edge in a rectangular label which is connected by the adjacent two target detection points is parallel to the target straight line direction.

[0013] Further, if the two vertexes of the to-be-measured edge where the first detection point is located are updated as the adjacent two target detection points, the edge in the same rectangular label which is originally distributed along the target straight line direction is set as the to-be-measured edge where the updated target detection point is located, so that the target straight line direction is updated to be parallel to the to-be-measured edge where the original first detection point is located; then the distance from the camera to the to-be-measured edge where the updated target detection point is located is set to be equal to d1.

[0014] In summary, the pinhole imaging model can be constructed from the perspective of the side view between the plane of the object to be measured, the pinhole plane of the camera and the imaging plane of the camera (in fact, the geometric relationship of similar triangles is used from the perspective of the side view (preferably perpendicular to the horizontal direction) of each plane), the length of the perpendicular segment of the camera to the first detection point or the second detection point on the edge to be measured is calculated, and in the case that the plane of the object to be measured, the pinhole plane of the camera and the imaging plane of the camera are parallel under the side view, the distance and angle information about the rectangular label is calculated, the calculation amount of the distance between the rectangular label and the camera is saved, and the distortion influence brought by the vertical direction or the horizontal direction is overcome.

[0015] Further, the step B specifically comprises: obtaining the distance U between the two adjacent target detection points in the plane of the object to be measured in advance; then calculating the distance d1 of the camera to the edge to be measured where the first detection point is located by using the pinhole imaging model, and calculating the distance d2 of the camera to the edge to be measured where the second detection point is located by using the pinhole imaging model; based on the side angle relationship of the triangle, the deviation angle of the rectangular label relative to the camera and the inclination angle of the plane where the rectangular label is located are obtained by the following formula:

[0016]

[0017] d2*sin(b4)=U*sin(q)+d1*sin(b1);

[0018] U*cos(q)=d2*cos(b4)+d1*cos(b1);

[0019] b2=90-b1;

[0020] b3=90-b4;

[0021] Wherein, the angle of the included angle between the perpendicular segment of the camera to the edge to be measured where the first detection point is located and the pinhole plane of the camera in the opposite direction of the target straight line direction is denoted as b1, and the angle of the included angle between the perpendicular segment of the camera to the edge to be measured where the second detection point is located and the pinhole plane of the camera in the target straight line direction is denoted as b4; wherein, the deviation angle of the rectangular label relative to the camera includes the included angle b2 between the perpendicular segment of the camera to the edge to be measured where the first detection point is located and the optical axis, and the included angle b3 between the perpendicular segment of the camera to the edge to be measured where the second detection point is located and the optical axis; wherein, the angle of the included angle between the plane of the object to be measured and the pinhole plane of the camera is set as the inclination angle of the plane of the object to be measured, and q is the inclination angle of the plane of the object to be measured; the inclination angle of the plane where the rectangular label is located is the inclination angle of the plane of the object to be measured; the monocular distance measuring principle in step B includes the pinhole imaging model.

[0022] In summary, the distance d1 and the distance d2 are calculated from the side view, and the distance U, the angle b1, the angle b2, the angle b3, the angle b4 and the angle q can be obtained from the top view, wherein the side view and the top view can be regarded as two directions perpendicular to each other, one of which is preferably a vertical direction, and the other is preferably a horizontal direction; therefore, when calculating the deviation angle of the rectangular label relative to the camera, the formula five is constructed from the vertical and horizontal directions to perform triangular geometry calculation, thereby ensuring the comprehensiveness and representativeness of the angle calculation. The angle can also be obtained under the premise of avoiding the influence of camera distortion, thereby improving the accuracy of the deviation angle of the rectangular label relative to the camera.

[0023] Further, when the robot identifies the two adjacent target detection points from the object plane to be measured, the product U*sin(q) of the distance U between the two adjacent target detection points in the object plane to be measured and the sine value of the tilt angle q of the object plane to be measured is calculated, and then U*sin(q) is set as the distance measurement error caused by the distortion of the camera. Thus, the product U*sin(q) is used to distinguish whether the rectangular label is distorted in the camera, and the size of the distance measurement error caused by the distortion.

[0024] Further, the method for determining the distortion of the rectangular label in the camera based on the tilt angle of the plane where the rectangular label is located includes: in the case that the object plane to be measured is not parallel to the pinhole plane of the camera, if the robot calculates that the distance measurement error caused by the distortion of the side of the rectangular label parallel to the target straight line direction in the camera is not equal to the value 0, it is determined that the side parallel to the target straight line direction is distorted in the camera; in the case that the object plane to be measured is not parallel to the pinhole plane of the camera, if the robot calculates that the distance measurement error caused by the distortion of the side to be measured in the rectangular label in the camera is equal to the value 0, it is determined that the side to be measured in the rectangular label is not distorted in the camera; wherein the side to be measured of the rectangular label is perpendicular to the target straight line direction and parallel to the pinhole plane of the camera. Therefore, by calculating the distance measurement error of each side of the rectangular label, it is determined whether the rectangular label is distorted, thereby saving the calculation amount and ensuring the distance measurement accuracy.

[0025] Further, when the distance between the two adjacent target detection points obtained by the robot is less than 0.5*(d1+d2), the robot sets the ranging error caused by the distortion of the line connecting the two adjacent target detection points in the camera to be equal to the value 0; wherein the preset distortion error value is 0.5*(d1+d2), so that when the distance between the two adjacent vertices in the edge line of the rectangular label is less than the preset distortion error value, the line connecting the two adjacent vertices in the edge line of the rectangular label is configured not to be distorted in the camera. In the process of executing the visual positioning method, the rectangular label in which the two adjacent target detection points with a distance less than 0.5*(d1+d2) is specially selected for distance and angle calculation, thereby improving the accuracy of the positioning calculation of the object plane and the rectangular label arranged on the surface thereof.

[0026] Further, in step A, the pre-processing method of the image collected by the robot includes: the robot first performs grayscale processing on the collected image to obtain a binary image; then performs Gaussian smoothing processing on the binary image to remove image noise and obtain a smoothed image; then performs edge detection on the smoothed image to obtain a target edge image; and then performs inflation processing on the target edge image to obtain the pre-processed image, so that the robot can identify the vertices and graphic attributes of the rectangular label from the pre-processed image. The pre-processed edge line is smoother and sharper, and can be clearly imaged on the imaging plane of the camera. The problem of image blurring and noise interference caused by a specific light field is overcome, so that clear imaging can be achieved by relying solely on a monocular camera with a fixed focal length, and all vertices in the rectangular label can be obtained at one time, thereby improving the calculation accuracy of the length of the edge formed by the vertices.

[0027] Further, in step A, the method for searching the vertices of the rectangular label in the pre-processed image includes: extracting a closed graph in the pre-processed image, so that the closed graph represents the rectangular label to be searched by the robot; wherein the closed graph is generated in the edge detection; then extracting the vertices of the rectangular label and the graphic attributes of the rectangular label based on the closed graph, and performing filtering processing on the extracted vertices of the rectangular label and the graphic attributes of the rectangular label to suppress the noise of the rectangular label; and then setting the filtered vertices as the vertices of the rectangular label searched by the robot in the pre-processed image, and setting the filtered graphic attributes as the graphic attributes of the rectangular label searched by the robot in the pre-processed image. The filtered graphic attributes reflect clear and sharp edge imaging, so that the robot can obtain accurate graphic attributes and vertices. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flowchart of a visual positioning method based on a rectangular label according to an embodiment of the present application.

[0029] Figure 2 is the principle diagram of calculating the distance between the camera and the rectangular label under the side view angle and calculating the deviation angle of the rectangular label relative to the camera and the tilt angle of the plane Lr1 where the rectangular label is located under the top view angle in the case that the plane Lr1 where the rectangular label is located is not parallel to the pinhole plane Lf of the camera.

[0030] Figure 3 is the principle diagram of calculating the distance between the camera and the rectangular label under the side view angle and calculating the distance between the camera and the rectangular label under the top view angle in the case that the plane Lr2 where the rectangular label is located is parallel to the pinhole plane Lf of the camera. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. It should be understood that the specific embodiments described below are only used to explain the present application and not to limit the present application. In the description of the application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the application.

[0032] The present application discloses a visual positioning method based on a rectangular label, and the execution subject of the visual positioning method is a robot equipped with a camera. The robot identifies and positions a device with a label pattern by executing the visual positioning method, and can be understood as obtaining the pose information of the label pattern as the pose information of the device with the label pattern, including the angle and distance information relative to the camera. It can be extended to be used for positioning a charging base and its charging interface, or for positioning other signal base stations and machine recycling base stations. Based on the special corner relationship of the rectangular label, the robot can gradually approach or even align the device with the label pattern using the collected image information about the rectangular label, so as to overcome the ranging error caused by the camera collecting the device from different angles.

[0033] As an embodiment, refer to Figure 1 , a visual positioning method based on a label pattern is disclosed, comprising:

[0034] Step A, the robot pre-processes the image collected by its camera, and searches for the graphical properties of the rectangular label and the vertices of the rectangular label in the pre-processed image; then Step B is performed. The rectangular label is arranged on the surface of the device to be positioned so that the image information of the label pattern, including the edge contour information, can be collected by the camera of the robot. Specifically, a set of rectangular labels can be arranged on the surface of the device to be positioned along the horizontal direction, and the shapes and sizes of the rectangular labels in the set can be symmetrically arranged on the surface of the device to be positioned. The robot first controls the camera to collect images at a certain position point, the camera is a monocular camera, and the field of view of the camera covers part of the rectangular labels or all the rectangular labels arranged in the device to be positioned. When the device to be positioned is a charging base for the robot to supplement power, the camera of the robot can collect a rectangular label arranged on the surface of the charging base, which can represent the direction corresponding to the charging interface.

[0035] Specifically, the rectangular label can include vertex positions, the graphical properties of the rectangular label include information of line segments connected to form the rectangular label, and the line segments connected to form the rectangular label can be identified by the robot from the image and the number of identified line segments can be counted. The rectangular label is arranged on the surface of the device to be positioned, which has a contact docking structure with the body part of the robot, and can be detected by the camera arranged at the front end of the robot (i.e. falls within the detection field of view of the camera). For example, the surface of the seat body of the charging base for the sweeping robot, specifically the charging interface installed on the surface of the seat body, is composed of charging electrodes and other mechanical structures cooperating with the charging electrodes of the robot, including but not limited to magnetic adsorption and elastic adsorption components. The charging interface is preferably arranged directly below the rectangular label, and the orientation of the charging interface is parallel to the perpendicular line of the plane on which the rectangular label is located.

[0036] Preferably, the robot first pre-processes the image collected by the camera by performing Step A to remove noise interference, obtain higher-precision graphical properties of the rectangular label and vertices of the rectangular label, improve label recognition accuracy, and solve the problem of image blurring when the monocular camera mounted on the robot collects images at a fixed focal length.

[0037] Step B, according to the graphical properties of the rectangular label and the vertices of the rectangular label, the distance between the rectangular label and the camera, the deviation angle of the rectangular label relative to the camera, and the inclination angle of the plane where the rectangular label is located are calculated by using monocular ranging principle, so that in the case that the plane where the rectangular label is located is not parallel to the pinhole plane of the camera, the distance between the rectangular label and the camera is the distance from the camera to the side of the rectangular label which is parallel to the target intersection line, and on this basis, the distance between the rectangular label and the camera, and the deviation angle of the rectangular label relative to the camera are set as the visual positioning result of the robot to the positioning device; Specifically, after determining the relative position information between the rectangular label and the camera and the ranging error caused by the distortion of the rectangular label in the camera, based on the size of the ranging error or the inclination angle of the plane where the rectangular label is located used to determine the size of the ranging error, the distance between the rectangular label and the camera, and the deviation angle of the rectangular label relative to the camera are selected as the visual positioning result of the robot to the positioning device, as the positioning information of the robot to the rectangular label in the case that the plane where the rectangular label is located is not parallel to the pinhole plane of the camera. In order to suppress the ranging error caused by the distortion of the side of the rectangular label which is at an angle with the target intersection line in the camera, and improve the accuracy of camera ranging.

[0038] It should be noted that the inclination angle of the plane where the rectangular label is located includes the angle of the intersection angle between the plane where the rectangular label is located and the pinhole plane of the camera, which is used to determine the ranging error caused by the distortion of the rectangular label in the camera, including the ranging error caused by the distortion of the side of the rectangular label which is perpendicular to the target intersection line in the camera. Specifically, the intersection line between the plane where the rectangular label is located and the pinhole plane of the camera is denoted as the target intersection line, which is parallel to the pinhole plane of the camera and also parallel to the imaging plane of the camera. At this time, the inclination angle of the plane where the rectangular label is located is equal to an integer multiple of 0 degrees or 180 degrees, the ranging error caused by the distortion of the target intersection line in the camera is equal to 0, and it is considered that no distortion occurs, while the side of the rectangular label which is at an angle with the target intersection line is distorted in the camera. In addition, in this embodiment, the deviation angle of the rectangular label relative to the camera can be represented by the intersection angle between the incident light at the corner point of the side of the rectangular label extracted from the surface of the positioning device and the optical axis of the camera, and the distance between the rectangular label and the camera can be represented by the distance between the side of the rectangular label perpendicular to the horizontal ground and the pinhole plane of the camera (which can be reduced to the lens center position of the camera), in order to suppress the distortion of the label pattern in the camera (lens).

[0039] Compared with the prior art, the robot uses the identified vertexes of the rectangular label to calculate the distance and angle information of the rectangular label relative to the same camera, and then predicts the relative position relationship between the rectangular label and the same camera in the case of suppressing camera distortion, thereby providing more accurate positioning information for robot navigation back to the device to be positioned. Moreover, instead of staying at the same position for too much time to collect multiple frames of images, the distance and angle information of the identified rectangular label is calculated within the same frame of image collected by using monocular ranging principle, which takes into account the influence of camera ranging error and reduces real-time positioning calculation amount.

[0040] As an embodiment, the visual positioning method further comprises: determining the case of distortion of the rectangular label in the camera based on the inclination angle of the plane where the rectangular label is located, so that when the distance between two adjacent vertexes in the edge line of the rectangular label is less than a preset distortion error value, the line connecting the two adjacent vertexes in the edge line of the rectangular label is configured not to be distorted in the camera. In this embodiment, the inclination angle of the plane where the rectangular label is located is used to determine the ranging error caused by the distortion of the rectangular label in the camera, and is obtained on the basis of the triangular geometric relationship established based on the inclination angle of the plane where the rectangular label is located, the distance between the two adjacent vertexes in the edge line of the rectangular label, and the distance between the rectangular label and the camera, wherein the distance between the rectangular label and the camera includes the distance from the edge where the two adjacent vertexes are located to the camera, so as to capture more distance information of the details of the rectangular label. Specifically, when the distance between the two adjacent vertexes in the edge line of the rectangular label is less than the preset distortion error value or equal to 0, the ranging error caused by the distortion of the line connecting the two adjacent vertexes in the edge line of the rectangular label in the camera is equal to 0, and the line is configured not to be distorted in the camera, and then the distance between the rectangular label and the camera and the deflection angle of the rectangular label relative to the camera can be set as the visual positioning result of the robot to the device to be positioned. The distance between the rectangular label and the camera includes the distance from the camera to any edge of the rectangular label, or the length of the perpendicular segment from the camera to the plane where the rectangular label is located.

[0041] Preferably, the visual positioning method further comprises: the distance between the two adjacent vertexes obtained by the robot is less than a preset distortion error value, wherein the preset distortion error value is equal to half of the sum of the distances from the two mutually parallel edges with the two adjacent vertexes as end points to the camera in the same rectangular label. Understandably, if half of the sum is greater than the length of the line connecting the two adjacent vertexes, the distance of the line connecting the two adjacent vertexes in the rectangular label is so small that it can be ignored, and the actual imaging width in the imaging plane may not differ much from the ideal imaging width, so the resulting distortion can also be ignored.

[0042] As an embodiment, when the plane where the rectangular label is located is parallel to the pinhole plane of the camera, the robot determines that any edge of the rectangular label does not produce distortion in the camera, that is, the edges of the rectangular label do not produce distortion in the camera, and thus the aforementioned ranging error disclosed in the embodiment is not generated when the four edges of the rectangular label are measured using the camera, and the shape of the rectangular label in the imaging plane of the camera does not change substantially, and the size may change proportionally, which is related to the focal length, object distance and image distance of the camera. Therefore, the distance between the rectangular label and the camera can be directly calculated by calling the pinhole imaging model, saving the participation of distortion parameters, and ensuring and improving the efficiency and accuracy of camera ranging without conversion through distortion parameters or distortion formulas commonly used in the prior art.

[0043] It should be noted that if the robot detects that there is an edge in the rectangular label that is not parallel to the pinhole plane of the camera, it is determined that the edge that is not parallel to the pinhole plane of the camera produces distortion in the camera; therefore, when the robot recognizes that the tilt angle of the plane of the object to be measured is not equal to 0 degrees and the tilt angle of the plane of the object to be measured is not equal to an integer multiple of 180 degrees, the robot determines that the rectangular label in the plane of the object to be measured produces distortion in the camera; if the robot detects that there is an edge in the rectangular label that is parallel to the pinhole plane of the camera, it is determined that the edge that is parallel to the pinhole plane of the camera does not produce distortion in the camera, but the rectangular label in the plane of the object to be measured may produce distortion in the camera. Thus, it is determined whether the rectangular label in the plane of the object to be measured produces distortion in the camera. The plane where the rectangular label is located is the plane of the object to be measured, which is the surface of the to-be-positioned device.

[0044] For the distortion phenomenon of the rectangular label in the camera, it should be noted that in the plane of the object to be measured with different tilt angles relative to the imaging plane of the camera, the intersection height of the chief ray of the rectangular label after passing through the monocular camera with a fixed focal length and the imaging plane of the camera is not consistent and is not equal to the ideal image height, and thus distortion occurs. Specifically, among the edges of the rectangular label and the edges of the triangular label in a plane of an object to be measured with a tilt angle relative to the pinhole plane, the edges relatively close to the optical center of the camera have a larger imaging size (e.g., imaging width or imaging height) in the imaging plane, which is larger than the ideal object height due to distortion; the edges relatively far from the optical center of the camera have a smaller imaging size (e.g., imaging width or imaging height) in the same imaging plane, which is smaller than the ideal object height due to distortion.

[0045] As an embodiment, in the step A, in the process of searching the graphic attribute of the rectangular label and the vertex of the rectangular label in the pretreated image, the robot detects the number of edges surrounding the closed graphic; wherein the graphic attribute of the rectangular label includes the edge line feature of the closed graphic, and the edge line feature of the closed graphic includes the number of edges surrounding the closed graphic and the number of vertices of the closed graphic; in the edge line of the closed graphic, the robot identifies the line connecting two adjacent vertices as an edge surrounding the closed graphic; in this embodiment, the edge line refers to the edge line excluding the diagonal line of the closed graphic; preferably, the closed graphic is connected by a plurality of line segments connected at the beginning and end, or is connected by a plurality of pixel points in sequence, wherein the pixel point at the corner position of the closed graphic is identified as the vertex of the closed graphic. Then, the robot counts the number of edges of a single closed graphic in the pretreated image, and when the robot counts that the number of edges surrounding the closed graphic is 4 and there are two groups of edges perpendicular to each other, the currently detected closed graphic is identified as a rectangular label; wherein each group of edges is provided with two parallel edges. Thus, the shape of the rectangular label and the distribution characteristics of the internal pixel points are analyzed to distinguish labels of different shapes.

[0046] As an embodiment, the method for calculating the distance between the rectangular label and the camera using the pinhole imaging model includes:

[0047] The robot sets two vertices in a rectangular label distributed along the target straight line direction as adjacent two target detection points in the plane of the object to be detected, and then in the adjacent two target detection points, the robot records one of the target detection points as a first detection point, and the distance d1 from the camera to the to-be-detected edge where the first detection point is located; the robot records the other target detection point as a second detection point, and the distance d2 from the camera to the to-be-detected edge where the second detection point is located; wherein the edge of the rectangular label perpendicular to the target straight line direction is the to-be-detected edge of the rectangular label; the plane of the object to be detected is used to represent the surface of the device to be positioned, and is also the plane where the rectangular label is located. Therefore, the robot records the edge with a certain inclination angle to the target straight line direction as the to-be-detected edge of the label pattern, which can be specifically divided into the to-be-detected edge where the first detection point is located and the to-be-detected edge where the second detection point is located, and the inclination angle between the to-be-detected edge and the target straight line direction is not 0, and the to-be-detected edge is not the edge connecting the first detection point and the second detection point. The distance between the rectangular label and the camera in the step B includes the distance from the camera to the to-be-detected edge where the first detection point is located and the distance from the camera to the to-be-detected edge where the second detection point is located.

[0048] The monocular ranging principle described in step B includes a pinhole imaging model. The method in which the robot can calculate the distance between the tag pattern and the camera using the pinhole imaging model includes: obtaining the lens focal length f of the camera in advance (pre-set fixed lens parameters), the side length w of the to-be-measured side (as the object height of the pinhole imaging model), and the pixel width p of the to-be-measured side formed in the imaging plane of the camera (the actual image height changes with the side length w of the to-be-measured side).

[0049] Then the distance between the relevant to-be-measured side and the camera is calculated by the following formula one:

[0050]

[0051] If one end point of the to-be-measured side is the first detection point, the to-be-measured side is the to-be-measured side where the first detection point is located, and then d is set to be equal to the distance from the to-be-measured side where the first detection point is located to the camera; the first detection point belongs to the target detection point; specifically, the case that one end point of the to-be-measured side is the first detection point is represented as: the intersection point of the line connecting the aforementioned two adjacent target detection points and the to-be-measured side is the first detection point, then the to-be-measured side is the first to-be-measured side, and w represents the side length of the first to-be-measured side, which is substituted into formula one to obtain the distance from the to-be-measured side where the first detection point is located to the camera. Similarly, if one end point of the to-be-measured side is the second detection point, the to-be-measured side is the to-be-measured side where the second detection point is located, and then d is set to be equal to the distance from the to-be-measured side where the second detection point is located to the camera; the second detection point also belongs to the target detection point; specifically, the case that one end point of the to-be-measured side is the second detection point is represented as: the intersection point of the line connecting the aforementioned two adjacent target detection points and the to-be-measured side is the second detection point, then the to-be-measured side is the second to-be-measured side, and w represents the side length of the second to-be-measured side, which is substituted into formula one to obtain the distance from the to-be-measured side where the second detection point is located to the camera.

[0052] It should be noted that the lens focal length f of the camera and the side length w of the to-be-measured side can be obtained through mechanical design, or obtained through simple ruler measurement, or directly obtained by querying the lens factory parameters (including curvature radius, distance, refractive index, etc.), and the pixel width p of the to-be-measured side formed in the imaging plane of the camera can be obtained in the coordinate system of the imaging plane of the camera.

[0053] It is worth noting that when the to-be-measured object plane and the pinhole plane of the camera are not parallel, the distance from the to-be-measured side where the first detection point is located to the camera is not equal to the distance from the to-be-measured side where the second detection point is located to the camera, and even more not equal to the distance from the camera to the to-be-measured object plane when the optical axis of the camera is perpendicular to the to-be-measured object plane (directly calculated by the pinhole imaging model), therefore, the inclination angle between the to-be-measured object plane and the pinhole plane of the camera produces a ranging error.

[0054] As an implementation of calculating the distance between the to-be-measured edge of the rectangular label and the camera, in combination with Figure 2 It can be seen that the two adjacent target detection points are two vertices in the edge of the rectangular label parallel to the target straight line direction, which are respectively denoted as a first detection point and a second detection point; the two edges of the rectangular label perpendicular to the target straight line direction are respectively a to-be-measured edge where the first detection point is located and a to-be-measured edge where the second detection point is located, wherein the to-be-measured edge where the first detection point is located is denoted as a first to-be-measured edge, and the to-be-measured edge where the second detection point is located is denoted as a second to-be-measured edge. Wherein the to-be-measured edge where the first detection point is located and the to-be-measured edge where the second detection point is located are both parallel to the pinhole plane of the camera and do not produce distortion in the camera. Thus, based on the two adjacent target detection points recognized by the robot, the effective to-be-measured edge in the same rectangular label is extracted.

[0055] When the rectangular label recognized by the robot is Figure 2 rectangular MNCD, the robot sets two vertices M and N in the rectangular MNCD along the target straight line direction as the two adjacent target detection points in the to-be-measured object plane Lr1, and the edge MN is parallel to the target straight line direction; the robot records the target detection point deviating to the left as the first detection point M, and records the distance from the to-be-measured edge where the first detection point M is located to the position O (denoted as the optical center of the camera) of the camera as d1, which can be replaced by d in the aforementioned formula one; the robot records the target detection point deviating to the right as the second detection point N, and records the distance from the to-be-measured edge where the second detection point N is located to the position O (denoted as the optical center of the camera) of the camera as d2, which can be replaced by d in the aforementioned formula one. The line connecting the two vertices in a rectangular label along the first included angle with the target straight line direction is denoted as the first to-be-measured edge of the rectangular label, which corresponds to the edge MC of the rectangular label, as the to-be-measured edge where the first detection point M is located, in Figure 2 Figure 2 , the length of the first to-be-measured edge CM is denoted by w, which is equivalent to w in formula one; the pixel width of the first to-be-measured edge CM formed in the imaging plane of the camera is denoted by p1, which is equivalent to p1 in formula one. In Figure 2 the schematic diagram of the side view of the right side, the optical center O of the camera is located on the pinhole plane Lf of the camera, and the distance from the optical center O of the camera to the imaging plane Lp of the camera is equal to the fixed lens focal length f.

[0056] Based on the similarity triangle theorem, the distance from the to-be-measured edge where the first detection point M is located to the position O (denoted as the optical center of the camera) of the camera is obtained by the following formula two:

[0057]

[0058] ​Therefore, the distance d1 between the first to-be-measured edge CM and the optical center O of the camera can be regarded as constructing a pinhole imaging model between the to-be-measured object plane Lr1, the pinhole plane Lf of the camera and the imaging plane Lp of the camera from the perspective of the side view (actually, the geometric relationship of similar triangles is used for calculation from the perspective of the side view of each plane, see Figure 2 The length of the perpendicular segment from the camera to the to-be-measured edge where the first detection point M is located is calculated, and then the distance d1 from the first to-be-measured edge to the camera is obtained from the perspective of the top view.

[0059] Similarly, the distance d2 between the second to-be-measured edge DN and the optical center O of the camera can be regarded as constructing a pinhole imaging model between the to-be-measured object plane Lr1, the pinhole plane Lf of the camera and the imaging plane Lp of the camera from the perspective of the side view (actually, the geometric relationship of similar triangles is used for calculation from the perspective of the side view of each plane), the length of the perpendicular segment from the camera to the to-be-measured edge where the second detection point N is located is calculated, and then the distance d2 from the second to-be-measured edge to the camera is obtained from the perspective of the top view.

[0060] In combination with Figure 2 It can be seen that the to-be-measured object plane Lr1 is not parallel to the pinhole plane Lf of the camera, among the adjacent two target detection points M and N distributed in a rectangular label, the distance d1 from the camera to the first to-be-measured edge MC is not equal to the distance d2 from the camera to the second to-be-measured edge ND, and then a distance difference is generated, which corresponds to the distance measurement error generated by the distortion of the rectangular label. When the to-be-measured object plane Lr1 is not parallel to the pinhole plane Lf of the camera, the intersection line (not shown in the figure) between the to-be-measured object plane Lr1 and the pinhole plane Lf of the camera is arranged vertically to the direction of the target straight line, and the edge formed by the adjacent two target detection points M and N in a rectangular label is parallel to the direction of the target straight line. It is worth noting that when the to-be-measured object plane is parallel to the pinhole plane of the camera, the distance from the camera to the first to-be-measured edge of the rectangular label is equal to the distance from the camera to the second to-be-measured edge of the same rectangular label, that is, the distance d2 is equal to the distance d1. At this time, the rectangular label does not produce distortion in the camera.

[0061] In combination with Figure 3 It can be seen that the to-be-measured object plane Lr1 of the rectangular label MNCD becomes parallel to the pinhole plane Lf of the camera, and then any edge of the rectangular label MNCD does not produce distortion in the camera; specifically, the edges MN and CD parallel to the direction of the target straight line are both parallel to the pinhole plane Lf of the camera from the perspective of the top view; the edges MC and ND parallel to the vertical direction of the target straight line are both parallel to the pinhole plane Lf of the camera from the perspective of the side view; as Figure 3As shown, no matter whether the rectangular edge MN (denoted as the long edge, representing the width of the rectangular label) in the plane Lr1 is viewed from the top view perspective or the rectangular edge CM (denoted as the short edge, representing the height of the rectangular label) in the plane Lr1 is viewed from the side view perspective, the ranging error caused by their distortion is equal to the value 0, which can determine that they do not occur distortion in the camera; since the distance between the rectangular label and the camera includes the distance from the camera to any edge of the rectangular label or the length of the perpendicular segment from the camera to the plane where the rectangular label is located, therefore, the distance from the optical center O of the camera to the rectangular edge CM is represented by using the length of the perpendicular segment from the optical center O to the rectangular edge CM, and the distance from the optical center O of the camera to the rectangular edge MN is represented by using the length of the perpendicular segment from the optical center O to the rectangular edge MN, which are both applicable to directly calculate using the pinhole imaging model to obtain the distance between the rectangular label MNCD and the camera.

[0062] When the rectangular label recognized by the robot is the rectangular MNCD in Figure 3 , the robot sets two vertices M and N in the rectangular MNCD which are distributed along the direction of the target straight line as two adjacent target detection points in the object plane Lr1, as shown in Figure 3 The edge MN is parallel to the direction of the target straight line, as shown in the top view on the right side; the robot records the target detection point deviated to the left as the first detection point M, and records the distance from the edge where the first detection point M is located to the position O (denoted as the optical center of the camera) where the camera is located as Figure 3 d3 on the left side, which can be replaced by d in the aforementioned formula one; the line connecting the two vertices in the rectangular MNCD which are distributed along the angle of 90 degrees with the direction of the target straight line is recorded as the first measured edge of the rectangular label, which corresponds to Figure 3 the edge MC of the rectangular in Figure 3 , the length of the first measured edge CM is represented by w, which is equivalent to w in formula one; the pixel width formed by the first measured edge CM in the imaging plane of the camera is represented by p2, which is equivalent to p in formula one.

[0063] In Figure 3 the side view perspective diagram on the left side, the optical center O of the camera is located on the pinhole plane Lf of the camera, and the distance from the optical center O of the camera to the imaging plane Lp of the camera is equal to the fixed focal length f of the lens.

[0064] Based on the similarity triangle theorem, the distance from the first measured edge CM where the first detection point M is located to the position O (denoted as the optical center of the camera) where the camera is located is obtained by using the following formula three:

[0065]

[0066] Therefore, the distance d3 between the first to-be-measured edge CM and the optical center O of the camera can be regarded as constructing a pinhole imaging model between the to-be-measured object plane Lr1, the pinhole plane Lf of the camera, and the imaging plane Lp of the camera from the perspective of the side view (actually, the geometric relationship of similar triangles is used for calculation from the perspective of each plane side view, which can be referred to as Figure 3 the similar triangle relationship on the left side) to calculate the length of the perpendicular segment from the camera to the to-be-measured edge where the first detection point M is located, and then obtain the distance d3 from the first to-be-measured edge to the camera from the perspective of the top view.

[0067] On the basis of the above embodiment, if the two vertices of the to-be-measured edge where the first detection point is located are updated to the adjacent two target detection points, the edge originally distributed along the target straight line direction in the same rectangular label is set as the to-be-measured edge where the updated target detection point is located, the target straight line direction is updated to be parallel to the to-be-measured edge where the original first detection point is located, and then the distance from the camera to the to-be-measured edge where the updated target detection point is located is set to be equal to d1. Corresponding to Figure 3 the first to-be-measured edge CM where the first detection point C is located, the two vertices of the to-be-measured edge CM are updated to the adjacent two target detection points, that is, the adjacent two target detection points are updated to the first detection point C and the second detection point M, the edge MN originally distributed along the target straight line direction in the same rectangular label is set as the to-be-measured edge where the first detection point M is located, and is updated to the first to-be-measured edge MN, the target straight line direction is updated to be parallel to the to-be-measured edge CM where the original first detection point is located, and then the distance from the camera to the to-be-measured edge MN where the updated target detection point C is located is set to be equal to d4.

[0068] In Figure 3 , the length of the first to-be-measured edge MN is denoted by U, which is equivalent to w in Formula One; the pixel width formed by the first to-be-measured edge MN in the imaging plane of the camera is denoted by p3, which is equivalent to p in Formula One. In Figure 3 the side view perspective diagram on the right side, the optical center O of the camera is located on the pinhole plane Lf of the camera, and the distance from the optical center O of the camera to the imaging plane Lp of the camera is equal to the fixed focal length f. Based on the theorem of similar triangles, the distance from the to-be-measured edge MN where the first detection point M is located to the position O (denoted as the optical center of the camera) of the camera is obtained by the following Formula Four:

[0069]

[0070] Therefore, the distance d4 between the first to-be-measured edge MN and the optical center O of the camera can be regarded as constructing a pinhole imaging model between the to-be-measured object plane Lr1, the pinhole plane Lf of the camera, and the imaging plane Lp of the camera from the perspective of the side view (actually, the geometric relationship of similar triangles is used from the perspective of the side view of each plane, which will be described in detail below Figure 2 The similar triangle relationship on the right is constructed), the length of the perpendicular segment from the camera to the to-be-measured edge MN where the first detection point M is located is calculated, and then the distance d4 from the updated first to-be-measured edge MN to the camera is obtained from the perspective of the top view.

[0071] In summary, the foregoing embodiments can be regarded as constructing a pinhole imaging model between the to-be-measured object plane, the pinhole plane of the camera, and the imaging plane of the camera from the perspective of the side view (actually, the geometric relationship of similar triangles is used from the perspective of the side view of each plane, which is preferably perpendicular to the horizontal direction), calculating the length of the perpendicular segment from the camera to the to-be-measured edge where the first detection point or the second detection point is located, and calculating the distance and angle information about the rectangular label in the case that the to-be-measured object plane, the pinhole plane of the camera, and the imaging plane of the camera are parallel to each other from the perspective of the side view, thereby saving the calculation amount of the distance between the rectangular label and the camera and overcoming the distortion effect in the vertical direction or the horizontal direction.

[0072] As an embodiment, the step B specifically comprises: the robot sets the target straight line direction as the extension direction of the to-be-measured object plane on the horizontal plane, and sets the target straight line direction as parallel to the walking plane of the robot, the robot sets the walking direction of the target straight line direction as parallel to the extension direction of the horizontal coordinate axis in the imaging plane of the camera on the horizontal plane, and sets the horizontal coordinate axis in the imaging plane of the camera as parallel to the walking plane of the robot, so as to identify the rectangular label in the imaging plane of the camera, and the vertical plane perpendicular to the target straight line direction is perpendicular to the walking plane of the robot, and can also be parallel to the pinhole plane of the camera to suppress the influence of distortion error, the pinhole plane of the camera is parallel to the imaging plane of the camera, for example, traversing the vertex of the identified rectangular label along the positive direction of the horizontal coordinate axis in the imaging plane, the identified rectangular label herein is the rectangular label that has been clearly imaged in the imaging plane of the camera, and the robot determines the graphical attribute and the vertex thereof.

[0073] Then, the robot sets two vertices in a rectangular label distributed along the target straight line direction as adjacent two target detection points in the plane of the object to be measured, as two end points of an edge of a rectangular label, and pre-obtains the distance U between the adjacent two target detection points in the plane of the object to be measured, which can represent the actual width of a rectangular label. In the same rectangular label, the distance U between the adjacent two target detection points and the length of the edge to be measured can be obtained through mechanical design, or through simple ruler measurement, or directly obtained by querying lens factory parameters (including curvature radius, distance, refractive index, etc.). The distance U between the adjacent two target detection points can be calculated when the graphical attributes and vertices of the rectangular label are identified.

[0074] In the embodiment, the monocular distance measuring principle includes a pinhole imaging model and a geometric proportion relationship of similar triangles. Among the two adjacent target detection points, the robot records one of the target detection points as a first detection point, which is generally the left-biased target detection point, and calculates the distance d1 of the edge to be measured where the first detection point is located to the camera using the pinhole imaging model, which corresponds to Figure 2 the distance d1; the robot records the other target detection point as a second detection point, which is generally the right-biased target detection point, and calculates the distance d2 of the edge to be measured where the second detection point is located to the camera using the pinhole imaging model, which corresponds to Figures 2 to 3 the distance d1; correspondingly, the lens of the camera has a pinhole plane passing through the center of the lens (the optical center of the camera), and an imaging plane located behind the lens, which correspond to Figure 2 the plane Lf (pinhole plane) and the plane Lp (imaging plane) in , respectively. Here, the planes are displayed as a line segment in the top view (perpendicular to the walking plane direction of the robot) and the side view (parallel to the walking plane of the robot), and both are displayed as a line segment. The pinhole plane is the plane where the optical center of the camera is located in the calculation scenario of the pinhole imaging model, and the optical center of the camera is the center of the lens.

[0075] Based on the edge-angle relationship of the triangle, the following formula five is used to obtain the deviation angle of the rectangular label relative to the camera and the inclination angle of the plane where the rectangular label is located:

[0076]

[0077] d2*sin(b4)=U*sin(q)+d1*sin(b1);

[0078] U*cos(q)=d2*cos(b4)+d1*cos(b1);

[0079] b2=90-b1;

[0080] b3=90-b4;

[0081] The deviation angle of the rectangular label relative to the camera includes an included angle b2 between the vertical line segment of the camera to the to-be-measured edge MC where the first detection point M is located and the optical axis, and an included angle b3 between the vertical line segment of the camera to the to-be-measured edge ND where the second detection point N is located and the optical axis. The angle of the included angle between the vertical line segment of the camera to the to-be-measured edge where the first detection point is located and the pinhole plane of the camera in the direction opposite to the target straight line direction is denoted as b2. The vertical line segment of the camera to the to-be-measured edge where the first detection point M is located can be understood as a vertical line segment passing through the optical center O of the camera and being perpendicular to the edge in the label pattern passing through the first detection point M, and is suitable for calculating the distance between the camera and the to-be-measured edge where the first detection point M is located in the pinhole imaging model. The deviation angle of the rectangular label relative to the camera includes the included angle b2 between the vertical line segment of the camera to the to-be-measured edge where the first detection point M is located and the optical axis. Here, the correspondence between the top-down view angle and the side view angle can be understood, that is, the distance d1 of the to-be-measured edge MC where the first detection point M is located to the camera can be regarded as constructing a pinhole imaging model between the to-be-measured object plane, the pinhole plane of the camera and the imaging plane of the camera from the side view angle (preferably the direction perpendicular to the horizontal ground) (in fact, the geometric relationship of similar triangles is used from each plane side view angle (preferably the direction perpendicular to the horizontal ground)), the length of the vertical line segment of the camera to the to-be-measured edge where the first detection point M is located is calculated, and then the distance d1 of the to-be-measured edge where the first detection point is located to the camera is obtained from the top-down view angle. Based on this, the length of the vertical line segment of the camera to the to-be-measured edge MC where the first detection point M is located is equivalent to the distance d1 of the to-be-measured edge where the first detection point is located to the camera.

[0082] An angle between the perpendicular segment of the camera to the to-be-measured edge ND where the second detection point N is located and the pinhole plane Lf of the camera in the target straight line direction is denoted as b4, the perpendicular segment of the camera to the to-be-measured edge ND where the second detection point N is located can be understood as a perpendicular segment of the optical center O of the camera and the edge in the rectangular label where the second detection point N is located, which is suitable for calculating the distance between the camera and the to-be-measured edge ND where the second detection point N is located in the pinhole imaging model, and the deflection angle of the rectangular label relative to the camera includes the difference between the angle b3 between the perpendicular segment of the camera to the to-be-measured edge ND where the second detection point N is located and the optical axis and 90 degrees. Here, the corresponding relationship between the top view angle and the side view angle can be understood, that is, the distance d2 of the to-be-measured edge where the second detection point N is located to the camera can be regarded as constructing a pinhole imaging model (actually, the geometric relationship of similar triangles is used from the side view angle (preferably, the direction perpendicular to the horizontal ground) of each plane) between the to-be-measured object plane, the pinhole plane of the camera, and the imaging plane of the camera from the side view angle, calculating the length of the perpendicular segment of the camera to the to-be-measured edge where the second detection point is located, and then obtaining the distance d2 of the to-be-measured edge where the second detection point is located to the camera from the top view angle; based on this, the length of the perpendicular segment of the camera to the to-be-measured edge where the second detection point is located is equivalent to the distance d2 of the to-be-measured edge where the second detection point is located to the camera.

[0083] It should be noted that the distance between the rectangular label and the camera in step B includes the distance from the to-be-measured edge where the first detection point is located to the camera and the distance from the to-be-measured edge where the second detection point is located to the camera; the monocular distance measuring principle in step B includes the pinhole imaging model.

[0084] The distance between the rectangular label and the camera includes the distance from the to-be-measured edge where the first detection point of the rectangular label is located to the camera and the distance from the to-be-measured edge where the second detection point of the same rectangular label is located to the camera; similarly, the deflection angle of the rectangular label relative to the camera includes the deflection angle of the rectangular label relative to the camera.

[0085] The robot sets the angle between the plane Lr1 of the object to be measured and the pinhole plane Lf of the camera as the tilt angle of the plane Lr1 of the object to be measured, and q is the tilt angle of the plane of the object to be measured; the tilt angle of the plane on which the rectangular label is located is the tilt angle of the plane of the object to be measured. In Formula V, the distances d1, d2 and U are known quantities, and the angles b1, b2, b3, b4 and q are unknown quantities. The equation constructed by Formula V can be used to obtain the angles b1, b2, b3, b4 and q. Thus, the function relationship between the cosine theorem and the side angle of the triangle corresponding to the aforementioned formula is constructed by the pinhole imaging model, the tilt angle of the plane of the object to be measured (the plane on which the charging interface is located) and the azimuth angle information of the label pattern at the first detection point and the second detection point relative to the camera are calculated, and then the specific angle information of the label pattern plane relative to the camera can be determined by using the monocular camera with a fixed focal length.

[0086] It should be noted that, referring to Figure 2 It can be seen that the rectangular label recognized by the robot is the rectangle MNCD, and the plane Lr1 on which the rectangle MNCD is located is not parallel to the imaging plane Lp of the camera. The robot sets two vertices M and N in the rectangle MNCD which are distributed along the target straight line direction as two target detection points in the plane Lr1 of the object to be measured, wherein the distance between the target detection point M and the target detection point N is denoted as U, that is, the length of the side MN of the rectangle; the robot denotes the target detection point M deviated to the left as the first detection point M, and the distance from the side on which the first detection point M is located to the position O of the camera (denoted as the optical center of the camera) is denoted as d1, which is obtained by pre-calculation using Formula II corresponding to the pinhole imaging model disclosed in the foregoing embodiments; the robot denotes the target detection point N deviated to the right as the second detection point N, and the distance from the side on which the second detection point N is located to the position O of the camera (denoted as the optical center of the camera) is denoted as d2, which is obtained by pre-calculation using the pinhole imaging model.

[0087] In summary, the distances d1 and d2 are obtained from the side view, and the distance U, the angles b1, b2, b3, b4 and q can be obtained from the top view, wherein the side view and the top view can be regarded as two directions perpendicular to each other, one of which is preferably the vertical direction, and the other of which is preferably the horizontal direction; therefore, when calculating the deviation angle of the rectangular label relative to the camera, the robot will construct Formula V from the vertical and horizontal directions to calculate the corresponding target detection points and the sides on which they are located, so as to ensure the comprehensiveness and representativeness of the angle calculation. The angle can also be obtained under the premise of avoiding the influence of camera distortion, thereby improving the accuracy of the deviation angle of the rectangular label relative to the camera.

[0088] As an embodiment, when the robot identifies the two adjacent target detection points from the object plane to be measured, the product of the distance U between the two adjacent target detection points in the object plane to be measured and the sine value of the tilt angle q of the object plane to be measured, i.e., U*sin(q), is calculated, and U*sin(q) is set as the ranging error caused by the distortion of the line connecting the two adjacent target detection points in the camera. When the robot identifies that the tilt angle q of the object plane to be measured is not equal to 0 degree and the tilt angle q of the object plane to be measured is not equal to an integer multiple of 180 degrees, the robot determines that the object plane to be measured is not parallel to the pinhole plane of the camera, and thus the ranging error caused by the distortion of the line connecting the two adjacent target detection points in the camera can not be 0, and it can be determined that the rectangular label in the object plane to be measured is distorted in the camera. Therefore, the product U*sin(q) is used to distinguish whether the rectangular label is distorted in the camera and the size of the ranging error caused by the distortion.

[0089] When the plane where the rectangular label is located is not parallel to the imaging plane of the camera, the edge line of the rectangular label is distorted after being captured by the camera, wherein the size difference (for example, the difference between the actual image height and the ideal image height) of the edge line where the two adjacent target detection points of the rectangular label are located before and after the distortion is obtained by the product of the distance U between the two adjacent target detection points in the object plane to be measured and the sine value of the tilt angle of the object plane to be measured in the embodiment.

[0090] In combination Figure 3It can be seen that, in the case that the plane Lr1 on which the rectangular label is located is not parallel to the pinhole plane Lf of the camera, among the two adjacent vertices M and N distributed in the edge line of a rectangular label, the ranging error generated by the distortion of the line connecting the two adjacent vertices M and N is equal to the product of the length U of the line segment MN and the sine value of the inclination angle q of the plane Lr1 on which the rectangular label is located, the inclination angle q of the plane on which the rectangular label is located being represented by the angle between the target straight line direction between the plane Lr1 of the object to be measured and the second virtual plane Lv2, from the perspective of the top view, the first virtual plane Lv1 is a virtual plane passing through the rectangular edge CM on which the point N is located and parallel to the pinhole plane Lf of the camera, and the second virtual plane Lv2 is a virtual plane passing through the rectangular edge on which the point M is located and parallel to the pinhole plane Lf of the camera; the product of the length U of the line segment MN and the sine value of the inclination angle q of the plane on which the rectangular label is located is equal to the distance between the first virtual plane Lv1 and the second virtual plane Lv2, the smaller the inclination angle q of the plane on which the rectangular label is located and / or the smaller the length U of the line segment MN is set, the smaller the distance between the first virtual plane Lv1 and the second virtual plane Lv2 becomes, and the ranging error generated by the distortion of the line connecting the two adjacent vertices M and N becomes smaller, and the inclination angle q of the plane Lr1 (the plane of the object to be measured) on which the rectangular label is located also becomes smaller, thereby causing the ranging error generated by the distortion of the rectangular label to become smaller.

[0091] If the inclination angle q of the plane on which the rectangular label is located becomes an integer multiple of 0 degrees or 180 degrees, the plane Lr1 on which the rectangular label is located becomes parallel to the pinhole plane Lf of the camera, as shown in Figure 2 whether it is the rectangular edge MN (denoted as a long side, representing the width of the rectangular label) in the plane Lr1 from the perspective of the top view or the rectangular edge CM (denoted as a short side, representing the height of the rectangular label) in the plane Lr1 from the perspective of the side view, the ranging error generated by the distortion thereof is equal to the value 0, and it can be determined that no distortion occurs in the camera.

[0092] In combination with Figure 2It can be seen that, among the two adjacent target detection points M and N, the robot records the plane passing through the first detection point M on the edge to be measured and parallel to the pinhole plane Lf of the camera as a first virtual plane Lv1, and records the plane passing through the second detection point N on the edge to be measured and parallel to the pinhole plane Lf of the camera as a second virtual plane Lv2, and then sets the distance between the first virtual plane Lv1 and the second virtual plane Lv2 to be equal to the ranging error generated by the distortion of the line connecting the two adjacent target detection points M and N. According to the geometric relationship of the triangle, the ranging error is equal to the product of the length U of the line segment MN and the sine value of the inclination angle q of the plane on which the rectangular label is located. When the distance between the first virtual plane Lv1 and the second virtual plane Lv2 is smaller, the ranging error generated by the distortion of the rectangular label (including the distortion of the line connecting the two adjacent target detection points M and N) is smaller,

[0093] On the basis of the above-mentioned embodiments, the method for determining the distortion of the rectangular label in the camera based on the inclination angle of the plane on which the rectangular label is located comprises: in the case that the plane of the object to be measured is not parallel to the pinhole plane of the camera, if the robot calculates that the ranging error U*sin(q) generated by the distortion of the edge parallel to the target straight line direction in the rectangular label in the camera is not equal to the value 0, it is determined that the edge parallel to the target straight line direction in the rectangular label is distorted in the camera, and further it is determined that the rectangular label is distorted. Wherein, the plane on which the rectangular label is located is the aforementioned plane of the object to be measured, and the inclination angle of the plane on which the rectangular label is located is set to determine the ranging error generated by the distortion of the line connecting the two adjacent target detection points in the camera through U*sin(q). In the case that the plane of the object to be measured is not parallel to the pinhole plane of the camera, if the robot calculates that the ranging error U*sin(q) generated by the distortion of the edge to be measured in the rectangular label in the camera is equal to the value 0, it is determined that the edge to be measured in the rectangular label is not distorted in the camera, but the edge parallel to the target straight line direction is distorted in the camera, and then it is determined that the rectangular label as a whole is distorted in the camera, but the edge to be measured in the rectangular label is not distorted in the camera. It should be noted that the edge to be measured in the rectangular label is perpendicular to the target straight line direction and parallel to the pinhole plane of the camera. Therefore, by calculating the ranging error generated by each edge of the rectangular label, it can be determined whether the rectangular label is distorted, which saves the calculation amount and ensures the accuracy of the ranging.

[0094] Specifically, the to-be-measured edges in the rectangular label include the to-be-measured edge where the first detection point is located and the to-be-measured edge where the second detection point is located. Since the to-be-measured edge where the first detection point is located and the to-be-measured edge where the second detection point is located are both parallel to the pinhole plane of the camera, when the distance from the camera to the to-be-measured edge where the first detection point of the rectangular label is located and the distance from the camera to the to-be-measured edge where the second detection point of the rectangular label is located are calculated in step B, the robot determines that the ranging error caused by the distortion of the to-be-measured edge where the first detection point of the rectangular label is located in the camera is equal to the numerical value 0, and the ranging error caused by the distortion of the to-be-measured edge where the second detection point of the rectangular label is located in the camera is equal to the numerical value 0. Therefore, the calculation of the distance from the camera to the to-be-measured edge where the first detection point of the rectangular label is located and the distance from the camera to the to-be-measured edge where the second detection point of the rectangular label is located in step B can be calculated by using the pinhole imaging model, thereby saving the calculation amount and ensuring the ranging accuracy.

[0095] It should be noted that the pinhole plane of the camera is parallel to the imaging plane of the camera; wherein the optical center of the camera is located on the pinhole plane of the camera, and the distance from the optical center of the camera to the imaging plane of the camera is equal to the focal length of the camera. Specifically, the incident light from each target detection point corresponding to each rectangular label intersects at the optical center of the camera; each target detection point corresponding to each rectangular label is distributed in the to-be-measured object plane along a straight line direction and is identified in turn in the process of repeatedly performing step A by the robot, and is then used in step B to calculate the ranging error caused by the distortion of the rectangular label. The ranging error here includes the ranging error caused by the distortion of the line connecting the adjacent two target detection points in the camera, and of course, the ranging error corresponding to the adjacent two target detection points in the same rectangular label can be calculated.

[0096] Preferably, when the distance between the adjacent two target detection points obtained by the robot is less than 0.5*(d1+d2), the robot sets the ranging error caused by the distortion of the line connecting the adjacent two target detection points in the camera to be equal to the numerical value 0. Corresponding to ​In the embodiment, d1+d2 represents the sum of the distance d1 from the first side MC to the camera and the distance d2 from the second side ND to the camera in the same rectangular label, 0.5*(d1+d2) represents half of the sum, and if the half of the sum is greater than the length of the line connecting the two adjacent target detection points, i.e., greater than the size of the rectangular label in the target straight line direction, the distance in the rectangular label of the line connecting the two adjacent target detection points is small and can be ignored, the difference between the actual imaging width in the imaging plane and the ideal imaging width is not large, and the distortion of the rectangular label can also be ignored. In the preferred embodiment, the preset distortion error value is 0.5*(d1+d2), so that when the distance between the two adjacent vertices in the edge line of the rectangular label is less than the preset distortion error value, the line connecting the two adjacent vertices in the edge line of the rectangular label is configured to not be distorted in the camera. In summary, in the process of executing the visual positioning method, the rectangular label in which the distance between the two adjacent target detection points is less than 0.5*(d1+d2) is specially selected for distance and angle calculation, thereby improving the positioning calculation accuracy of the object plane and the rectangular label arranged on the surface thereof.

[0097] As an embodiment, in step A, the robot performs image preprocessing on the image collected by the camera of the robot, including: the robot first performs grayscale processing on the collected image to obtain a binary image; then performs Gaussian smoothing processing on the binary image to remove image noise to obtain a smoothed image; then performs edge detection on the smoothed image to obtain a target edge map, preferably using a Canny operator for edge detection, and the robot can extract a closed curve from the target edge map, which can enclose the closed graph disclosed in the foregoing embodiments. Then, the target edge map is dilated to obtain the preprocessed image, so that the robot can identify the vertices and graphical attributes of the rectangular label from the preprocessed image. Then, the smoothed image is edge detected to obtain a target edge map. The preprocessed edge line is smoother and sharper, and can be clearly imaged in the imaging plane of the camera. Therefore, a monocular camera with a fixed focal length can achieve clear imaging, and all vertices in the rectangular label can be obtained at one time, thereby improving the calculation accuracy of the length of the edge connected by the vertices.

[0098] Preferably, in step A, the method of searching the top point of the rectangular label in the pre-processed image comprises: extracting a closed graph in the pre-processed image, so that the closed graph represents the rectangular label to be searched by the robot; wherein the closed graph is generated in the edge detection; then extracting the top point of the rectangular label and the graphic attribute of the rectangular label based on the closed graph, and performing filtering processing on the extracted top point of the rectangular label and the graphic attribute of the rectangular label to suppress the noise of the rectangular label; and then setting the filtered top point as the top point of the rectangular label searched by the robot in the pre-processed image, and setting the filtered graphic attribute as the graphic attribute of the rectangular label searched by the robot in the pre-processed image. The filtering processing here is a conventional technical means of noise filtering for pixel points in an image, and the purpose is to exclude interference factors for identifying the top point of the rectangular label and the graphic attribute of the rectangular label. The graphic attribute after filtering processing reflects clear and sharp edges, so that the robot obtains accurate graphic attribute and top point.

[0099] Preferably, the closed graph is connected by a plurality of line segments connected at the head and tail, or connected by a plurality of pixel points in sequence, wherein the pixel point at the corner point position of the closed graph is identified as the top point of the closed graph; the graphic attribute of the rectangular label is the edge line feature of the closed graph, including the number of edges surrounding the closed graph and the number of top points of the closed graph, which can be selected for further image filtering processing after pre-processing to exclude noise information interference in the neighborhood. In the closed graph, the robot identifies the line connecting two adjacent top points as an edge surrounding the closed graph. When the robot searches the graphic attribute and the top point of the rectangular label in the pre-processed image, the robot distinguishes labels of different shapes by detecting the number of edges surrounding the closed graph. Thus, the rectangular label is distinguished according to the graphic attribute in the case of less noise interference.

[0100] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A visual positioning method based on rectangular labels, characterized in that, Visual positioning methods include: Step A: The robot preprocesses the images captured by its camera, and then searches for the graphic attributes and vertices of the rectangular labels within the preprocessed images. Step B: Based on the graphic attributes and vertices of the rectangular label, calculate the distance between the rectangular label and the camera, the offset angle of the rectangular label relative to the camera, and the tilt angle of the plane containing the rectangular label using the monocular ranging principle. This ensures that, even when the plane containing the rectangular label is not parallel to the pinhole plane of the camera, the distance between the rectangular label and the camera is the distance from the camera to the side of the rectangular label parallel to the line of intersection with the target. Then, set the distance between the rectangular label and the camera, and the offset angle of the rectangular label relative to the camera, as the visual positioning result of the robot for the positioning device or the rectangular label. The intersection line between the plane containing the rectangular label and the pinhole plane of the camera is denoted as the target intersection line; The rectangular label is placed on the surface of the device to be positioned.

2. The visual positioning method according to claim 1, characterized in that, The visual positioning method further includes: determining the distortion of the rectangular label in the camera based on the tilt angle of the plane where the rectangular label is located, so that when the distance between two adjacent vertices in the edge line of the rectangular label is less than a preset distortion error value, the line connecting two adjacent vertices in the edge line of the rectangular label is configured to not be distorted in the camera.

3. The visual positioning method according to claim 2, characterized in that, When the plane containing the rectangular label is parallel to the pinhole plane of the camera, the robot determines that no side of the rectangular label is distorted in the camera.

4. The visual positioning method according to claim 2, characterized in that, In step A, during the process of the robot searching for the graphic attributes and vertices of the rectangular labels in the preprocessed image, the robot detects the number of edges that form a closed shape. The graphic attributes of the rectangular labels include the edge features of the closed shape, which include the number of edges and vertices of the closed shape. In the edge lines of the closed shape, the robot identifies the line connecting two adjacent vertices as an edge that forms a closed shape. Within the preprocessed image, the robot counts the number of edges of a single closed shape. When the robot counts that there are 4 edges forming a closed shape and there are two sets of edges that are perpendicular to each other, it identifies the currently detected closed shape as a rectangle label. Each set of edges has two parallel edges.

5. The visual positioning method according to claim 4, characterized in that, Methods for calculating the distance between a rectangular label and a camera using a pinhole imaging model include: The robot sets two vertices of a rectangular label distributed along the target line as two adjacent target detection points in the plane of the object to be measured; the line connecting the two adjacent target detection points is parallel to the target line; the plane of the object to be measured is used to represent the surface of the device to be located; Then, among the two adjacent target detection points, the robot designates one of the target detection points as the first detection point and records the distance d1 from the camera to the side to be tested where the first detection point is located; the robot designates the other target detection point as the second detection point and records the distance d2 from the camera to the side to be tested where the second detection point is located; wherein, the side of the rectangular label that is perpendicular to the direction of the target line is the side to be tested of the rectangular label; the distance between the rectangular label and the camera mentioned in step B includes the distance from the camera to the side to be tested where the first detection point is located and the distance from the camera to the side to be tested where the second detection point is located; The lens focal length f of the camera, the side length w of the side to be measured, and the pixel width p formed by the side to be measured in the imaging plane of the camera are obtained in advance; the distance between the side to be measured and the camera is calculated using the following formula: If one endpoint of the edge to be tested is the first detection point, then the edge to be tested is the edge to be tested where the first detection point is located, and then d is set to be equal to the distance d1 from the edge to be tested where the first detection point is located to the camera; If one endpoint of the edge to be tested is the second detection point, then the edge to be tested is the edge to be tested where the second detection point is located. Then, d is set to be equal to the distance d2 from the edge to be tested where the second detection point is located to the camera. When the plane of the object to be measured is not parallel to the pinhole plane of the camera, d1 is not equal to d2; When the plane of the object to be tested is not parallel to the pinhole plane of the camera, the intersection line of the plane of the object to be tested and the pinhole plane of the camera is set perpendicular to the target straight line direction; the side of a rectangular label formed by connecting two adjacent target detection points is parallel to the target straight line direction.

6. The visual positioning method according to claim 5, characterized in that, If the two vertices of the edge to be tested where the first detection point is located are updated to the two adjacent target detection points, then the edges that were originally distributed along the target line direction in the same rectangular label are set as the edge to be tested where the updated target detection point is located, so that the target line direction is updated to be parallel to the edge to be tested where the original first detection point is located; then the distance from the camera to the edge to be tested where the updated target detection point is located is set to be equal to d1.

7. The visual positioning method according to claim 5, characterized in that, Step B specifically includes: The distance U between two adjacent target detection points in the plane of the object to be measured is obtained in advance; Then, the distance d1 from the camera to the side to be measured where the first detection point is located is calculated using the pinhole imaging model, and the distance d2 from the camera to the side to be measured where the second detection point is located is calculated using the pinhole imaging model. Based on the side-angle relationships of a triangle, the following formulas are used to calculate the offset angle of the rectangular label relative to the camera and the tilt angle of the plane containing the rectangular label: d2*sin(b4)=U*sin(q)+d1*sin(b1); U*cos(q)=d2*cos(b4)+d1*cos(b1); b2 = 90 - b1; b3 = 90 - b4; The angle between the perpendicular segment from the camera to the side to be tested where the first detection point is located and the pinhole plane of the camera in the opposite direction of the target straight line is denoted as b1, and the angle between the perpendicular segment from the camera to the side to be tested where the second detection point is located and the pinhole plane of the camera in the target straight line is denoted as b4. Among them, the angle of the rectangular label relative to the camera includes the angle b2 between the perpendicular segment of the camera to the side to be tested where the first detection point is located and the optical axis, and the angle b3 between the perpendicular segment of the camera to the side to be tested where the second detection point is located and the optical axis. The robot sets the angle between the plane of the object to be tested and the pinhole plane of the camera as the tilt angle of the plane of the object to be tested, where q is the tilt angle of the plane of the object to be tested; the tilt angle of the plane where the rectangular label is located is the tilt angle of the plane of the object to be tested. The monocular ranging principle in step B includes the pinhole imaging model.

8. The visual positioning method according to claim 7, characterized in that, When the robot identifies two adjacent target detection points in the plane of the object under test, it calculates the product U*sin(q) of the distance U between the two adjacent target detection points in the plane of the object under test and the sine of the tilt angle q of the plane of the object under test. Then, it sets U*sin(q) as the ranging error caused by the distortion of the line connecting the two adjacent target detection points in the camera.

9. The visual positioning method according to claim 8, characterized in that, The method for determining the distortion of the rectangular label in the camera based on the tilt angle of the plane containing the rectangular label includes: If the plane of the object to be measured is not parallel to the pinhole plane of the camera, and the robot calculates that the ranging error caused by the distortion of the side of the rectangular label that is parallel to the target line direction in the camera is not equal to the value of 0, then it is determined that the side that is parallel to the target line direction is distorted in the camera. If the plane of the object to be measured is not parallel to the pinhole plane of the camera, and the robot calculates that the ranging error caused by the distortion of the side to be measured in the rectangular label in the camera is equal to 0, then it is determined that the side to be measured in the rectangular label does not undergo distortion in the camera. The side of the rectangular label is perpendicular to the straight line direction of the target and parallel to the pinhole plane of the camera.

10. The visual positioning method according to claim 9, characterized in that, When the distance between two adjacent target detection points obtained by the robot is less than 0.5*(d1+d2), the robot sets the ranging error caused by the distortion of the line connecting the two adjacent target detection points in the camera to be equal to the value 0. The preset distortion error value is 0.5*(d1+d2) so that when the distance between two adjacent vertices in the edge line of the rectangular label is less than the preset distortion error value, the line connecting two adjacent vertices in the edge line of the rectangular label is configured to not be distorted in the camera.

11. The visual positioning method according to claim 4, characterized in that, In step A, the robot preprocesses the images captured by its camera in the following ways: The robot first converts the acquired image to grayscale to obtain a binarized image; then it performs Gaussian smoothing on the binarized image to remove image noise, resulting in a smoothed image; then it performs edge detection on the smoothed image to obtain a target edge map; and finally, it performs dilation on the target edge map to obtain the preprocessed image, which allows the robot to identify the vertices and graphic attributes of the rectangular label from the preprocessed image.

12. The visual positioning method according to claim 11, characterized in that, In step A, the method for searching for the vertices of the rectangular labels within the preprocessed image includes: Within the preprocessed image, a closed shape is extracted so that it represents the rectangular label that the robot needs to search for; wherein, the closed shape is generated during the edge detection. Then, based on the closed shape, the vertices and graphic attributes of the rectangular labels are extracted. The extracted vertices and graphic attributes of the rectangular labels are then filtered to suppress noise in the rectangular labels. Then, the filtered vertices are set as the vertices of the rectangular labels that the robot searches for in the preprocessed image, and the filtered graphic attributes are set as the graphic attributes of the rectangular labels that the robot searches for in the preprocessed image.

Images