Robotic vision positioning method and system for installing a door

Abstract

The application relates to the technical field of image data analysis, in particular to a robot visual positioning method and system for door installation, which comprises the following steps: constructing a robot coordinate system based on the position of a wide-angle camera installed on the robot body; determining the first position coordinates of the output end of a robot mechanical arm in the robot coordinate system based on the rotation information of each rotating shaft of the robot mechanical arm; determining the conversion relationship between the robot coordinate system and a world coordinate system based on the first image collected by the wide-angle camera installed on the robot body and a plurality of calibration points of the door installation position; determining the second position coordinates of the output end of the robot mechanical arm in the world coordinate system based on the first position coordinates and the conversion relationship; and positioning the door installation robot based on the second position coordinates and the second image collected by a long-focus micro-lens camera installed on the output end of the robot mechanical arm. The application effectively improves the door installation positioning precision by arranging a plurality of cameras.

Description

Technical Field

[0001] This invention relates to the field of image data analysis technology, and specifically to a robot vision positioning method and system for installing doors. Background Technology

[0002] Traditional industrial door installation relies on manual labor, involving precise operations such as measurement, drilling, and cutting. Inconsistent installation accuracy can arise due to variations in worker skill levels, and working at heights and handling heavy equipment can pose safety hazards. Modern robotics technology, such as robotic arms, combined with vision sensors and relevant algorithms, can accurately locate the robot's posture, replacing some manual labor and improving the efficiency and accuracy of industrial roller shutter door installation.

[0003] Common industrial door installation robot positioning typically involves placing a camera at the output end of the robotic arm and setting calibration points near the door installation location. The target installation position is then determined based on these calibration points. However, this single-camera system generally uses a wide-angle camera to ensure the calibration points are obtained. When the output end is close to the target point, its precision is not as good as that of a telephoto macro lens, resulting in lower positioning accuracy. On the other hand, when a telephoto macro camera is used at the output end of the robotic arm, the distance to the installation target is too close, preventing the acquisition of sufficient calibration information and further degrading positioning performance. Summary of the Invention

[0004] To address the aforementioned technical problem of low positioning accuracy during door installation, the present invention aims to provide a robot vision positioning method and system for door installation, the specific technical solution of which is as follows:

[0005] In a first aspect, the present invention provides a robot vision positioning method and system for installing doors, comprising the following steps:

[0006] A robot coordinate system is constructed based on the position of a wide-angle camera installed on the robot's body;

[0007] Based on the rotation information of each rotation axis of the robot arm, the first position coordinates of the output end of the robot arm in the robot coordinate system are determined;

[0008] Based on the first image captured by the wide-angle camera installed on the robot body and several calibration points at the door installation position, the transformation relationship between the robot coordinate system and the world coordinate system is determined;

[0009] Based on the first position coordinates and the transformation relationship, determine the second position coordinates of the robot arm output end in the world coordinate system;

[0010] Based on the second position coordinates and the second image captured by the telephoto macro camera installed at the output end of the robot arm, the door installation robot is positioned.

[0011] In conjunction with the first aspect above, in some possible implementations, the wide-angle camera mounted on the robot body includes two horizontally distributed wide-angle cameras, constructing a robot coordinate system, including:

[0012] The center point of the two horizontally distributed wide-angle cameras is taken as the origin of the robot coordinate system;

[0013] Based on the origin of the robot coordinate system, the robot coordinate system is determined. The plane formed by the x-axis and y-axis of the robot coordinate system is a horizontal plane, and the y-axis points in the direction of the robot arm.

[0014] In conjunction with the first aspect above, among some possible implementations, determining the first position coordinates of the robot arm's output end in the robot coordinate system includes:

[0015] Based on the current rotation information of each rotation axis of the robot arm, the lower limit rotation information of the current rotation information and the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system are obtained from the pre-acquired relational database information.

[0016] Based on the component influence rate and the difference between the current rotation information and the lower limit rotation information, the change in coordinate influence is determined;

[0017] Based on the lower limit rotation information in the relational database information, the lower limit position coordinates and the change in coordinate influence are used to determine the first position coordinates of the robot arm output end in the robot coordinate system.

[0018] In conjunction with the first aspect above, in some possible implementations, obtaining the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system includes:

[0019] For the target rotation component in the lower limit rotation information, several adjacent rotation information and their corresponding third position coordinates of the robot arm output end in the robot coordinate system are obtained from the relational database information.

[0020] Based on the influence of the target rotation component in the several adjacent rotation information on the change of the third position coordinate, the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system is determined.

[0021] In conjunction with the first aspect above, in some possible implementations, determining the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system includes:

[0022] The target rotation components in the lower limit rotation information and the several adjacent rotation information are arranged in ascending order to obtain the target rotation component sequence;

[0023] Determine the first difference between the third position coordinates of each target rotation component in the target rotation component sequence and the corresponding third position coordinates of the previous target rotation component, on the same coordinate axis.

[0024] Determine the second difference between each target rotation component and its previous target rotation component in the target rotation component sequence;

[0025] The ratio of the first difference to the second difference is determined, and the mean of all ratios is determined, thereby obtaining the component influence rate of the target rotation component relative to each coordinate axis of the robot coordinate system.

[0026] In conjunction with the first aspect mentioned above, among some possible implementation methods, determining the change in coordinate influence includes:

[0027] Based on the difference between each rotation component in the current rotation information and the lower limit rotation information, the rotation component difference value is obtained;

[0028] The product of the rotation component difference value and the corresponding component influence rate in the lower limit rotation information is accumulated to obtain the influence change value on each coordinate axis in the robot coordinate system;

[0029] Based on the influence change values ​​on each coordinate axis in the robot coordinate system, the amount of coordinate influence change is determined.

[0030] In conjunction with the first aspect above, in some possible implementations, determining the transformation relationship between the robot coordinate system and the world coordinate system includes:

[0031] Based on the fourth position coordinates of the aforementioned calibration points in the robot coordinate system and the fifth position coordinates of each point in the first image, and in conjunction with the camera intrinsic parameter matrix of the wide-angle camera, the translation matrix is ​​determined.

[0032] Determine the rotation matrix between the robot coordinate system and the world coordinate system;

[0033] Based on the rotation matrix and the translation matrix, the transformation relationship between the robot coordinate system and the world coordinate system is determined.

[0034] In conjunction with the first aspect mentioned above, among some possible implementations, the translation matrix is ​​determined, including:

[0035] Using OCR recognition technology, the plurality of calibrated points are matched with each point in the first image to determine the matching points of the plurality of calibrated points in the first image;

[0036] Based on the fourth position coordinates of the aforementioned calibration points in the robot coordinate system, the fifth position coordinates of the matching points in the first image, and in conjunction with the camera intrinsic parameter matrix of the wide-angle camera, the translation matrix is ​​determined using the PNP algorithm.

[0037] In conjunction with the first aspect above, in some possible implementations, the wide-angle camera mounted on the robot body includes two wide-angle cameras horizontally symmetrically distributed relative to the origin of the robot coordinate system, determining the transformation relationship between the robot coordinate system and the world coordinate system, including:

[0038] The mean of the translation matrices corresponding to the two wide-angle cameras is determined to obtain the average translation matrix;

[0039] Based on the rotation matrix and the average translation matrix, the transformation relationship between the robot coordinate system and the world coordinate system is determined.

[0040] Secondly, the present invention also provides a robot vision positioning system for installing doors, including a memory and a processor. The memory is used to store executable computer program code, and the processor is used to call and run the executable computer program code from the memory, causing the system to perform the methods in the first aspect or any possible implementation thereof.

[0041] Thirdly, the present invention also provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in the first aspect or any possible implementation thereof.

[0042] Fourthly, the present invention also provides a computer-readable storage medium storing computer program code that, when executed on a computer, causes the computer to perform the method described in the first aspect or any possible implementation thereof.

[0043] This invention offers the following advantages: By installing a multi-camera system on the robot—specifically, a wide-angle camera on the robot body and a telephoto macro camera at the output end of the robot arm—a robot coordinate system is constructed based on the wide-angle camera's position. Combined with the rotation information of each axis of the robot arm, the first position coordinates of the robot arm's output end within this coordinate system are determined, thus achieving positioning of the robot arm's output end. Furthermore, based on several pre-set calibration points at the door installation location and the first image captured by the wide-angle camera, a transformation relationship between the robot coordinate system and the world coordinate system is determined. This transformation relationship is then used to convert the coordinates of the robot arm's output end from the robot coordinate system to the world coordinate system, achieving positioning of the robot arm's output end. Finally, combined with the image captured by the telephoto macro camera at the robot arm's output end, precise positioning of the door installation location is achieved. This invention effectively improves positioning accuracy by using a wide-angle camera mounted on the robot body to position the robot arm's output end and a telephoto macro camera mounted at the same location to precisely position the installation target. Attached Figure Description

[0044] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the steps of a robot visual positioning method for installing doors according to an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of the robot coordinate system constructed according to an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of a robot vision positioning system for installing doors according to an embodiment of the present invention. Detailed Implementation

[0048] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0049] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0050] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0051] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0052] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0053] Although operations or steps are described in a specific order in the accompanying drawings in the embodiments of the present invention, this should not be construed as requiring these operations or steps to be performed in the specific order or serial order shown, or requiring all of the shown operations or steps to be performed to obtain the desired result. In the embodiments of the present invention, these operations or steps may be performed serially; they may be performed in parallel; or a portion of these operations or steps may be performed.

[0054] Furthermore, it is understood that the data involved in the technical solutions of this invention (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions. Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains, and all parameters or indicators in the formulas involved in this invention are normalized values ​​that have eliminated the influence of dimensions.

[0055] To address the aforementioned technical problem of low positioning accuracy during door installation, this invention provides a robot vision positioning method and system for door installation. This method uses a wide-angle camera mounted on the robot body to position the output end of the robot arm, and utilizes a telephoto macro camera mounted on the output end of the robot arm to accurately position the installation target, effectively improving positioning accuracy.

[0056] The following will describe in detail, with reference to the accompanying drawings, a robot vision positioning method and system for installing doors provided by an embodiment of the present invention.

[0057] Figure 1 This diagram illustrates a basic flowchart of a robot vision positioning method for installing doors, as provided in an embodiment of the present invention. Figure 1 As shown, the method specifically includes the following steps:

[0058] Step S100: Construct a robot coordinate system based on the position of the wide-angle camera installed on the robot body.

[0059] Specifically, during the door installation process of the robot, to address the issue of decreased positioning performance of a single camera system when the distance between the camera and the target is close, a multi-camera system is installed on the robot to achieve precise visual positioning. This multi-camera system includes three cameras: one camera is mounted at the front end of the robot's robotic arm (i.e., the output end of the robotic arm), which is a telephoto macro camera used to monitor the precise spatial position of the robotic arm in real time; this camera can also be called the working camera. The other two cameras are mounted on the robot's body, outside the robotic arm; these two cameras are wide-angle cameras, capable of capturing the widest possible image angle, used to assist in monitoring the spatial position of the robotic arm; these two cameras can also be called auxiliary cameras. The auxiliary cameras and the working camera have the same sampling frequency, and the two auxiliary cameras are mounted at the same height, but the working camera has a higher resolution than the auxiliary cameras.

[0060] For the two auxiliary cameras fixed to the robot, since their positions are fixed during installation and the distance between them remains constant, and because the robot moves via rollers or tracks, its distance from the ground remains constant, the robot coordinate system (a spatial coordinate system built around the robot) can be constructed using a horizontal plane. In the world coordinate system (a coordinate system constructed based on the door frame's calibration position), if its xOy plane is also set with the horizontal plane as the base plane, then the z-axis of the world coordinate system and the robot coordinate system are parallel, and the xOy plane is also parallel. Therefore, when converting the spatial coordinates of the robot arm's output end relative to the robot coordinate system to the world coordinate system, the z-axis can be calculated using the difference in origin coordinates. The x and y axes can be considered as a transformation between two coordinate systems within the same plane. Furthermore, based on the motion, the output end's coordinates are in the robot coordinate system, so a conversion to the world coordinate system is also necessary. This allows for the positioning of the robot arm's output end.

[0061] First, construct a robot coordinate system based on the position of the wide-angle camera mounted on the robot's body. For example, the position of any wide-angle camera can be used as the origin of the robot coordinate system, and the plane formed by the x-axis and y-axis of the robot coordinate system must be a horizontal plane, with the z-axis pointing vertically.

[0062] In one possible implementation, the robot coordinate system is constructed, including:

[0063] Step S101: Use the center point of the two horizontally distributed wide-angle cameras as the origin of the robot coordinate system;

[0064] Step S102: Based on the origin of the robot coordinate system, determine the robot coordinate system. The plane formed by the x-axis and y-axis of the robot coordinate system is a horizontal plane, and the y-axis points in the direction of the robot arm.

[0065] In a specific example, the horizontal line connecting the two auxiliary cameras is taken as the x-axis of the robot coordinate system, with the positive direction from left to right. The center point of the line is taken as the origin of the coordinate system. The direction perpendicular to the x-axis, which is in the same plane as the two auxiliary cameras and the origin, is taken as the y-axis of the coordinate system, with the positive direction pointing towards the robot arm. The z-axis is perpendicular to the ground and pointing upwards. Thus, the robot coordinate system can be obtained. Figure 2 A schematic diagram of the constructed robot coordinate system is shown. Here, the robot camera refers to the auxiliary camera. In the constructed robot coordinate system, the coordinates of the two auxiliary cameras are... and , It represents half the installation interval between the two auxiliary cameras.

[0066] Step S200: Based on the rotation information of each rotation axis of the robot arm, determine the first position coordinates of the robot arm output end in the robot coordinate system.

[0067] Specifically, for a multi-camera system mounted on a robot, the working camera located at the output end of the robotic arm is mainly used for precise positioning of mechanical components. At close range, it cannot capture calibration information, leading to a decrease in positioning performance. Therefore, two auxiliary cameras located at other positions on the robot are needed to assist in positioning and increase accuracy. Typically, the robotic arm is an integral part of the robot, moving through multiple rotating axes. The range of motion is related to the rotation angle of each axis. Therefore, after the robotic arm is mounted on the robot, the base is usually stationary. The position of the output end is controlled by adjusting the rotation of each axis. Since the length between each axis is determined during design and manufacturing, the output end's position relative to itself remains unchanged when each axis rotates to the same angle. Even if the robot as a whole moves, its position relative to itself does not change. Therefore, a coordinate relationship between the rotation of the axes and the robot's own coordinates can be established. By collecting real-time data on the rotation of the axes, the robot's coordinates relative to itself can be calculated and converted to world coordinates using the coordinate transformation relationship established in space by the auxiliary cameras. The robot coordinates obtained by using rotation relationships, after transformation, become the world coordinates of the robot's output end working camera at this moment. This can be considered as the positioning of the output end working camera in space, thus assisting the working camera in precise positioning.

[0068] In one possible implementation, determining the first position coordinates of the robot arm's output end in the robot coordinate system includes:

[0069] Step S201: Based on the current rotation information of each rotation axis of the robot arm, obtain the lower limit rotation information of the current rotation information and the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system from the pre-acquired relational database information.

[0070] Specifically, for a robotic arm with multiple rotation axes, taking a robotic arm with three rotation axes as an example, the rotation angle of each rotation axis is used as its feature. The overall rotation information can then be represented as [R1, R2, R3], where R1, R2, and R3 represent the rotation angles of the three rotation axes, respectively. Simultaneously, the coordinates Q of the working camera (i.e., the output end of the robotic arm) in the robot coordinate system are recorded based on this rotation information. Based on the rotation information of the multiple rotation axes of the robotic arm and the coordinates of the robotic arm's output end in the robot coordinate system, a relational database of information for each rotation axis of the robotic arm can be constructed.

[0071] Because real-time adjustment of the robotic arm's rotation angle may require greater precision, the rotation information resulting from this angle may not be found in the pre-built relational database, but rather within the measured angle range. For example, if the robotic arm has three rotation axes, and the real-time rotation information is [34.123, 36.235, -12.120], the built relational database only shows 34.12 and 34.13 for the first rotation component and 36.23 and 36.24 for the second. However, the effect of a single axis's movement within a small range on the coordinates can be considered linear. Therefore, the rate of change of the single axis component relative to the coordinates can be calculated using the relational database information, thus obtaining the coordinates of the robotic arm's output end in the machine coordinate system corresponding to the actual motion vector.

[0072] Therefore, for the current rotation information of each rotation axis of the robotic arm, the lower limit rotation information of each rotation axis is obtained from the relational database information of each rotation axis of the robotic arm. Specifically, each rotation component (i.e., the rotation angle of each rotation axis) in the lower limit rotation information is lower than each rotation component (i.e., the rotation angle of each rotation axis) in the current rotation information, and each rotation component in the lower limit rotation information is closest to each rotation component in the previous rotation information. For example, for the aforementioned rotation information [34.123, 36.235, -12.120], the corresponding lower limit rotation information obtained from the relational database is [34.12, 36.23, -12.12]. Simultaneously, based on the rotation information and corresponding coordinates in the relational database, the rate of change of the relative coordinates of a single axis component is calculated, thereby obtaining the coordinates of the robotic arm output end in the machine coordinate system corresponding to the actual motion vector.

[0073] Furthermore, in one possible implementation, the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system is obtained, including: for the target rotation component in the lower limit rotation information, obtaining several adjacent rotation information and their corresponding third position coordinates of the robot arm output end in the robot coordinate system from the relational database information; and determining the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system based on the influence of the target rotation component in the several adjacent rotation information on the change of the third position coordinates.

[0074] Furthermore, in one possible implementation, determining the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system includes: arranging the target rotation components in the lower limit rotation information and several adjacent rotation information in ascending order to obtain a target rotation component sequence; determining the first difference between each target rotation component in the target rotation component sequence and the previous target rotation component on the same coordinate axis corresponding to the third position coordinate; determining the second difference between each target rotation component in the target rotation component sequence and the previous target rotation component; determining the ratio of the first difference to the second difference, and determining the mean of all ratios, thereby obtaining the component influence rate of the rotation axis corresponding to the target rotation component relative to each coordinate axis in the robot coordinate system.

[0075] In a specific example, each rotation component in the lower limit rotation information is controlled as a variable, while the other rotation components remain unchanged. Several adjacent rotation information components of the lower limit rotation information are obtained from the relational database. That is, for the i-th rotation component in the lower limit rotation information, several rotation components are obtained before and after the i-th rotation component, such as 5 rotation components, thus obtaining several adjacent rotation information components. Simultaneously, the coordinates of the robot arm output end in the robot coordinate system are also obtained for each of these adjacent rotation information components. For example, if the first component of the lower limit rotation information [34.12, 36.23, -12.12] is used as a variable, and the other two rotation components remain unchanged, then [34.07, 36.23, -12.12], [34.08, 36.23, -12.12], ..., [34.17, 36.23, -12.12] are taken as several adjacent rotation information components, and the corresponding coordinates of the robot arm output segment in the machine coordinate system are obtained. , , ..., .

[0076] Based on the acquired lower limit rotation information, its several adjacent rotation information, and the corresponding coordinates of the robot arm output segment in the machine coordinate system, the rate of change of the influence of the rotation axis corresponding to each rotation component on the coordinate is calculated. Specifically, the rate of influence of the rotation axis corresponding to the i-th rotation component on the x-axis in the robot coordinate system is:

[0077] ;

[0078] In the formula, The value represents the influence rate of the rotation axis corresponding to the i-th rotation component in the rotation information on the x-axis in the robot coordinate system, that is, the rate of change of the influence of the i-th rotation component on the x-axis component in the robot coordinate system when the i-th rotation component is controlled to change; N represents the number of adjacent rotation information selected from the relational database information by the lower limit rotation information. The x-axis component value of the position coordinate of the robot arm output end in the robot coordinate system corresponding to the j-th rotation information in the lower limit rotation information and its adjacent rotation information; The x-axis component value of the position coordinate of the robot arm output end in the robot coordinate system corresponding to the (j-1)th rotation information in the lower limit rotation information and its adjacent rotation information; This represents the i-th rotation component of the j-th rotation information among the lower limit rotation information and its adjacent rotation information; It represents the i-th rotation component of the (j-1)-th rotation information in the lower limit rotation information and its adjacent rotation information.

[0079] The change rate was calculated based on the above. In the same way, the influence rate of the rotation axis corresponding to other rotation components on each coordinate axis in the robot coordinate system is calculated, thereby determining the local influence rate of the rotation component of a single rotation axis on each coordinate axis.

[0080] Step S202: Based on the component influence rate and combined with the difference between the current rotation information and the lower limit rotation information, determine the change in coordinate influence.

[0081] Specifically, the difference between the current rotation information and its lower limit rotation information is analyzed, and the change in coordinate influence is determined by combining the influence rate of this component.

[0082] Furthermore, in one possible implementation, determining the change in coordinate influence includes: obtaining the rotation component difference value based on the difference between each rotation component in the current rotation information and the lower limit rotation information; accumulating the product of the rotation component difference value in the lower limit rotation information and the corresponding component influence rate to obtain the change in influence value on each coordinate axis in the robot coordinate system; and determining the change in coordinate influence based on the change in influence value on each coordinate axis in the robot coordinate system.

[0083] In a specific example, by controlling each rotation component and calculating its rate of change of influence on the coordinates, the influence of each rotation component on the coordinate components can be obtained. Taking the influence of each rotation axis on the x-axis in the robot coordinate system as an example, the total change in coordinate influence is obtained by combining the changes in the influence of all rotation components on the coordinate axes:

[0084] ;

[0085] In the formula: This represents the amount of change in the x-axis coordinate in the robot coordinate system caused by each rotation axis. This represents the influence rate of the rotation axis corresponding to the i-th rotation component in the rotation information on the x-axis in the robot coordinate system. This represents the i-th rotation component of the current rotation information (i.e., the data when the robotic arm is actually rotating); This represents the i-th rotation component of the lower limit rotation information; This represents the difference value of the rotational components, that is, the increment of the i-th rotational axis relative to the lower limit value; This indicates the total number of rotation axes of the robot arm.

[0086] Based on the above determination, the amount of change in the influence of each rotation axis on the x-axis in the robot coordinate system is determined. Using the same method, the changes in the influence of each rotation axis on the y-axis and z-axis of the robot coordinate system can be obtained separately. and Therefore, the amount of change caused by the coordinates can be determined. .

[0087] Step S203: Based on the lower limit rotation information in the relational database information, the lower limit position coordinates and the change in the influence of the coordinates, determine the first position coordinates of the robot arm output end in the robot coordinate system.

[0088] Specifically, the lower limit position coordinates of the lower limit rotation information based on the current rotation information (i.e., the data when the robotic arm is actually rotating) in the relational database information. And in combination with the coordinate influence change determined above This allows us to determine the position coordinates of the robot arm's output end in the robot coordinate system. At this time there is .

[0089] The above method constructs a machine coordinate system using the position of an auxiliary camera mounted on the robot, and obtains the position coordinates of the robot arm's output end in the robot coordinate system using the rotation information of each rotation axis of the robotic arm. .

[0090] Step S300: Based on the first image captured by the wide-angle camera installed on the robot body and several calibration points at the door installation position, determine the transformation relationship between the robot coordinate system and the world coordinate system.

[0091] Specifically, by using images captured in real time by an auxiliary camera mounted on the robot, the spatial position of the camera represented by each frame of the image is calculated. Then, the transformation relationship between the robot coordinate system and the world coordinate system is constructed, and the robot coordinates at the output end of the robotic arm are converted into world coordinates using this transformation relationship, thus completing the fine positioning.

[0092] Furthermore, in one possible implementation, the transformation relationship between the robot coordinate system and the world coordinate system is determined, including:

[0093] Step S301: Based on the fourth position coordinates of several calibration points in the robot coordinate system and the fifth position coordinates of each point in the first image, and combined with the camera intrinsic parameter matrix of the wide-angle camera, determine the translation matrix.

[0094] Specifically, to identify the world coordinates of the camera based on key points in images captured by an auxiliary camera mounted on the robot's body, calibration points were set at specific locations on the door frame to be installed. Multiple calibration points were set based on the door frame's installation location, with reasonable spacing between each point to ensure the auxiliary camera could acquire at least four points. Each frame of the auxiliary camera captures the calibration points in the real world (door frame). Based on perspective principles, by identifying the points in the image and mapping them to the actual points, the distance between the camera's image point and the calibration points can be obtained. Thus, a three-dimensional coordinate system of the real world can be established, and the camera's spatial coordinates can be obtained in real time based on each frame of the image.

[0095] Furthermore, in one possible implementation, determining the translation matrix includes: using OCR recognition technology to match several calibration points with each point in the first image to determine the matching points of the several calibration points in the first image; and using the PNP algorithm to determine the translation matrix based on the fourth position coordinates of the several calibration points in the robot coordinate system, the fifth position coordinates of the matching points in the first image, and in combination with the camera intrinsic parameter matrix of the wide-angle camera.

[0096] In a specific example, based on real-time images acquired by an auxiliary camera, the calibration point information in the images is identified. Using the perspective principle and the PNP algorithm, combined with camera intrinsic parameters, a real coordinate system is established to determine the spatial position of the camera within this system, thereby determining the translation matrix. The specific implementation process includes:

[0097] First, a world coordinate system is established based on the calibration points set on the door frame to be installed. When establishing the world coordinate system, one of the actual calibration points is generally used as the origin, and the xOy plane is set with the horizontal plane as the base plane, and the vertical upward direction is used as the y-axis. To facilitate the transformation between the robot coordinate system and the world coordinate system, the xOy plane of the constructed world coordinate system should be at the same horizontal height as the xOy plane of the robot coordinate system. Then, based on the distances between the actual calibration points, the spatial coordinates of the other calibration points in the world coordinate system are determined.

[0098] Secondly, OCR recognition technology is used to obtain character information of the calibration points in the image and match them to determine that each point in the image represents the actual calibration point.

[0099] Next, the camera intrinsic parameters are obtained based on the factory information and Zhang Zhengyou's calibration, and the camera's position in the world coordinate system at the time corresponding to each frame of the image is obtained according to the PNP algorithm.

[0100] The PNP algorithm takes as input the coordinate information in the world coordinate system, the coordinates of the corresponding points in the image coordinate system, and the camera intrinsic parameter matrix; and outputs the rotation matrix R and the translation matrix T of the camera coordinate system.

[0101] For example, in the world coordinate system, there is a point Its coordinates are The corresponding point in the image is In the camera coordinate system, Coordinates are In the image coordinate system, it is The unit is described as mm, not pixel coordinates, therefore:

[0102] ;

[0103] ;

[0104] In the formula: and These represent the rotation matrix and translation matrix calculated by the PNP algorithm, respectively. The size is Translation matrix The size is ; A matrix representing the camera's intrinsic parameters.

[0105] Finally, based on the locations in the image that can match the actual calibration points, obtain the points in the world coordinate system. With at least four points, the image pixels can be converted to camera coordinates. Combining these multiple matching points, a system of equations is formed, and the rotation matrix is ​​solved using methods such as least squares. Translation matrix Among them, the translation matrix The three components can be considered as the movement components of the camera's location in the world coordinate system, thus if We can assume that the coordinates of the camera origin in the world coordinate system are... Rotation matrix This represents the camera's rotation in the world coordinate system.

[0106] Step S302: Determine the rotation matrix between the robot coordinate system and the world coordinate system.

[0107] Specifically, the above process obtains the camera world coordinates corresponding to each frame of the auxiliary camera. Because using a single auxiliary camera can only determine the position of the auxiliary camera in the world coordinate system, it is impossible to accurately construct the transformation relationship between the robot coordinate system and the world coordinate system. Therefore, a robot coordinate system is established by using two auxiliary cameras, and the coordinate transformation matrix of the robot coordinate system is calculated according to existing technology, thereby converting the robot coordinate system into the world coordinate system, and thus converting the coordinates of the robotic arm output end in the robot coordinate system into the coordinates in the world coordinate system.

[0108] In a specific example, since the xOy planes of the machine coordinate system and the world coordinate system are parallel and their z-axis is parallel, the rotation matrix between the robot coordinate system and the world coordinate system can be determined using the transformation relationship between coordinate system rotations. Therefore, based on this rotation matrix and the translation matrix determined above. This allows us to obtain the coordinates of the robot's coordinates in the world coordinate system. This is because obtaining the rotation matrix... The specific process is existing technology and will not be elaborated here.

[0109] Step S303: Based on the rotation matrix and translation matrix, determine the transformation relationship between the robot coordinate system and the world coordinate system.

[0110] Specifically, the rotation matrix of the camera coordinate system can be determined through the above process. Translation matrix The translation matrix These are the coordinates of the camera origin in the world coordinate system, therefore based on the rotation matrix. Translation matrix This allows us to determine the transformation relationship between the robot coordinate system and the world coordinate system.

[0111] Furthermore, in one possible implementation, determining the transformation relationship between the robot coordinate system and the world coordinate system includes: determining the mean of the translation matrices corresponding to the two wide-angle cameras to obtain the average translation matrix; and determining the transformation relationship between the robot coordinate system and the world coordinate system based on the rotation matrix and the average translation matrix.

[0112] In a specific example, since the two auxiliary cameras are horizontally symmetrically distributed relative to the origin of the robot coordinate system, the coordinates of the origin of the robot coordinate system in the world coordinate system are the average of the translation matrices of the two auxiliary cameras. Let the translation matrices of the two auxiliary cameras be T1 and T2, then the coordinates of the origin of the robot coordinate system in world space are... The translation matrix when transforming the robot coordinate system to the world coordinate system. Let T1 and T2 be the average translation matrices of the two auxiliary cameras, then we have .

[0113] Based on the rotation matrix between the robot coordinate system and the world coordinate system determined above. and translation matrix The coordinates in the robot coordinate system can be converted to coordinates in the world coordinate system. The corresponding conversion relationship is as follows:

[0114] ;

[0115] In the formula: This represents the coordinates in the world coordinate system obtained after coordinate transformation; This represents the coordinates in the robot's coordinate system before the coordinate transformation; This represents the angle by which the world coordinate system rotates counterclockwise relative to the robot coordinate system's xOy plane; This represents the coordinates of the origin in the robot's coordinate system in the world coordinate system. and These represent the rotation and translation matrices between the robot coordinate system and the world coordinate system, respectively.

[0116] The above analysis of images acquired by two auxiliary cameras and calibration points determines the rotation and translation matrices between the robot coordinate system and the world coordinate system, thereby accurately determining the transformation relationship between the robot coordinate system and the world coordinate system.

[0117] Step S400: Based on the first position coordinates and the transformation relationship, determine the second position coordinates of the robot arm output end in the world coordinate system.

[0118] Specifically, by utilizing the transformation relationship between the robot coordinate system and the world coordinate system, the position coordinates of the robot arm's output end in the robot coordinate system can be transformed to the world coordinate system, thereby obtaining the position coordinates of the robot arm's output end in the world coordinate system:

[0119] ;

[0120] In the formula: This indicates the position coordinates of the robot arm's output end in the robot coordinate system. Position coordinates in the world coordinate system after coordinate system transformation; and These represent the rotation and translation matrices between the robot coordinate system and the world coordinate system, respectively.

[0121] Step S500: Based on the second position coordinates and the second image captured by the telephoto macro camera installed at the output end of the robot arm, the door installation robot is positioned.

[0122] Specifically, the above method determines the position coordinates of the robot arm's output end in the world coordinate system. This method can achieve coarse spatial positioning of the working camera at the output end of the robot arm, and then combine the fine features of the working camera to finally determine the fine position coordinates of the working camera in the world coordinate system, thereby realizing the positioning of the door installation robot. Common methods include: using the fine image acquired by the working camera to compare with the globally preset image to obtain its precise position.

[0123] Furthermore, after the door installation robot is positioned, the specific door installation process is carried out based on the precise position coordinates of the working camera at the output end of the robotic arm. For example, the difference between the precise position coordinates of the working camera at the output end of the robotic arm and the pre-installation target position of the door is identified, and the rotation angles of each rotating axis of the robotic arm are adjusted until the positioning requirements are met.

[0124] In a specific example, after identifying the precise position coordinates of the robotic arm's output end in the world coordinate system, the spatial distance between it and the pre-installation target position of the door is obtained according to a preset installation procedure. Using an adjustment controller, the movement of each rotation axis of the robotic arm is adjusted to transport the door installation material towards the pre-installation target position. When the coordinates of the robotic arm's output end and the pre-installation target position of the door reach the preset required accuracy, relevant equipment is activated for installation, such as drilling holes at predetermined positions.

[0125] Based on the same inventive concept, embodiments of the present invention also provide a robot vision positioning system for installing doors, such as... Figure 3 As shown, the system includes: a memory 301, a processor 302, and computer program code 303 stored in the memory 301 and running on the processor 302, wherein when the processor 302 executes the computer program code 303, the system can execute any of the robot vision positioning methods for installing doors described above.

[0126] In this embodiment of the invention, the system can be divided into functional modules according to the above method example. For example, each module can correspond to a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0127] Based on the same inventive concept, embodiments of the present invention also provide a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute any of the robot vision positioning methods for installing doors described above.

[0128] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing computer program code, which, when executed on a computer, causes the computer to perform any of the aforementioned robot vision positioning methods for installing doors.

[0129] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A robot vision positioning method for installing doors, characterized in that, Includes the following steps: A robot coordinate system is constructed based on the position of a wide-angle camera installed on the robot's body; Based on the rotation information of each rotation axis of the robot arm, the first position coordinates of the output end of the robot arm in the robot coordinate system are determined; Based on the first image captured by the wide-angle camera installed on the robot body and several calibration points at the door installation position, the transformation relationship between the robot coordinate system and the world coordinate system is determined; Based on the first position coordinates and the transformation relationship, determine the second position coordinates of the robot arm output end in the world coordinate system; Based on the second position coordinates and the second image captured by the telephoto macro camera installed at the output end of the robot arm, the door installation robot is positioned.

2. The robot vision positioning method for installing doors according to claim 1, characterized in that, The wide-angle camera mounted on the robot's body includes two horizontally distributed wide-angle cameras, which construct the robot's coordinate system, including: The center point of the two horizontally distributed wide-angle cameras is taken as the origin of the robot coordinate system; Based on the origin of the robot coordinate system, the robot coordinate system is determined. The plane formed by the x-axis and y-axis of the robot coordinate system is a horizontal plane, and the y-axis points in the direction of the robot arm.

3. The robot vision positioning method for installing doors according to claim 1, characterized in that, Determining the first position coordinates of the robot arm's output end in the robot coordinate system includes: Based on the current rotation information of each rotation axis of the robot arm, the lower limit rotation information of the current rotation information and the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system are obtained from the pre-acquired relational database information. Based on the component influence rate and the difference between the current rotation information and the lower limit rotation information, the change in coordinate influence is determined; Based on the lower limit rotation information in the relational database information, the lower limit position coordinates and the change in coordinate influence are used to determine the first position coordinates of the robot arm output end in the robot coordinate system.

4. The robot vision positioning method for installing doors according to claim 3, characterized in that, Obtaining the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system includes: For the target rotation component in the lower limit rotation information, several adjacent rotation information and their corresponding third position coordinates of the robot arm output end in the robot coordinate system are obtained from the relational database information. Based on the influence of the target rotation component in the several adjacent rotation information on the change of the third position coordinate, the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system is determined.

5. The robot vision positioning method for installing doors according to claim 4, characterized in that, Determining the component influence rate of each rotation axis relative to each coordinate axis in the robot coordinate system includes: The target rotation components in the lower limit rotation information and the several adjacent rotation information are arranged in ascending order to obtain the target rotation component sequence; Determine the first difference between the third position coordinates of each target rotation component in the target rotation component sequence and the corresponding third position coordinates of the previous target rotation component, on the same coordinate axis. Determine the second difference between each target rotation component and its previous target rotation component in the target rotation component sequence; The ratio of the first difference to the second difference is determined, and the mean of all ratios is determined, thereby obtaining the component influence rate of the target rotation component relative to each coordinate axis of the robot coordinate system.

6. The robot vision positioning method for installing doors according to claim 3, characterized in that, Determine the changes in coordinate influence, including: Based on the difference between each rotation component in the current rotation information and the lower limit rotation information, the rotation component difference value is obtained; The product of the rotation component difference value and the corresponding component influence rate in the lower limit rotation information is accumulated to obtain the influence change value on each coordinate axis in the robot coordinate system; Based on the influence change values ​​on each coordinate axis in the robot coordinate system, the amount of coordinate influence change is determined.

7. The robot vision positioning method for installing doors according to claim 1, characterized in that, Determining the transformation relationship between the robot coordinate system and the world coordinate system includes: Based on the fourth position coordinates of the aforementioned calibration points in the robot coordinate system and the fifth position coordinates of each point in the first image, and in conjunction with the camera intrinsic parameter matrix of the wide-angle camera, the translation matrix is ​​determined. Determine the rotation matrix between the robot coordinate system and the world coordinate system; Based on the rotation matrix and the translation matrix, the transformation relationship between the robot coordinate system and the world coordinate system is determined.

8. The robot vision positioning method for installing doors according to claim 7, characterized in that, Determine the translation matrix, including: Using OCR recognition technology, the plurality of calibrated points are matched with each point in the first image to determine the matching points of the plurality of calibrated points in the first image; Based on the fourth position coordinates of the aforementioned calibration points in the robot coordinate system, the fifth position coordinates of the matching points in the first image, and in conjunction with the camera intrinsic parameter matrix of the wide-angle camera, the translation matrix is ​​determined using the PNP algorithm.

9. The robot vision positioning method for installing doors according to claim 7, characterized in that, The wide-angle camera mounted on the robot's body includes two wide-angle cameras horizontally symmetrically distributed relative to the origin of the robot's coordinate system. It determines the transformation relationship between the robot's coordinate system and the world coordinate system, including: The mean of the translation matrices corresponding to the two wide-angle cameras is determined to obtain the average translation matrix; Based on the rotation matrix and the average translation matrix, the transformation relationship between the robot coordinate system and the world coordinate system is determined.

10. A robot vision positioning system for installing doors, characterized in that, The method includes a memory, a processor, and executable computer program code stored in the memory and executable on the processor, wherein the processor executes the computer program code to perform a method as described in any one of claims 1 to 9.

Images