Automatic mounting system control method and control device based on parallel visual servo
Through parallel visual servo technology to identify the occlusion condition of the hook in a narrow space and calculate its center position, automatic mounting without marks is realized, solving the problems of low mounting accuracy and efficiency in the existing technology, and reducing the need for manual assisted operation .
Patent Information
- Application Number
- CN202210330422.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-03-30
AI Technical Summary
When the existing technology is equipped with automatic mounting in a narrow space, intelligent mounting cannot be achieved and the docking upper and lower hooks require manual auxiliary operation when blocking, resulting in low hanging accuracy and efficiency.
Using the automatic mounting system control method based on parallel visual servo, the relative position of the upper and lower platforms is determined by obtaining image information, the occlusion of the hook is identified by using two-dimensional image segmentation technology and three-dimensional point cloud segmentation technology, and the center position of the hook is calculated, and the Implement tagless hook docking.
It improves the accuracy and efficiency of hanging and installing, reduces the intensity of manual labor, realizes automatic hanging and installing in a narrow space, and avoids the problem of docking difficulties.
Smart Images

Figure CN114924598B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of automatic equipment mounting, and in particular to a control method and a control device for an automatic mounting system based on parallel visual servoing. Background Art
[0002] Visual servoing refers to the behavior of automatically receiving and processing the image of a real object through optical devices and non-contact sensors, and using the information fed back by the image to allow the machine system to further control the machine or make corresponding adaptive adjustments.
[0003] At present, automatic mounting of equipment (mounting parts) is often achieved through visual servoing and fusion with laser marking. However, this method has the following disadvantages: (1) When the upper and lower hooks are blocked, the recognition and positioning of the upper and lower hooks are unstable and require manual assistance; (2) In some mounting tasks, laser marking is not allowed and its use is restricted.
[0004] Therefore, there is an urgent need to provide a mature and reliable rapid mounting technology that can be operated in a narrow space, so as to ensure the smooth implementation of automatic mounting. Summary of the invention
[0005] The purpose of the present invention is to provide an automatic mounting system control method and control device based on parallel visual servoing, so as to solve the problem that the equipment cannot be mounted intelligently when laser marking is not allowed, and the problem that manual assistance is required when the upper and lower hooks of the docking are blocked, thereby improving the mounting accuracy and efficiency and reducing the labor intensity.
[0006] In a first aspect, an embodiment of the present invention provides a control method for an automatic mounting system based on parallel visual servoing, comprising:
[0007] Acquire image information of each target position, and determine the relative position and posture of the upper platform and the lower platform in the automatic mounting system based on the image information;
[0008] Based on the relative posture of the upper platform and the lower platform and a pre-stored kinematic model, determining the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system;
[0009] Controlling the joint movement of the posture adjustment mechanism according to the motion control amount to achieve docking of the hook pairs at each target position in the automatic mounting system;
[0010] Wherein, each target position is the position of each hook pair in the automatic mounting system.
[0011] According to the automatic mounting system control method based on parallel visual servoing provided by the present invention, the hook pair consists of an upper hook and a lower hook;
[0012] Determining the relative position of the upper platform and the lower platform in the automatic mounting system based on the image information includes:
[0013] Determine, according to the image information, the occlusion conditions of the upper hook and the lower hook in each hook pair;
[0014] Determine the center of the upper hook and the center of the lower hook of each hook pair by using the shielding conditions of the upper hook and the lower hook in each hook pair;
[0015] Based on the upper hook circle center and the lower hook circle center of each hook pair, the relative position and posture of the upper platform and the lower platform are determined.
[0016] According to the automatic mounting system control method based on parallel visual servoing provided by the present invention, determining the occlusion conditions of the upper hook and the lower hook in each hook pair according to the image information includes:
[0017] Based on the two-dimensional image segmentation technology and the three-dimensional point cloud segmentation technology, the edge of the image information is extracted to obtain the plane contour image and the three-dimensional point cloud image of each hook pair;
[0018] Identifying the plane contour image and the three-dimensional point cloud image of each hook pair;
[0019] For each hook pair, if it is identified that the upper hook and the lower hook in the hook pair are not blocked, the blocking condition of the upper hook and the lower hook in the hook pair is a first type of blocking condition;
[0020] If it is identified that the upper hook and the lower hook in the hook pair are blocked and the ratio between the blocked area and the total area of the blocked hook does not exceed the preset ratio, then the blocking situation of the upper hook and the lower hook in the hook pair is the second type of blocking situation;
[0021] Otherwise, the obstruction condition of the upper hook and the lower hook in the hook pair is the third type of obstruction condition.
[0022] According to the automatic mounting system control method based on parallel visual servoing provided by the present invention, the upper hook circle center and the lower hook circle center of each hook pair are determined by utilizing the occlusion conditions of the upper hook and the lower hook in each hook pair, including:
[0023] If the occlusion condition of the upper hook and the lower hook in the hook pair is not the third type of occlusion condition, using the three-dimensional point cloud image of the hook pair to determine the center of the upper hook and the center of the lower hook of the hook pair;
[0024] Otherwise, using the plane contour image of the hook pair and the distance between the pixel points in the plane contour image of the hook pair, the upper hook circle center and the lower hook circle center of the hook pair are determined;
[0025] Among them, the distance between pixel points in the plane contour image of the hook pair is determined based on the plane contour image of the hook pair when it experiences the first type of occlusion or the second type of occlusion and the actual positions of its corresponding upper hook center and lower hook center.
[0026] According to the automatic mounting system control method based on parallel visual servoing provided by the present invention, the upper hook circle center and the lower hook circle center of the hook pair are determined by using the three-dimensional point cloud map of the hook pair, including:
[0027] Downsampling the upper / lower hook point clouds in the three-dimensional point cloud image of the hook pair, and performing normal estimation on the downsampled upper / lower hook point clouds;
[0028] According to the normal of each point in the down-sampled upper / lower hook point cloud, the down-sampled upper / lower hook point cloud is segmented by clustering to obtain the largest segmented point cloud block;
[0029] When the shielding condition of the upper hook and the lower hook in the hook pair is the first type of shielding condition:
[0030] Determine the inner boundary of the outer surface of the upper / lower hook using the maximum point cloud block;
[0031] Using the inner boundary to fit a plane circle, and using the center of the plane circle as the center of the upper / lower hook;
[0032] When the shielding condition of the upper hook and the lower hook in the hook pair is the second type of shielding condition:
[0033] Averaging the normal lines of each point in the largest point cloud block to obtain an average normal line;
[0034] Based on the average normal line and the planar contour image of the hook pair, a planar Hough circle is determined using a Hough circle detection algorithm;
[0035] Calculating a three-dimensional projection of the planar Hough circle;
[0036] Determine the spatial center of the circle corresponding to the center of the planar Hough circle according to the three-dimensional projection of the planar Hough circle;
[0037] The center of the space is used as the center of the upper / lower hook.
[0038] According to the automatic mounting system control method based on parallel visual servoing provided by the present invention, the relative position and posture of the upper platform and the lower platform are determined based on the upper hook circle center and the lower hook circle center of each hook pair, including:
[0039] The plane where the centers of the upper hook circles of all hook alignments are located is taken as the upper plane, and an upper plane coordinate system is established with the centers of the upper hook circles of all hook alignments as the origin;
[0040] For each hook pair, an upper hook coordinate system corresponding to the upper hook in the hook pair is established according to the standard coordinate system establishment rules;
[0041] Based on the coordinates of the center of the upper hook in the hook pair in the camera coordinate system, determine the conversion relationship matrix between the camera coordinate system and the upper hook coordinate system, and based on the coordinates of the center of the upper hook in the hook pair in the upper plane coordinate system, determine the conversion relationship matrix between the upper hook coordinate system and the upper plane coordinate system;
[0042] Based on the coordinates of the center of the lower hook of the hook pair in the camera coordinate system and the conversion relationship matrix between the camera coordinate system and the upper hook coordinate system, determine the coordinates of the center of the lower hook of the hook pair in the upper hook coordinate system;
[0043] Based on the coordinates of the center of the lower hook of the hook pair in the upper hook coordinate system and the transformation relationship matrix between the upper hook coordinate system and the upper plane coordinate system, determine the coordinates of the center of the lower hook of the hook pair in the upper plane coordinate system;
[0044] Fitting the lower plane using the coordinates of the center of the lower hook of all hook pairs in the upper plane coordinate system, and determining the rotation matrix of the lower plane relative to the upper plane;
[0045] The rotation matrix of the lower plane relative to the upper plane is used to characterize the relative position and posture of the upper platform and the lower platform.
[0046] According to the automatic mounting system control method based on parallel visual servoing provided by the present invention, the standard coordinate system establishment rule is:
[0047] The origin is located at the center of the hook circle on the hook centering, and the X-axis direction, the Y-axis direction and the Z-axis direction are consistent with the X-axis direction, the Y-axis direction and the Z-axis direction of the upper plane.
[0048] In a second aspect, an embodiment of the present invention provides an automatic mounting system control device based on parallel visual servoing, the device comprising:
[0049] The upper and lower platforms relative posture determination module is used to obtain image information of each target position, and determine the relative posture of the upper platform and the lower platform in the automatic mounting system based on the image information of each target position;
[0050] A motion control amount determination module, used to determine the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system based on the relative posture of the upper platform and the lower platform and a pre-stored kinematic model;
[0051] A motion control module, used to control the joint motion of the posture adjustment mechanism according to the motion control amount, so as to achieve docking of the hooks at each target position in the automatic mounting system;
[0052] Wherein, each target position is the position of each hook pair in the automatic mounting system.
[0053] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the automatic mounting system control method based on parallel visual servoing as described in the first aspect is implemented.
[0054] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic mounting system control method based on parallel visual servoing as described in the first aspect.
[0055] The present invention provides a control method and control device for an automatic mounting system based on parallel visual servo, which obtains image information of each target position, and determines the relative posture of the upper platform and the lower platform in the automatic mounting system based on the image information; uses a multi-eye parallel servo structure and a pure visual algorithm to track the positioning of the upper hook and the lower hook in each hook pair, to compensate for the problem of alignment difficulties caused by the deformation of the mounting parts to a certain extent, and does not require the arrangement of various marks, which is relatively safe and convenient under special mounting requirements; based on the relative posture of the upper platform and the lower platform and the pre-stored kinematic model, determines the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system; controls the joint motion of the posture adjustment mechanism according to the motion control amount, so as to realize the docking of the hook pairs at each target position in the automatic mounting system; wherein, each target position is the position of each hook pair in the automatic mounting system. Automatic mounting of unmarked mounting parts is realized by docking each hook pair, which improves the mounting accuracy and efficiency and reduces the labor intensity. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0057] Figure 1It is a flow chart of the automatic mounting system control method based on parallel visual servoing provided by the present invention;
[0058] Figure 2 It is a schematic diagram of upper and lower posture fitting provided by the present invention;
[0059] Figure 3 It is a structural schematic diagram of an automatic mounting system control device based on parallel visual servoing provided by the present invention;
[0060] Figure 4 It is a physical diagram of the automatic mounting system control device based on parallel visual servoing provided by the present invention;
[0061] Figure 5 This is an example diagram of the human-computer interaction interface provided by the present invention;
[0062] Figure 6 It is a structural schematic diagram of an electronic device for realizing an automatic mounting system control method based on parallel visual servoing provided by the present invention. DETAILED DESCRIPTION
[0063] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] The automatic mounting system is set with multiple hook pairs consisting of upper hooks and lower hooks. In each hook pair, the upper hook is fixed to the upper platform, and the lower hook is fixed to the lower platform carrying the equipment. During the entire mounting process, the upper platform is always controlled to be stationary, and the joints of the posture adjustment mechanism are controlled to drive the lower platform to move, so as to achieve the docking of the upper hook and the lower hook in each hook pair, thereby achieving the mounting of the equipment.
[0065] Combine the following Figure 1-Figure 6 The present invention is described.
[0066] First, as Figure 1 As shown, the automatic mounting system control method based on parallel visual servoing provided by the present invention includes:
[0067] S11, acquiring image information of each target position, and determining the relative position and posture of the upper platform and the lower platform in the automatic mounting system based on the image information;
[0068] The present invention realizes automatic mounting of equipment by means of parallel visual servoing, that is, each pair of hooks is monitored separately to determine the relative posture of the upper platform and the lower platform in the automatic mounting system; the relative posture of the upper and lower platforms is used to realize the motion control of each joint of the posture adjustment mechanism, thereby realizing the docking of the hooks in the automatic mounting system.
[0069] S12, determining the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system based on the relative posture of the upper platform and the lower platform and a pre-stored kinematic model;
[0070] It can be understood that the pre-stored kinematic model is a decoupled kinematic model of the posture adjustment mechanism, which is established by relevant personnel after the automatic mounting system is arranged.
[0071] The present invention adopts the existing algorithm to calculate the motion control amount of each joint in the posture adjustment mechanism when the decoupled kinematic model of the posture adjustment mechanism and the relative posture of the upper platform and the lower platform are known, which will not be elaborated here.
[0072] S13, controlling the joint movement of the posture adjustment mechanism according to the motion control amount, so as to achieve docking of the hook pairs at each target position in the automatic mounting system;
[0073] Wherein, each target position is the position of each hook pair in the automatic mounting system.
[0074] The present invention provides a control method and control device for an automatic mounting system based on parallel visual servo, which obtains image information of each target position, and determines the relative position and posture of the upper platform and the lower platform in the automatic mounting system based on the image information; uses a multi-eye parallel servo structure and a pure visual algorithm to track the positioning of the upper hook and the lower hook in each hook pair, to compensate for the problem of alignment difficulties caused by the deformation of the mounting parts to a certain extent, and does not require the arrangement of various marks, which is relatively safe and convenient under special mounting requirements; based on the relative position and posture of the upper platform and the lower platform and the pre-stored kinematic model, determines the motion control amount of each joint in the posture adjustment mechanism; controls the joint motion of the posture adjustment mechanism according to the motion control amount, so as to achieve the docking of the hook pairs at each target position in the automatic mounting system; wherein, each target position is the position of each hook pair in the automatic mounting system. Automatic mounting of unmarked mounting parts is achieved by docking each hook pair, which improves the mounting accuracy and efficiency and reduces the labor intensity.
[0075] Based on the above embodiments, as an optional embodiment, the hook pair consists of an upper hook and a lower hook;
[0076] Determining the relative position of the upper platform and the lower platform in the automatic mounting system based on the image information includes:
[0077] Determine, according to the image information, the occlusion conditions of the upper hook and the lower hook in each hook pair;
[0078] It is understandable that during the lifting stage of the mounting component, the upper and lower hook images transition from an unobstructed state to a fully obstructed state, and the obstruction of the upper and lower hooks will affect the precise positioning of the upper and lower hook positions, thereby affecting the docking effect of the upper and lower hooks.
[0079] Therefore, when determining the relative posture of the upper and lower platforms, the present invention takes the shielding conditions of the upper and lower hooks of each hook as a key factor.
[0080] Determine the center of the upper hook and the center of the lower hook of each hook pair by using the shielding conditions of the upper hook and the lower hook in each hook pair;
[0081] It can be understood that the docking of the upper and lower hooks in the center of each hook is equivalent to the alignment of the upper and lower hook docking holes in the center of each hook. Therefore, the center of the upper hook docking hole is taken as the center of the upper hook, referring to the upper hook position, and the center of the lower hook docking hole is taken as the center of the lower hook, referring to the lower hook position; the relative position between the centers of the upper hook and the lower hook is the relative posture of the upper and lower hooks.
[0082] In order to ensure the accuracy and stability of the docking process, the present invention specifically provides a method for calculating the relative positions of the centers of the upper and lower hooks based on the shielding conditions of the upper and lower hooks, laying the foundation for determining the relative positions of the upper and lower platforms.
[0083] Based on the upper hook circle center and the lower hook circle center of each hook pair, the relative position and posture of the upper platform and the lower platform are determined.
[0084] The relative postures of the upper and lower hooks of each hook pair are integrated into the relative posture of the upper and lower platforms, providing data basis for the motion control of each joint of the posture adjustment mechanism, ensuring the smooth execution of automatic hanging of the hanging parts.
[0085] The present invention determines the calculation method of the relative posture of the upper and lower hooks in a targeted manner according to the shielding conditions of the upper and lower hooks during the hanging process, thereby improving the calculation accuracy of the relative posture of the upper and lower platforms, and solving the problem that the upper and lower hook docking holes cannot be automatically aligned due to the shielding of the upper and lower hooks at the end of the hanging stage and must be manually assisted in the alignment.
[0086] Based on the above embodiments, as an optional embodiment, determining the shielding conditions of the upper hook and the lower hook in each hook pair according to the image information includes:
[0087] Based on the two-dimensional image segmentation technology and the three-dimensional point cloud segmentation technology, the edge of the image information is extracted to obtain the plane contour image and the three-dimensional point cloud image of each hook pair;
[0088] The present invention can adopt the deep learning instance segmentation based on Mask-RCNN Plus combined with the traditional region growing algorithm target segmentation scheme to achieve the pixel-level segmentation of the two-dimensional image of the upper and lower hooks and the corresponding three-dimensional point cloud segmentation.
[0089] It can be understood that both the two-dimensional image segmentation technology and the three-dimensional point cloud segmentation technology are types of visual detection technologies. The present invention uses a purely visual method to locate and track the upper and lower hooks without the need to arrange various markers. It is relatively safe and convenient under special hanging requirements.
[0090] Identifying the plane contour image and the three-dimensional point cloud image of each hook pair;
[0091] The positional relationship between the upper and lower hooks in the hook pair is clear in the plane contour image and the three-dimensional point cloud image obtained after edge extraction processing, and the occlusion of the upper and lower hooks in the hook pair can be determined directly.
[0092] For each hook pair, if it is identified that the upper hook and the lower hook in the hook pair are not blocked, the blocking condition of the upper hook and the lower hook in the hook pair is a first type of blocking condition;
[0093] If it is identified that the upper hook and the lower hook in the hook pair are blocked and the ratio between the blocked area and the total area of the blocked hook does not exceed the preset ratio, the blocking condition of the upper hook and the lower hook in the hook pair is the second type of blocking condition;
[0094] Otherwise, the obstruction condition of the upper hook and the lower hook in the hook pair is the third type of obstruction condition.
[0095] The lifting stage of the hanging part can be divided into three small stages. Assuming the preset ratio is 50%, the lower hook is blocked at the end of hanging; then the upper and lower hooks do not overlap in the first stage, that is, the upper hook and the lower hook are not blocked; in the second stage, the lower hook begins to enter the groove of the upper hook, that is, the lower hook is blocked by the upper hook and the blocked area of the lower hook does not exceed half of the total area of the lower hook; in the third stage, the lower hook is blocked by the upper hook and the blocked area of the lower hook exceeds half of the total area of the lower hook; in the first and second stages, the planes where the upper and lower hooks are located are in an intersecting relationship, and in the third stage, the planes where the upper and lower hooks are located are in a parallel relationship.
[0096] The present invention performs segmentation of the upper and lower hooks in each hook alignment through visual detection, and identifies the occlusion of the upper and lower hooks based on the segmentation result, with high identification accuracy.
[0097] Based on the above embodiments, as an optional embodiment, the method of determining the upper hook circle center and the lower hook circle center of each hook pair by using the shielding conditions of the upper hook and the lower hook in each hook pair includes:
[0098] If the occlusion condition of the upper hook and the lower hook in the hook pair is not the third type of occlusion condition, using the three-dimensional point cloud image of the hook pair to determine the center of the upper hook and the center of the lower hook of the hook pair;
[0099] For hook pairs that do not meet the third type of occlusion conditions, it is necessary to calculate the relative positions of the upper and lower hooks through point cloud processing.
[0100] Otherwise, using the plane contour image of the hook pair and the distance between the pixel points in the plane contour image of the hook pair, the upper hook circle center and the lower hook circle center of the hook pair are determined;
[0101] Among them, the distance between pixel points in the plane contour image of the hook pair is determined based on the plane contour image of the hook pair when it experiences the first type of occlusion or the second type of occlusion and the actual positions of its corresponding upper hook center and lower hook center.
[0102] It meets the third type of occlusion situation, which means that the planes where the upper and lower hooks are located are parallel. In this case, there is no need to calculate the posture. Only the left and right and up and down movement relationship between the upper and lower platforms needs to be calculated. At this time, the relative posture of the upper and lower hooks can be determined based on the two-dimensional plane contour image and the distance between the pixels in the plane contour image.
[0103] It should be understood that when the hook pair experiences the third occlusion situation, it must have experienced the first occlusion situation and the second occlusion situation. In this case, by using the plane contour image of the hook pair in the first occlusion situation or the second occlusion situation and the actual positions of the upper and lower hook centers of the hook pair obtained through point cloud processing, the scale information of the plane contour image of the hook pair (that is, the distance between pixels) can be inferred.
[0104] The present invention identifies the relative posture of the upper and lower hooks of each hook pair by combining point cloud processing with two-dimensional image processing, laying a foundation for the relative posture calculation of the upper and lower platforms.
[0105] Based on the above embodiments, as an optional embodiment, the use of the three-dimensional point cloud image of the hook pair to determine the upper hook circle center and the lower hook circle center of the hook pair includes:
[0106] Downsampling the upper / lower hook point clouds in the three-dimensional point cloud image of the hook pair, and performing normal estimation on the downsampled upper / lower hook point clouds;
[0107] The purpose of downsampling the upper / lower hook point clouds in the three-dimensional point cloud image is to reduce the scale of the upper / lower hook point clouds in the three-dimensional point cloud image while maintaining the topological features; the normal estimation adopts a conventional algorithm, which will not be repeated here.
[0108] According to the normal of each point in the down-sampled upper / lower hook point cloud, the down-sampled upper / lower hook point cloud is segmented by clustering to obtain the largest segmented point cloud block;
[0109] After point cloud segmentation, several relatively smooth point cloud blocks can be obtained. Here, the largest point cloud block is selected.
[0110] When the shielding condition of the upper hook and the lower hook in the hook pair is the first type of shielding condition:
[0111] Determine the inner boundary of the outer surface of the upper / lower hook using the maximum point cloud block;
[0112] At this time, the outer surface of the upper / lower hook is first obtained by plane fitting; then the projection points are vectored and the angle between adjacent vectors is calculated; finally, the edge points are determined according to the size of the angle, the edge is extracted, and the inner boundary of the outer surface of the upper / lower hook is obtained.
[0113] Using the inner boundary to fit a plane circle, and using the center of the plane circle as the center of the upper / lower hook;
[0114] The center of the circle can be determined by fitting a plane circle according to the inner boundary and the RANSAC algorithm.
[0115] When the shielding condition of the upper hook and the lower hook in the hook pair is the second type of shielding condition:
[0116] Averaging the normal lines of each point in the largest point cloud block to obtain an average normal line;
[0117] Based on the average normal line and the planar contour image of the hook pair, a planar Hough circle is determined using a Hough circle detection algorithm;
[0118] Calculating a three-dimensional projection of the planar Hough circle;
[0119] Determine the spatial center of the circle corresponding to the center of the planar Hough circle according to the three-dimensional projection of the planar Hough circle;
[0120] The center of the space is used as the center of the upper / lower hook.
[0121] The present invention adopts different methods for determining the relative postures of the upper and lower hooks according to different shielding conditions of the upper and lower hooks, thereby improving the recognition accuracy of the relative postures of the upper and lower hooks of each hook pair and ensuring the accuracy and stability of the docking process.
[0122] Based on the above embodiments, as an optional embodiment, determining the relative position of the upper platform and the lower platform based on the upper hook circle center and the lower hook circle center of each hook pair includes:
[0123] The plane where the centers of the upper hook circles of all hook alignments are located is taken as the upper plane, and an upper plane coordinate system is established with the centers of the upper hook circles of all hook alignments as the origin;
[0124] For each hook pair, an upper hook coordinate system corresponding to the upper hook in the hook pair is established according to the standard coordinate system establishment rules;
[0125] Based on the coordinates of the center of the upper hook in the hook pair in the camera coordinate system, determine the conversion relationship matrix between the camera coordinate system and the upper hook coordinate system, and based on the coordinates of the center of the upper hook in the hook pair in the upper plane coordinate system, determine the conversion relationship matrix between the upper hook coordinate system and the upper plane coordinate system;
[0126] Based on the coordinates of the center of the lower hook of the hook pair in the camera coordinate system and the conversion relationship matrix between the camera coordinate system and the upper hook coordinate system, determine the coordinates of the center of the lower hook of the hook pair in the upper hook coordinate system;
[0127] Based on the coordinates of the center of the lower hook of the hook pair in the upper hook coordinate system and the transformation relationship matrix between the upper hook coordinate system and the upper plane coordinate system, determine the coordinates of the center of the lower hook of the hook pair in the upper plane coordinate system;
[0128] Fitting the lower plane using the coordinates of the center of the lower hook of all hook pairs in the upper plane coordinate system, and determining the rotation matrix of the lower plane relative to the upper plane;
[0129] The rotation matrix of the lower plane relative to the upper plane is used to characterize the relative position and posture of the upper platform and the lower platform.
[0130] The present invention determines the rotation matrix of the lower plane relative to the upper plane by establishing a coordinate system and transforming coordinates, thereby representing the relative posture of the upper and lower platforms and laying a foundation for the motion control of each joint of the posture adjustment mechanism. Figure 2 An example of a schematic diagram of upper and lower pose fitting is shown. The corresponding automatic mounting system has four hook pairs. Taking the first hook pair as an example, the upper hook is represented by up1, the lower hook is represented by down1, the camera coordinate system is represented by c1, and the upper plane coordinate system is represented by up.
[0131] Therefore, the coordinates of the center of the upper hook in the camera coordinate system can be expressed as:
[0132]
[0133] The coordinates of the center of the lower hook in the camera coordinate system can be expressed as:
[0134]
[0135] The upper hook coordinate system corresponding to the upper hook is expressed in the camera coordinate system as follows:
[0136]
[0137] in, and Respectively represent the projection vector of the unit length of each coordinate axis of the upper hook coordinate system on the x, y and z axes of the camera coordinate system;
[0138] Therefore, the center of the lower hook circle in the upper hook coordinate system can be expressed as follows:
[0139]
[0140] Assuming that the upper plane is a rectangle, and the long and short sides are a and b respectively, the upper hook coordinate system is expressed in the upper plane coordinate system as follows:
[0141]
[0142] The coordinates of the center of the lower hook circle in the upper plane coordinate system are calculated by the following formula:
[0143]
[0144] So far, the coordinates of the four lower hook center points in the upper plane coordinate system have been obtained, the lower plane is fitted through four sets of coordinate values, and the rotation matrix of the lower plane relative to the upper plane is calculated.
[0145] Because the motion of the lower platform (motion platform) is relative to the reference coordinate system parallel to the ground, under the assumption that the upper platform is parallel to the ground, it can be calculated by Calculate the motion control amount.
[0146] Based on the above embodiments, as an optional embodiment, the standard coordinate system establishment rule is:
[0147] The origin is located at the center of the hook circle on the hook centering, and the X-axis direction, the Y-axis direction and the Z-axis direction are consistent with the X-axis direction, the Y-axis direction and the Z-axis direction of the upper plane.
[0148] This embodiment only provides a feasible standard coordinate system establishment rule. In actual application, the standard coordinate system establishment rule that can achieve the same purpose can be freely set according to the working conditions.
[0149] In the second aspect, the automatic mounting system control device based on parallel visual servoing provided by the present invention is described. The automatic mounting system control device based on parallel visual servoing described below and the automatic mounting system control method based on parallel visual servoing described above can be referenced to each other. Figure 3The following is a structural diagram of an automatic mounting system control device based on parallel visual servoing. Figure 3 As shown, the device includes: a module 21 for determining the relative position of the upper and lower platforms, a module 22 for determining the motion control amount, and a module 23 for motion control;
[0150] The upper and lower platform relative posture determination module 21 is used to obtain image information of each target position, and determine the relative posture of the upper platform and the lower platform in the automatic mounting system based on the image information of each target position;
[0151] A motion control amount determination module 22, for determining the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system based on the relative posture of the upper platform and the lower platform and a pre-stored kinematic model;
[0152] A motion control module 23, used to control the joint motion of the posture adjustment mechanism according to the motion control amount, so as to achieve docking of the hooks at each target position in the automatic mounting system;
[0153] Wherein, each target position is the position of each hook pair in the automatic mounting system.
[0154] Based on the above embodiments, as an optional embodiment, the hook pair is composed of an upper hook and a lower hook; the upper and lower platform relative posture determination module 21 includes:
[0155] A visual detection unit, used to determine the occlusion conditions of the upper hook and the lower hook in each hook pair according to the image information;
[0156] An upper and lower hook center position determination unit, used to determine the upper hook center and the lower hook center of each hook pair by using the shielding conditions of the upper hook and the lower hook in each hook pair;
[0157] The upper and lower platform relative posture determination unit is used to determine the relative posture of the upper platform and the lower platform based on the upper hook circle center and the lower hook circle center of each hook pair.
[0158] Based on the above embodiments, as an optional embodiment, the visual detection unit includes: the visual detection unit includes:
[0159] An image segmentation submodule is used to extract edges of the image information based on two-dimensional image segmentation technology and three-dimensional point cloud segmentation technology to obtain a plane contour image and a three-dimensional point cloud image of each hook pair;
[0160] A recognition submodule, used for recognizing the plane contour image and the three-dimensional point cloud image of each hook pair;
[0161] The upper and lower hook occlusion determination submodule is used for, for each hook pair, if it is identified that the upper hook and the lower hook in the hook pair are not occluded, then the occlusion of the upper hook and the lower hook in the hook pair is the first type of occlusion;
[0162] If it is identified that the upper hook and the lower hook in the hook pair are blocked and the ratio between the blocked area and the total area of the blocked hook does not exceed the preset ratio, the blocking condition of the upper hook and the lower hook in the hook pair is the second type of blocking condition;
[0163] Otherwise, the obstruction condition of the upper hook and the lower hook in the hook pair is the third type of obstruction condition.
[0164] Based on the above embodiments, as an optional embodiment, the upper and lower hook center position determination unit includes:
[0165] A judging submodule, used for judging whether the shielding condition of the upper hook and the lower hook in the hook pair is the third type of shielding condition;
[0166] an execution submodule, for analyzing the plane contour image of the hook pair if yes, and determining the upper hook circle center and the lower hook circle center of the hook pair;
[0167] If not, the upper hook circle center and the lower hook circle center of the hook pair are determined using the three-dimensional point cloud image of the hook pair.
[0168] Based on the above embodiments, as an optional embodiment, the execution submodule specifically includes:
[0169] A normal estimation subunit, used for downsampling the upper / lower hook point clouds in the three-dimensional point cloud image of the hook pair, and performing normal estimation on the downsampled upper / lower hook point clouds;
[0170] A point cloud segmentation subunit, configured to perform point cloud segmentation on the downsampled upper / lower hook point cloud in a clustering manner according to the normals of each point in the downsampled upper / lower hook point cloud, to obtain a maximum point cloud block;
[0171] An inner boundary determination subunit, configured to determine the inner boundary of the outer surface of the upper / lower hook using the maximum point cloud block when the occlusion condition of the upper hook and the lower hook in the hook pair is the first type of occlusion condition;
[0172] A fitting subunit, used for fitting a plane circle using the inner boundary, and taking the center of the plane circle as the center of the upper / lower hook;
[0173] an average normal determination subunit, configured to average the normals of the points in the maximum point cloud block to obtain an average normal when the occlusion condition of the upper hook and the lower hook in the hook pair is the second type of occlusion condition;
[0174] A Hough circle detection subunit, configured to determine a planar Hough circle by using a Hough circle detection algorithm based on the average normal line and the planar contour image of the hook pair;
[0175] A calculation subunit, used for calculating the three-dimensional projection of the planar Hough circle;
[0176] A space circle center determination subunit, used to determine the space circle center corresponding to the center of the planar Hough circle according to the three-dimensional projection of the planar Hough circle;
[0177] The setting subunit is used to use the center of the space as the center of the upper / lower hook.
[0178] Based on the above embodiments, as an optional embodiment, the upper and lower platforms relative posture determination unit includes:
[0179] The upper plane coordinate system establishment submodule is used to take the plane where the centers of the upper hook circles of all hook pairs are located as the upper plane, and establish the upper plane coordinate system with the centers of the upper hook circles of all hook pairs as the origin;
[0180] An upper hook coordinate system establishment submodule is used to establish, for each hook pair, an upper hook coordinate system corresponding to the upper hook in the hook pair according to a standard coordinate system establishment rule;
[0181] A submodule for determining a conversion relationship matrix between coordinate systems, which determines a conversion relationship matrix between the camera coordinate system and the upper hook coordinate system based on the coordinates of the center of the upper hook in the hook pair in the camera coordinate system, and determines a conversion relationship matrix between the upper hook coordinate system and the upper plane coordinate system based on the coordinates of the center of the upper hook in the hook pair in the upper plane coordinate system;
[0182] A first coordinate transformation submodule, for determining the coordinates of the center of the lower hook of the hook pair in the upper hook coordinate system based on the coordinates of the center of the lower hook of the hook pair in the camera coordinate system and the transformation relationship matrix between the camera coordinate system and the upper hook coordinate system;
[0183] A second coordinate transformation submodule, for determining the coordinates of the center of the lower hook of the hook pair in the upper plane coordinate system based on the coordinates of the center of the lower hook of the hook pair in the upper hook coordinate system and the transformation relationship matrix between the upper hook coordinate system and the upper plane coordinate system;
[0184] A rotation matrix determination submodule, used to fit the lower plane using the coordinates of the center of the lower hook of all hook pairs in the upper plane coordinate system, and determine the rotation matrix of the lower plane relative to the upper plane;
[0185] The rotation matrix of the lower plane relative to the upper plane is used to characterize the relative position and posture of the upper platform and the lower platform.
[0186] Based on the above embodiments, as an optional embodiment, the standard coordinate system establishment rule is:
[0187] The origin is located at the center of the hook circle on the hook centering, and the X-axis direction, the Y-axis direction and the Z-axis direction are consistent with the X-axis direction, the Y-axis direction and the Z-axis direction of the upper plane.
[0188] Figure 4 A physical diagram of an automatic mounting system control device based on parallel visual servoing is provided, such as Figure 4 As shown, the entity includes: multiple front-end visual processing modules, industrial computers, routers, motion controllers and human-computer interaction interfaces;
[0189] Each front-end visual processing module corresponds to a hook pair, and includes a binocular camera and an image preprocessor, wherein the binocular camera is used to photograph the corresponding hook pair, and the image preprocessor is used to process the corresponding photographed image;
[0190] In addition, the front-end visual processing module adopts a high-performance edge computing framework based on deep learning. The binocular camera inside it can use the Intel Realsense series depth camera, and the image front processor can use a high-performance computer with the Nvidia Jetson series chip as the core.
[0191] The industrial computer, according to the two-dimensional plane contour map and the three-dimensional point cloud map of the hook map provided by the front visual processing module, sequentially executes the corresponding functions of the upper and lower hook circle center position determination unit, the upper and lower platform relative posture determination unit and the motion control module to obtain the motion control amount of each joint of the posture adjustment mechanism;
[0192] The router transfers the motion control quantity of each joint of the posture adjustment mechanism sent by the industrial computer to the motion controller;
[0193] Based on the motion control amount, the motion controller automatically controls the joints of the posture adjustment mechanism through the built-in motion control card and the actuator, that is, executes the corresponding functions of the motion control module;
[0194] The human-computer interaction interface is located in an external computer and directly sends human-computer interaction instructions to the motion controller to perform manual control of the joints of the posture adjustment mechanism or control state switching.
[0195] It should be noted that, in the entity of the automatic mounting system control device, if working conditions permit, two binocular cameras can also share one image pre-processor.
[0196] Figure 5 It is an example of a human-computer interaction interface, which includes operation instructions of lifting, moving forward, moving left, friendship, and automatic alignment;
[0197] It is understandable that the present invention can also set the human-computer interaction function in the motion controller, and the specific setting is subject to the actual working conditions.
[0198] Thirdly, Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the automatic mounting system control method based on parallel visual servoing, the method comprising: acquiring image information of each target position, and determining the relative posture of the upper platform and the lower platform in the automatic mounting system based on the image information; determining the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system based on the relative posture of the upper platform and the lower platform and the pre-stored kinematic model; controlling the joint motion of the posture adjustment mechanism according to the motion control amount to achieve the docking of the hook pairs at each target position in the automatic mounting system; wherein each target position is the position of each hook pair in the automatic mounting system.
[0199] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0200] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned automatic mounting system control method based on parallel visual servoing, obtain image information of each target position, and determine the relative posture of the upper platform and the lower platform in the automatic mounting system based on the image information; determine the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system based on the relative posture of the upper platform and the lower platform and a pre-stored kinematic model; control the joint movement of the posture adjustment mechanism according to the motion control amount to achieve the docking of the hook pairs at each target position in the automatic mounting system; wherein, each target position is the position of each hook pair in the automatic mounting system.
[0201] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A control method for an automatic mounting system based on parallel visual servoing, characterized in that: The method comprises: Acquire image information of each target position, and determine the relative position and posture of the upper platform and the lower platform in the automatic mounting system based on the image information; Based on the relative posture of the upper platform and the lower platform and a pre-stored kinematic model, determining the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system; Controlling the joint movement of the posture adjustment mechanism according to the motion control amount to achieve docking of the hook pairs at each target position in the automatic mounting system; Wherein, each target position is the position of each hook pair in the automatic mounting system; The hook pair consists of an upper hook and a lower hook; Determining the relative position of the upper platform and the lower platform in the automatic mounting system based on the image information includes: Determine, according to the image information, the occlusion conditions of the upper hook and the lower hook in each hook pair; Determine the center of the upper hook and the center of the lower hook of each hook pair by using the shielding conditions of the upper hook and the lower hook in each hook pair; Based on the upper hook circle center and the lower hook circle center of each hook pair, the relative position and posture of the upper platform and the lower platform are determined.
2. The automatic mounting system control method based on parallel visual servoing according to claim 1 is characterized in that: Determining the occlusion conditions of the upper hook and the lower hook in each hook pair according to the image information includes: Based on the two-dimensional image segmentation technology and the three-dimensional point cloud segmentation technology, the edge of the image information is extracted to obtain the plane contour image and the three-dimensional point cloud image of each hook pair; Identifying the plane contour image and the three-dimensional point cloud image of each hook pair; For each hook pair, if it is identified that the upper hook and the lower hook in the hook pair are not blocked, the blocking condition of the upper hook and the lower hook in the hook pair is a first type of blocking condition; If it is identified that the upper hook and the lower hook in the hook pair are blocked and the ratio between the blocked area and the total area of the blocked hook does not exceed the preset ratio, then the blocking situation of the upper hook and the lower hook in the hook pair is the second type of blocking situation; Otherwise, the obstruction condition of the upper hook and the lower hook in the hook pair is the third type of obstruction condition.
3. The automatic mounting system control method based on parallel visual servoing according to claim 2 is characterized in that: The method of determining the upper hook circle center and the lower hook circle center of each hook pair by utilizing the shielding conditions of the upper hook and the lower hook in each hook pair comprises: If the occlusion condition of the upper hook and the lower hook in the hook pair is not the third type of occlusion condition, using the three-dimensional point cloud image of the hook pair to determine the center of the upper hook and the center of the lower hook of the hook pair; Otherwise, using the plane contour image of the hook pair and the distance between the pixel points in the plane contour image of the hook pair, the upper hook circle center and the lower hook circle center of the hook pair are determined; Among them, the distance between pixel points in the plane contour image of the hook pair is determined based on the plane contour image of the hook pair when it experiences the first type of occlusion or the second type of occlusion and the actual positions of its corresponding upper hook center and lower hook center.
4. The automatic mounting system control method based on parallel visual servoing according to claim 3 is characterized in that: The method of using the three-dimensional point cloud image of the hook pair to determine the upper hook circle center and the lower hook circle center of the hook pair includes: Downsampling the upper / lower hook point clouds in the three-dimensional point cloud image of the hook pair, and performing normal estimation on the downsampled upper / lower hook point clouds; According to the normal of each point in the down-sampled upper / lower hook point cloud, the down-sampled upper / lower hook point cloud is segmented by clustering to obtain the largest segmented point cloud block; When the shielding condition of the upper hook and the lower hook in the hook pair is the first type of shielding condition: Determine the inner boundary of the outer surface of the upper / lower hook using the maximum point cloud block; Using the inner boundary to fit a plane circle, and using the center of the plane circle as the center of the upper / lower hook; When the shielding condition of the upper hook and the lower hook in the hook pair is the second type of shielding condition: Averaging the normal lines of each point in the largest point cloud block to obtain an average normal line; Based on the average normal line and the planar contour image of the hook pair, a planar Hough circle is determined using a Hough circle detection algorithm; Calculating a three-dimensional projection of the planar Hough circle; Determine the spatial center of the circle corresponding to the center of the planar Hough circle according to the three-dimensional projection of the planar Hough circle; The center of the space is used as the center of the upper / lower hook.
5. The automatic mounting system control method based on parallel visual servoing according to claim 1 is characterized in that: The determining the relative position of the upper platform and the lower platform based on the upper hook circle center and the lower hook circle center of each hook pair includes: The plane where the centers of the upper hook circles of all hook alignments are located is taken as the upper plane, and an upper plane coordinate system is established with the centers of the upper hook circles of all hook alignments as the origin; For each hook pair, an upper hook coordinate system corresponding to the upper hook in the hook pair is established according to the standard coordinate system establishment rules; Based on the coordinates of the center of the upper hook in the hook pair in the camera coordinate system, determine the conversion relationship matrix between the camera coordinate system and the upper hook coordinate system, and based on the coordinates of the center of the upper hook in the hook pair in the upper plane coordinate system, determine the conversion relationship matrix between the upper hook coordinate system and the upper plane coordinate system; Based on the coordinates of the center of the lower hook of the hook pair in the camera coordinate system and the conversion relationship matrix between the camera coordinate system and the upper hook coordinate system, determine the coordinates of the center of the lower hook of the hook pair in the upper hook coordinate system; Based on the coordinates of the center of the lower hook of the hook pair in the upper hook coordinate system and the transformation relationship matrix between the upper hook coordinate system and the upper plane coordinate system, determine the coordinates of the center of the lower hook of the hook pair in the upper plane coordinate system; Fitting the lower plane using the coordinates of the center of the lower hook of all hook pairs in the upper plane coordinate system, and determining the rotation matrix of the lower plane relative to the upper plane; The rotation matrix of the lower plane relative to the upper plane is used to characterize the relative position and posture of the upper platform and the lower platform.
6. The automatic mounting system control method based on parallel visual servoing according to claim 5 is characterized in that: The standard coordinate system establishment rules are: The origin is located at the center of the hook circle on the hook centering, and the X-axis direction, the Y-axis direction and the Z-axis direction are consistent with the X-axis direction, the Y-axis direction and the Z-axis direction of the upper plane.
7. An automatic mounting system control device based on parallel visual servoing, characterized in that: The device comprises: The upper and lower platforms relative posture determination module is used to obtain image information of each target position, and determine the relative posture of the upper platform and the lower platform in the automatic mounting system based on the image information of each target position; A motion control amount determination module, used to determine the motion control amount of each joint in the posture adjustment mechanism of the automatic mounting system based on the relative posture of the upper platform and the lower platform and a pre-stored kinematic model; A motion control module, used to control the joint motion of the posture adjustment mechanism according to the motion control amount, so as to achieve docking of the hooks at each target position in the automatic mounting system; Wherein, each target position is the position of each hook pair in the automatic mounting system; The hook pair consists of an upper hook and a lower hook; Determining the relative position of the upper platform and the lower platform in the automatic mounting system based on the image information includes: Determine, according to the image information, the occlusion conditions of the upper hook and the lower hook in each hook pair; Determine the center of the upper hook and the center of the lower hook of each hook pair by using the shielding conditions of the upper hook and the lower hook in each hook pair; Based on the upper hook circle center and the lower hook circle center of each hook pair, the relative position and posture of the upper platform and the lower platform are determined.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the automatic mounting system control method based on parallel visual servoing as described in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the automatic mounting system control method based on parallel visual servoing as described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Robotic arm target guiding system and method based on visual servo
CN110116410A
Automatic missile hanging vehicle alignment method and system
CN110341978A
Self-loading and unloading mechanism operation image auxiliary control method
CN110532905A
Calibration method, device and equipment
CN111912310A
Sensing equipment and method for automatic grabbing process of tower crane lifting hook
CN113911914A
