A screw locking system and its locking method

Abstract

The present invention relates to the technical field of mechanical assembly, and more specifically, to a screw locking system and a locking method thereof. The system includes a mobile frame, a multi-axis robotic arm, a locking device, a shooting device, a feeding device, and a control and processing device. The locking device includes a locking mechanism and a bird's beak. The locking method includes: S1. Obtaining a training model; S2. The shooting device performs eye-outside-hand calibration; S3. Obtaining the position transformation from the end of the arm to the bird's beak; S4. Planning the moving path of the mobile frame; S5. Using the training model, predicting the hole position of the image to be locked obtained by the shooting device, and obtaining the position movement information of the multi-axis robotic arm; S6. Controlling the movement of the multi-axis robotic arm, and causing the locking device to perform locking operations at the current position; S7. If the mobile frame is at the end designated position, all locking operations are completed; otherwise, the mobile frame is controlled to move to the next position according to the planned path, and then returns to S5. The present invention can improve the efficiency of locking operations, as well as improve the safety of locking operations.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical assembly, and more particularly to a screw locking system and a screw locking method thereof. Background Art

[0002] In the application of screw locking, most of the time, a screw locking device is used to tighten the screws in the preset hole positions of the product to complete the locking operation. In order to improve production efficiency, the existing technology also provides a mobile device with vision to drive the screw locking device to move and control the screw locking device to lock according to the identified hole positions, which can reduce the labor intensity of the operators and avoid the phenomenon of screw missing. However, in the actual locking process, in addition to the preset hole positions for screw locking, the holes set on the product may also have hole positions with other shapes, or have hole positions that are different from the state to be locked. The existing mobile device with vision is difficult to identify different hole positions, so that the screw locking device may tighten the screws in the hole positions other than the preset hole positions, which may cause damage to the screw locking device or the product, which is not conducive to further improving the efficiency of the locking operation. Summary of the Invention

[0003] The purpose of the present invention is to overcome the difficulty in identifying different hole positions during the locking operation, and to provide a screw locking system and a locking method thereof, which can improve the efficiency of the locking operation and can improve the safety of the locking operation.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A screw locking system is provided, comprising a movable frame, wherein a plurality of multi-axis robotic arms are connected to the movable frame via a base, a locking device is connected to the end of each multi-axis robotic arm, and a photographing device and a feeding device are also connected to the movable frame, which are arranged in a one-to-one correspondence with the locking devices; a control processing device is also included, and the movable frame, the multi-axis robotic arms, the locking device, the photographing device, and the feeding device are all communicatively connected to the control processing device; wherein the locking device comprises a locking mechanism and a bird's beak connected to the locking mechanism, the locking end of the locking mechanism extends into the bird's beak; and the discharge end of the feeding device is communicated with the bird's beak.

[0006] Preferably, the feeding device includes a mounting base installed on the movable frame, the mounting base is connected to a feeding track, a transport channel is provided in the mounting base, and the discharge end of the transport channel is connected to a nozzle; it also includes a sliding mechanism, the sliding end of the sliding mechanism is connected to a transport part, and the sliding mechanism can drive the transport part to communicate with the feeding track and the transport channel; the nozzle and the bird's beak are connected through a pipe.

[0007] The present invention also provides a screw locking method, comprising the following steps:

[0008] S1. Obtain multiple sample images of the area to be locked by sampling, and mark all the hole positions in all the sample images to obtain a labeled data set and train it to obtain a training model;

[0009] S2. The camera is calibrated outside the hand by its corresponding locking device;

[0010] S3. Place the mobile frame in the area to be locked and set a fixed point at any position within the area to be locked. Then control the multi-axis robotic arm to drive the locking device to the location of the fixed point multiple times in different positions and calculate the position transformation from the arm end to the bird's beak.

[0011] S4. Move the mobile rack to the initial designated position and plan the moving path of the mobile rack;

[0012] S5. The camera captures an image of the area to be locked at the current position, then uses the trained model from step S1 to predict the hole positions on the image to be locked, performs masking, and obtains a mask image with the specified hole positions. Then, the circular hole information matrix of the specified hole positions in the mask image is obtained. Then, combined with the calibration from step S2, the circular hole information matrix is converted to obtain position movement information of the multi-axis robotic arm.

[0013] S6. Controlling the multi-axis robot arm to move according to the position movement information of the multi-axis robot arm, so that the locking device moves to the top of the designated hole position, the feeding device delivers the screw to the locking device, and the locking device performs the locking operation at the current position;

[0014] S7. If the current position of the mobile rack is the end designated position in the planned path, all locking operations are completed; otherwise, the mobile rack is controlled to move to the next position according to the planned path, and then returns to step S5.

[0015] Furthermore, in step S1, the Yolo algorithm is used to train the annotated data set; the annotated data set includes feature categories of each hole position and position information of each hole position; wherein the feature categories include normal holes, irregular holes, and locked holes; the position information includes the vertex coordinates of the rectangular box corresponding to each hole position in its corresponding image coordinate system.

[0016] Furthermore, in step S3, the position transformation formula from the end of the arm to the beak is:

[0017] Tip T TcpTcp T End End T Base = Tip T Base ;

[0018] Where, Tip T Tcp represents the homogeneous transformation matrix from the beak to the fixed point position, Tcp T End represents the homogeneous transformation matrix from the end of the arm to the beak position, End T Base represents the homogeneous transformation matrix from the base to the arm end position, Tip T Base represents the homogeneous transformation matrix from the base to the fixed cusp position; wherein:

[0019]

[0020] Where T represents the position information of the homogeneous transformation matrix, [x, y, z] represents the position of the origin of the coordinate system of the bird's beak in the coordinate system of the arm end;

[0021] Therefore, we can get:

[0022] Tcp T End End T Base = Tip T Base ;

[0023] By moving the multi-axis robotic arm multiple times, at least three different postures are obtained and a matrix equation system can be obtained, and the position transformation from the arm end to the bird's beak can be obtained by the matrix equation system; wherein the matrix equation system is expressed as:

[0024]

[0025] Where, End T Base_i represents the i-th position of the origin of the coordinate system of the end of the arm in the coordinate system of the base.

[0026] Furthermore, in step S2, the homogeneous transformation matrix from the base to the camera position is obtained by calibration of the eye outside the hand. Cam T Base , and get the internal parameter matrix Intr Cam ;in:

[0027]

[0028] Where, f x 、fy Respectively represent the focal length of the camera in the x direction and the y direction, u x 、u y Represents the pixel density in the x direction and y direction respectively;

[0029] Step S5 includes the following steps:

[0030] S51. The camera obtains an image of the area to be locked, and then uses the training model to predict the hole position of the image to be locked to obtain a rectangular frame corresponding to the feature category of the irregular hole and the normal hole in the image to be locked;

[0031] S52. Output the center points of the corresponding rectangular boxes of all irregularly shaped holes to obtain the circular hole information matrix of the irregularly shaped holes;

[0032] S53. Mask the locked image so that the contents of the rectangular box corresponding to the normal hole are retained and the rest are replaced with a solid background to obtain the mask image and obtain the circular hole information matrix of all normal holes in the mask image;

[0033] S54. Convert the circular hole information matrix of the normal hole to obtain the position movement information of the multi-axis robotic arm.

[0034] Furthermore, in step S52, the circular hole information matrix of the irregular-shaped holes is a matrix of six columns and n rows, where n is the number of irregular-shaped holes identified, and each row is represented as [x, y, z, r, p, y];

[0035] In step S53, the circular hole information matrix of the normal hole is a matrix with three columns and N rows, and each row is in the form of:

[0036] [x px y px 1];

[0037] Where x px is the horizontal coordinate of the normal hole in the image coordinate system to be locked, y px is the vertical coordinate of the normal hole in the coordinate system of the image to be locked; the minimum depth value depth inside the rectangular frame of each normal hole is used as the distance between the center of the normal hole and the optical center of the camera, which is 1;

[0038] In step S54, each row P in the circular hole information matrix of the normal hole is obtained. Hole And the minimum depth value depth inside the rectangular frame of each normal hole, calculate the normal hole coordinates of each normal hole in the camera coordinate system:

[0039]

[0040] P Hole=[x px y px 1] T

[0041] The homogeneous transformation matrix from the camera coordinate system to the normal hole position is:

[0042]

[0043] The homogeneous transformation matrix from the base coordinate system to the normal hole position is:

[0044] Hole T Base = Hole T Cam Cam T Base ;

[0045] Then, the homogeneous transformation matrix of the obtained base coordinate system to the normal hole position is converted into [x base ,y base ,z base ,r base ,p base ,y base ] format of the position vector plus the rotation vector in the coordinate system of the base, which is the movement information of the multi-axis robotic arm.

[0046] Furthermore, step S4 includes the following steps:

[0047] S41. The number of mobile racks is set: n mobile racks are selected for operation, and the area to be locked is divided into S working areas, where: The working area of the first mobile rack is the n working areas closest to the starting position, the working area of the second mobile rack is the n-1 working areas closest to the starting position among the remaining working areas, and so on;

[0048] S42. Setting the number and position distribution of the multi-axis robotic arms: Set the number of the multi-axis robotic arms to k, set the working space diameter of the multi-axis robotic arms to d, and set the width of the area to be locked to D, and the three satisfy the following relationship:

[0049]

[0050] S43. Movement setting of the mobile rack: When the mobile rack is responsible for at least two working areas, the mobile rack first moves to the working area farthest from the starting position at a speed of v1. After completing the locking operation at that position, the mobile rack moves toward the starting position at a speed of v2. The distance is shortened, and the locking operation of the next position is performed, and so on; wherein, v1>v2.

[0051] Furthermore, in step S42, the movement setting of the multi-axis robotic arm is also included: the multi-axis robotic arm locks the designated hole positions in the working area one by one in a serpentine motion trajectory.

[0052] Furthermore, in step S43, after the mobile frame completes all the work areas for which it is responsible, the mobile frame returns to the starting position at a speed of v1.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention includes a screw locking system and a locking method thereof, wherein a shooting device can be used to shoot an image of an area to be locked, and a control processing device is used to extract all hole position information from the image to be locked, and can identify the specified hole position to be locked from all hole position information, and then convert the information of the specified hole position into position movement information of a multi-axis robotic arm, so that the locking device is located above the corresponding hole position, and then the screw is fed to the bird's mouth of the locking device through a feeding device, and the locking mechanism performs screw locking; by identifying the setting of the specified hole position to be locked, it is possible to avoid the locking device from performing screw locking on hole positions other than the specified hole position, which may cause damage to the locking device or the area to be locked; the setting of the mobile frame can be used for the screw locking system to perform screw locking operations on a larger range of areas to be locked, thereby improving the efficiency of the locking operation, wherein, by planning the moving path of the mobile frame, the utilization rate of the multi-axis robotic arm can be improved, the locking operation efficiency can be improved, and the safety of the locking operation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a structural diagram of a screw locking system of the present invention;

[0056] Figure 2 It is a structural schematic diagram of the locking device of the present invention;

[0057] Figure 3 It is a structural schematic diagram of the feeding device of the present invention;

[0058] Figure 4 This is a flow chart of a locking method of a screw locking system according to the present invention;

[0059] Figure 5 A schematic diagram of the working areas of multiple mobile racks in step S41 of the present invention;

[0060] Figure 6 Schematic diagram of the position distribution and movement of the multi-axis robotic arm relative to the mobile frame in step S42 of the present invention.

[0061] The icon markings are explained as follows:

[0062] 1-mobile frame, 11-roller mechanism, 2-base, 3-multi-axis robotic arm, 31-arm end, 4-locking device, 41-bird's beak, 411-feeding port, 42-locking mechanism, 421-rotation mechanism, 422-slide rail and slider mechanism, 43-force sensor, 5-shooting device, 6-feeding device, 61-mounting seat, 611-first slide groove, 612-second slide groove, 62-feeding track, 63-nozzle, 64-sliding mechanism, 641-cylinder, 642-slide key, 65-conveying part. DETAILED DESCRIPTION

[0063] The present invention is further described below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only and are schematic, not actual, representations. They should not be construed as limiting this patent. To better illustrate the embodiments of the present invention, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted from the drawings.

[0064] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the drawings, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0065] Example 1

[0066] like Figures 1 to 3 The figure shows an embodiment of a screw locking system of the present invention, which includes a mobile frame 1, and a plurality of multi-axis robotic arms 3 are connected to the mobile frame 1 through a base 2. The arm end 31 of each multi-axis robotic arm 3 is connected to a locking device 4. The mobile frame 1 is also connected to a shooting device 5 and a feeding device 6 which are arranged in a one-to-one correspondence with the locking device 4; the figure also includes a control processing device, and the mobile frame 1, the multi-axis robotic arms 3, the locking device 4, the shooting device 5, and the feeding device 6 are all communicatively connected to the control processing device; wherein, the locking device 4 includes a locking mechanism 42 and a bird's beak 41 connected to the locking mechanism 42, and the locking end of the locking mechanism 42 extends into the bird's beak 41; the discharge end of the feeding device 6 is connected to the bird's beak 41.

[0067] like Figure 1As shown, a roller mechanism 11 is connected to the bottom of the mobile frame 1 and is in communication with the switch. This configuration enables the movement of the mobile frame 1. The base 2 is fixedly connected to the bottom of the mobile platform of the mobile frame 1. The multi-axis robotic arm 3 drives the locking device 4 to move horizontally. The camera 5 includes a depth camera and a mounting bracket. The depth camera is connected to the mobile platform of the mobile frame 1 via the mounting bracket. The depth camera's range of capture covers the range of motion of the locking device 4.

[0068] like Figure 2 As shown, the locking mechanism 42 includes a rotating mechanism 421 and a slide rail slider mechanism 422 that are communicatively connected to the control processing device. The slide rail slider mechanism 422 is vertically arranged, and the fixed end of the rotating mechanism 421 is connected to the slider end of the slide rail slider mechanism 422. The rotating end of the rotating mechanism 421 extends into the bird's beak 41. The locking device 4 also includes a force sensor 43 connected to the slide rail end of the slide rail slider mechanism 422. The setting of the force sensor 43 can be used to detect the change in the force applied by the locking mechanism 42 to the arm end 31 during the locking process. The slide rail slider mechanism 422 can drive the rotating mechanism 421 to move up and down, and the rotation of the rotating mechanism 421 can drive the screw located in the bird's beak 41 to rotate and lock. It should be noted that the rotating mechanism 421 is a prior art and will not be described in detail here.

[0069] like Figure 1 and Figure 3 As shown, the feeding device 6 includes a mounting base 61 mounted on the mobile platform of the mobile frame 1, the top of the mounting base 61 is connected to a feeding track 62, a conveying channel is provided in the mounting base 61, the discharge end of the conveying channel is connected to a nozzle 63, and the nozzle 63 is connected to the feed port 411 of the bird's beak 41 through a pipe, as shown in FIG. Figure 2 The feeding device 6 further includes a sliding mechanism 64, a sliding end of the sliding mechanism 64 is connected to a transport portion 65, and the sliding mechanism 64 can drive the transport portion 65 to communicate with the feeding track 62 and the transport channel.

[0070] In this embodiment, the sliding mechanism 64 includes a cylinder 641. A first slot 611 is defined at the top of the mounting base 61, and a sliding key 642 slides within the first slot 611. The cylinder base of the cylinder 641 is fixedly connected to the mounting base 61, and the piston rod of the cylinder 641 extends into the mounting base 61 and is connected to the sliding key 642. A second slot 612 is also defined at the top of the mounting base 61, communicating with the feed end of the conveying channel. The conveying portion 65 is fixedly connected to the sliding key 642 and is located within the second slot 612. In this embodiment, the conveying portion 65 includes a slot for conveying screws.

[0071] Specifically, the first chute 611 includes a first sliding section and a second sliding section arranged at an angle to the first sliding section. The first end of the first sliding section serves as the material receiving endpoint, while the second end of the first sliding section communicates with the first end of the second sliding section, and the second end of the second sliding section serves as the material feeding endpoint. The second chute 612 extends in a straight line. The first and second sliding sections are arranged parallel to each other, while the second sliding section bends away from the second chute 612, allowing the sliding key 642 to gradually move away from the second chute 612 as it slides to the material feeding endpoint.

[0072] When the feeding device 6 needs to transport screws, the piston rod of the cylinder 641 retracts. At this time, the sliding key 642 is located at the material receiving end point. The slot of the conveying part 65 and the second chute 612 are vertically aligned, and the slot of the conveying part 65 is connected to the feeding track 62. The screw can be transported from the feeding track 62 to the slot of the conveying part 65. When the piston rod of the cylinder 641 gradually extends, the sliding key 642 moves from the material receiving end point to the material feeding end point in the first chute 611. At the same time, the conveying part 65 also moves toward the conveying channel. During this movement, the slot of the conveying part 65 gradually no longer overlaps with the second chute 612. The screw located at the slot of the conveying part 65 can detach from the slot and fall into the conveying channel. Then, it is transferred to the bird's beak 41 through the nozzle 63 to complete the feeding.

[0073] Example 2

[0074] like Figures 4 and 5 FIG. 1 is a first embodiment of a screw locking method of the present invention. The screw locking method includes the following steps:

[0075] S1. Acquire multiple sample images of the area to be locked by sampling, and annotate all hole positions in all sample images to obtain an annotated dataset. Then, use the Yolo v8 algorithm to train the annotated dataset to obtain a training model.

[0076] S2. The depth camera is calibrated outside the hand by its corresponding lock payment device 4;

[0077] S3. Place the mobile frame 1 in the area to be locked and set a fixed point at any position within the area to be locked. Then, control the multi-axis robot arm 3 to drive the locking device 4 to the fixed point position multiple times in different positions and calculate the position transformation from the arm end 31 to the bird's beak 41.

[0078] S4. Move the mobile rack 1 to the initial designated position and plan the moving path of the mobile rack 1; then execute step S5;

[0079] S5. The depth camera acquires an image of the area to be locked at the current position. The trained model from step S1 is then used to predict the hole positions of the image to be locked. Masking is then performed to obtain a mask image with the specified hole positions. A circular hole information matrix of the specified hole positions in the mask image is then obtained. The circular hole information matrix is then converted, combined with the calibration from step S2, to obtain position movement information of the multi-axis robotic arm 3. Step S6 is then executed.

[0080] S6. According to the position movement information of the multi-axis manipulator 3, the multi-axis manipulator 3 is controlled to move, so that the locking device 4 is moved above the designated hole position, the feeding device 6 transmits the screw to the locking device 4, and the locking device 4 performs the locking operation at the current position;

[0081] S7. If the current position of the mobile rack 1 is the end designated position in the planned path, all locking operations are completed; otherwise, the mobile rack 1 is controlled to move to the next position according to the planned path, and then returns to step S5.

[0082] It should be noted that there is no particular order between steps S1 to S3.

[0083] The depth camera can be used to capture an image of the area to be locked, and the control processing device extracts all hole position information from the image to be locked. The designated hole position to be locked can be identified from all hole position information. The information of the designated hole position is then converted into position movement information of the multi-axis robot 3, so that the locking device 4 is located above the corresponding hole position. The feeding device 6 then feeds screws to the bird's beak 41 of the locking device 4, and the locking mechanism 42 performs the screw locking. By identifying the designated hole position to be locked, the locking device 4 can be prevented from screwing holes other than the designated hole position, which may cause damage to the locking device 4 or the area to be locked. The setting of the mobile frame 1 allows the screw locking system to perform screw locking operations on a larger area to be locked, improving the efficiency of the locking operation. By planning the movement path of the mobile frame 1, the utilization rate of the multi-axis robot 3 can be improved, the efficiency of the locking operation can be improved, and the safety of the locking operation can be improved. The area to be locked can be an area on the container floor.

[0084] Specifically, in step S1, when the area to be locked is a container floor, the sampled image is also an image of the container floor, and the sampled image contains all hole locations of the feature categories. In this embodiment, the sampled image is a sampled two-dimensional image. Furthermore, the annotated dataset includes the feature category of each hole location and the location information of each hole location. Feature categories include normal holes, irregular holes, and locked holes. The location information includes the coordinates of the vertices of the rectangular box corresponding to each hole location in its corresponding image coordinate system, with the image coordinate system having the upper left corner of the sampled image as the origin. It should be noted that the designated hole locations are normal holes, which are used for the hole locations to be locked.

[0085] Specifically, in step S2, the homogeneous transformation matrix from the base 2 to the depth camera position is obtained by calibration of the eye outside the hand Cam T Base , and get the internal parameter matrix Intr Cam ;in,

[0086]

[0087] Where, f x 、f y Respectively represent the focal length of the depth camera in the x direction and y direction, u x 、u y Represents the pixel density in the x direction and y direction respectively.

[0088] Specifically, in step S3, the position transformation formula from the arm end 31 to the bird's beak 41 is:

[0089] Tip T Tcp Tcp T End End T Base = Tip T Base ;

[0090] Where, Tip T Tcp Represents the homogeneous transformation matrix from the beak 41 to the fixed cusp position, Tcp T End The homogeneous transformation matrix representing the position of the arm end 31 to the beak 41, End T Base Represents the homogeneous transformation matrix from the base 2 to the arm end 31 position, Tip T Base Represents the homogeneous transformation matrix from base 2 to the fixed cusp position; where:

[0091]

[0092] Wherein, T represents the position information of the homogeneous transformation matrix, [x, y, z] represents the position of the origin of the coordinate system of the bird's beak 41 in the coordinate system of the arm end 31;

[0093] Therefore, we can get:

[0094] Tcp T End End T Base = Tip T Base ;

[0095] By moving the multi-axis manipulator 3 multiple times, at least three different positions and postures are obtained, and a matrix equation group can be obtained. The position transformation from the arm end 31 to the beak 41 can be obtained through the matrix equation group. The matrix equation group is expressed as:

[0096]

[0097] Where, End T Base_i The ith position of the origin of the coordinate system of the arm tip 31 in the coordinate system of the base 2 is represented.

[0098] Specifically, step S5 includes the following steps:

[0099] S51. The depth camera obtains the image of the area to be locked, and then uses the training model of step S1 to predict the hole position of the image to be locked, and obtains the corresponding rectangular boxes of the feature categories of irregular holes and normal holes in the image to be locked;

[0100] The image to be locked includes a two-dimensional image to be locked and a depth image to be locked aligned with the two-dimensional image to be locked. Alignment means that the two-dimensional image to be locked and the depth image to be locked have the same pixel count, length and width, and the depth image to be locked also contains depth information. When the area to be locked is a container floor, the image to be locked is the image of the container floor to be locked.

[0101] S52. Output the center points of the corresponding rectangular boxes of all irregularly shaped holes to obtain a circular hole information matrix for the irregularly shaped holes. The circular hole information matrix for the irregularly shaped holes is a matrix with six columns and n rows, where n is the number of irregularly shaped holes identified, and each row is represented as [x, y, z, r, p, y]. It should be noted that the obtained circular hole information matrix facilitates operators to quickly obtain the position information of the irregularly shaped holes and perform subsequent processing on the irregularly shaped holes.

[0102] S53. Masking the locked image so that the contents of the rectangular frame corresponding to the normal hole are retained and the rest are replaced with a solid background to obtain a mask image; by adjusting the parameters of the circle center detection algorithm, the normal holes in the mask image can be accurately detected, and the circular hole information matrix of all normal holes in the mask image is obtained;

[0103] The circular hole information matrix of a normal hole is a matrix with three columns and N rows, and each row is in the form of:

[0104] [x px y px 1];

[0105] Where x px is the horizontal coordinate of the normal hole in the image coordinate system to be locked, y px is the vertical coordinate of the normal hole in the coordinate system of the image to be locked; the minimum depth value depth inside the rectangular frame of each normal hole is used as the distance between the center of the normal hole and the optical center of the depth camera, which is 1; since the 2D image to be locked is aligned with the depth image to be locked, the corresponding depth information can be obtained through the pixel position information in the 2D image to be locked;

[0106] S54. The circular hole information matrix of the normal hole is converted to obtain the position movement information of the multi-axis manipulator 3; wherein each row P in the circular hole information matrix of the normal hole is obtained Hole And the minimum depth value depth inside the rectangular frame of each normal hole, calculate the normal hole coordinates of each normal hole in the depth camera coordinate system:

[0107]

[0108]

[0109] P Hole =[x px y px 1] T

[0110] The homogeneous transformation matrix from the depth camera coordinate system to the normal hole position is:

[0111]

[0112] The homogeneous transformation matrix from the coordinate system of base 2 to the normal hole position is:

[0113] Hole T Base = Hole T Cam Cam T Base ;

[0114] Then, the obtained homogeneous transformation matrix of the coordinate system of the base 2 to the normal hole position is converted into [x base ,y base ,z base ,r base ,p base ,y base ] format of the position vector plus the rotation vector in the coordinate system of the base 2, which is the position movement information of the multi-axis robotic arm 3.

[0115] Example 3

[0116] This embodiment is similar to embodiment 2, except that Figures 5 and 6 As shown, in this embodiment, step S4 includes the following steps:

[0117] S41. Set the number of mobile racks 1: Select n mobile racks 1 for operation, and then divide the area to be locked into S working areas, where The working area of the first mobile rack 1 is the n working areas closest to the starting position, the working area of the second mobile rack 1 is the n-1 working areas closest to the starting position in the remaining working areas, and so on; Figure 5 The figure shows the distribution of the areas to be locked when there are three mobile racks 1, where A1, A2, and A3 are the working areas of the first mobile rack 1, B1 and B2 are the working areas of the second mobile rack 1, and C is the working area of the third mobile rack 1.

[0118] S42. Setting the number and position distribution of the multi-axis manipulators 3: Set the number of multi-axis manipulators 3 to k, the diameter of the working space of the multi-axis manipulators 3 to d, and the width of the area to be locked to D. The three satisfy the following relationship:

[0119]

[0120] The multi-axis manipulators 3 are symmetrically distributed on the mobile frame 1, and the multi-axis manipulators 3 adjacent to each other on the same side are kept The distance between the multi-axis robot arm 3 and the mobile frame 1 can maintain its working space. The width is outside the mobile frame 1; the working space range of the multi-axis manipulator 3 located at adjacent positions on the same side has an overlap, specifically, maintaining The width overlap can complete the locking operation more efficiently, such as Figure 6 As shown; it should be noted that Figure 6 In the circular area of the multi-axis robot arm 3 with a diameter of d, the center of the circle is the position of the base 2, and the circular area is the locking operation range of the bird's beak 41.

[0121] In step S42 , the movement setting of the multi-axis robot arm 3 is also included: the multi-axis robot arm 3 locks the designated hole positions in the working area one by one with a serpentine motion trajectory, which can avoid collisions between the multi-axis robot arms 3 .

[0122] S43. Mobile setup of mobile rack 1: When mobile rack 1 is responsible for at least two work areas, mobile rack 1 first moves to the work area farthest from the starting position at speed v1. After completing the locking operation at that position, mobile rack 1 moves toward the starting position at speed v2. The distance is increased, and the locking operation of the next position is carried out, and so on; wherein, v1>v2;

[0123] In step S43 , after the mobile rack 1 completes all the work areas for which it is responsible, the mobile rack 1 returns to the starting position at a speed of v1 .

[0124] It should be noted that there is no specific order between steps S41 to S43. Step S4 also includes setting the dimensions of the mobile rack 1: the length of the mobile rack 1 can be set to be consistent with the width D of the container floor. This setting ensures that the workspace of the symmetrically arranged multi-axis manipulators 3 on the mobile rack 1 covers as many holes as possible in the area, thereby improving the utilization rate of the multi-axis manipulators 3. The width of the mobile rack 1 is consistent with the minimum unit width of the work area allocated by the container floor. This setting can reduce the number of movements of the mobile rack 1.

[0125] Specifically, in this embodiment, the control processing device includes a host and a switch connected to the host. The multi-axis robot arm 3 and the depth camera are both connected to the switch. The host can simultaneously obtain data information from the depth camera and the multi-axis robot arm 3 through the switch and perform data information processing and analysis. In step S6, based on the position movement information of the multi-axis robot arm 3 obtained in step S5, the host transmits a signal to the switch, and then transmits a signal to the multi-axis robot arm 3 through the switch to control the movement of the multi-axis robot arm 3, so that the bird's beak 41 of the locking device 4 located at the end 31 of the arm moves to the specified hole position, that is, directly above the normal hole. The feeding device 6 delivers the screw to the bird's beak 41 and performs the locking operation.

[0126] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for locking a screw locking system, characterized in that: The screw locking system comprises a mobile frame (1), wherein a plurality of multi-axis robotic arms (3) are connected to the mobile frame (1) via a base (2), and the arm end (31) of each multi-axis robotic arm (3) is connected to a locking device (4), and the mobile frame (1) is also connected to a shooting device (5) and a feeding device (6) arranged in a one-to-one correspondence with the locking device (4); and further comprises a control processing device, wherein the mobile frame (1), the multi-axis robotic arms (3), the locking device (4), the shooting device (5), and the feeding device (6) are all communicatively connected to the control processing device; wherein the locking device (4) comprises a locking mechanism (42) and a bird's beak (41) connected to the locking mechanism (42), and the locking end of the locking mechanism (42) extends into the bird's beak (41); the discharge end of the feeding device (6) is communicated with the bird's beak (41); and the locking method comprises the following steps: S1. Obtain multiple sample images of the area to be locked by sampling, and mark all the hole positions in all the sample images to obtain a labeled data set and train it to obtain a training model; S2. The photographing device (5) performs calibration of the eye outside the hand through the corresponding locking device (4); S3. The mobile frame (1) is placed in the area to be locked, and a fixed sharp point is set at any position in the area to be locked, and then the multi-axis robot arm (3) is controlled to drive the locking device (4) to reach the position of the fixed sharp point multiple times in different postures, and the position transformation from the end of the arm (31) to the beak (41) is calculated; S4. Moving the mobile rack (1) to an initial designated position and planning a moving path of the mobile rack (1); S5. The photographing device (5) obtains the image to be locked at the current position of the area to be locked, and then uses the training model of step S1 to predict the hole position of the image to be locked, and performs mask processing to obtain a mask image with the specified hole position, and then obtains the circular hole information matrix of the specified hole position in the mask image; then, combined with the calibration of step S2, the circular hole information matrix is converted to obtain the position movement information of the multi-axis robot arm (3); S6. Controlling the multi-axis robotic arm (3) to move according to the position movement information of the multi-axis robotic arm (3), so that the locking device (4) moves to the top of the designated hole position, the feeding device (6) delivers the screw to the locking device (4), and the locking device (4) performs a locking operation at the current position; S7. If the current position of the mobile rack (1) is the end designated position in the planned path, all locking operations are completed; otherwise, the mobile rack (1) is controlled to move to the next position according to the planned path, and then returns to step S5.

2. The method for locking a screw locking system according to claim 1, wherein: In step S1, the Yolo algorithm is used to train the annotated dataset; the annotated dataset includes a feature category for each hole position and position information of each hole position; wherein the feature category includes normal holes, irregular holes, and locked holes; the position information includes the coordinates of the vertices of the rectangular box corresponding to each hole position in its corresponding image coordinate system.

3. The screw locking method according to claim 1, wherein: In step S3, the position transformation formula from the arm end (31) to the bird's beak (41) is: ； Where, represents the homogeneous transformation matrix from the beak (41) to the fixed cusp position, The homogeneous transformation matrix representing the position of the arm end (31) to the beak (41) is The homogeneous transformation matrix representing the position of the base (2) to the arm end (31) is: Represents the homogeneous transformation matrix from the base (2) to the fixed cusp position; wherein: Where, T Represents the position information of the homogeneous transformation matrix, [ x , y , z ] represents the position of the origin of the coordinate system of the bird's beak (41) in the coordinate system of the arm end (31); Therefore, we can get: ； By moving the multi-axis robotic arm (3) multiple times, at least three different postures are obtained and a matrix equation group can be obtained, and the position transformation from the arm end (31) to the bird's beak (41) can be obtained through the matrix equation group; wherein the matrix equation group is expressed as: Where, Indicates the origin of the coordinate system of the arm end (31) in the coordinate system of the base (2) i locations.

4. The method for locking a screw locking system according to claim 1, wherein: In step S2, the homogeneous transformation matrix of the position of the base (2) to the shooting device (5) is obtained by calibration of the eye outside the hand. , and obtain the internal parameter matrix ;in: ； Where, f x 、 f y Respectively represent the shooting device (5) in x direction, y focal length in the direction, u x 、 u y Respectively expressed in x direction, y Pixel density in direction; Step S5 includes the following steps: S51. The photographing device (5) obtains an image of the area to be locked, and then uses the training model to predict the hole positions of the image to be locked, and obtains corresponding rectangular frames of the feature categories of irregular holes and normal holes in the image to be locked; S52. Output the center points of the corresponding rectangular boxes of all irregularly shaped holes to obtain the circular hole information matrix of the irregularly shaped holes; S53. Mask the locked image so that the contents of the rectangular box corresponding to the normal hole are retained and the rest are replaced with a solid background to obtain the mask image and obtain the circular hole information matrix of all normal holes in the mask image; S54. Convert the circular hole information matrix of the normal hole to obtain the position movement information of the multi-axis robot arm (3).

5. The method for locking a screw locking system according to claim 4, wherein: In step S52, the circular hole information matrix of the irregular hole is six columns n The matrix of rows, n is the number of identified special-shaped holes, where each row represents [ x , y , z , r , p , y ]; In step S53, the circular hole information matrix of the normal hole is a matrix with three columns and N rows, and each row is in the form of: ； Where, is the horizontal coordinate of the normal hole in the image coordinate system to be locked, The vertical coordinate of the normal hole in the image coordinate system to be locked; the minimum depth value inside the rectangular frame of each normal hole depth As the distance between the center of the normal hole and the optical center of the photographing device (5), it is 1; In step S54, each row of the circular hole information matrix of the normal hole is obtained And the minimum depth value inside the rectangular box of each normal hole depth , calculate the normal hole coordinates of each normal hole in the coordinate system of the shooting device (5): The homogeneous transformation matrix from the coordinate system of the shooting device (5) to the normal hole position is: ； The homogeneous transformation matrix from the coordinate system of the base (2) to the normal hole position is: ； Then, the obtained homogeneous transformation matrix of the coordinate system of the base (2) to the normal hole position is converted into The format of the position vector plus the rotation vector in the coordinate system of the base (2) is the movement information of the multi-axis robotic arm (3).

6. The method for locking a screw locking system according to claim 1, wherein: Step S4 includes the following steps: S41. Setting the number of the mobile racks (1): Select n The mobile frame (1) is operated, and then the area to be locked is divided into S working areas, wherein: ; and the working area of the first mobile frame (1) is the one closest to the starting position n The working area of the second mobile frame (1) is the one closest to the starting position among the remaining working areas. n -1 working area, and so on; S42. The number and position distribution of the multi-axis robotic arms (3): Set the number of the multi-axis robotic arms (3) to k , set the working space diameter of the multi-axis robot arm (3) to d , set the width of the area to be locked to D , the three satisfy the following relationship: ； S43. Mobile setting of the mobile rack (1): When the mobile rack (1) is responsible for at least two working areas, the mobile rack (1) first The moving frame (1) moves to the working area farthest from the starting position at a speed of Move towards the starting position at a speed of The distance is then used to lock the next position, and so on; > .

7. The screw locking method according to claim 6, wherein: In step S42, the movement setting of the multi-axis robot arm (3) is also included: the multi-axis robot arm (3) performs locking operations on the designated hole positions in the working area one by one with a serpentine motion trajectory.

8. The method for locking a screw locking system according to claim 6, wherein: In step S43, when the mobile rack (1) completes all the work areas it is responsible for, the mobile rack (1) is Return to the starting position at a speed of 9. The method for locking a screw locking system according to claim 3, wherein: The feeding device (6) includes a mounting base (61) mounted on the mobile frame (1), a feeding track (62) connected to the mounting base (61), a transport channel provided in the mounting base (61), and a nozzle (63) connected to the discharge end of the transport channel; and further includes a sliding mechanism (64), a sliding end of the sliding mechanism (64) connected to a transport portion (65), and the sliding mechanism (64) can drive the transport portion (65) to communicate with the feeding track (62) and the transport channel; the nozzle (63) is connected to the bird's beak (41) through a pipe.

Images