A container automatic pick-up and release system based on machine vision and operation method
Through the automatic container grabbing and releasing system based on machine vision, the container lock hole image is obtained using multi-camera and pinhole cameras, and the pose adjustment parameters of the spreader are calculated, which solves the problems of high difficulty in traditional lifting operation and low efficiency in automatic grabbing and releasing operations. , realizes efficient and accurate automatic box grabbing and placing.
Patent Information
- Application Number
- CN202111210454.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-18
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-10-18
AI Technical Summary
The traditional container lifting operation is difficult. The existing automatic box grabbing and placing solution cannot obtain the keyhole image when the spreader grabs the container movement, and the system overhead is high and the operation efficiency cannot be guaranteed.
Adopt a container automatic grabbing and placement system based on machine vision, including a gantry, a spreader, a ground identification unit, an image acquisition module, an image processing module and a control module. The container keyhole images are obtained in real time through multiple industrial cameras and pinhole cameras, and the container keyhole images are obtained in real time. The N-point perspective algorithm calculates the pose adjustment parameters of the spreader to realize closed-loop control.
Real-time, accurate and robust automatic box grabbing and placing operations, improving the real-time and reliability of the system, and being able to stably complete box grabbing and placing operations at different container placements.
Smart Images

Figure CN113879984B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of port crane hoisting, and in particular to an automatic grabbing and releasing system and an operating method thereof based on machine vision. Background Art
[0002] Traditional container lifting operations are difficult to operate and require certain skills and experience from operators. With the improvement of port automation, more and more ports are beginning to deploy fully automatic operations, and automatic container grabbing and releasing is an important technology.
[0003] The existing solution for automatically grabbing and releasing boxes is to use a sling with a lock in combination with a camera and an acceleration sensor to obtain the current sling position of the key hole on the container below the sling and guide the sling to approach the key hole position. However, this method is only applicable to the situation where the sling grabs nothing. When the sling grabs the container and moves, it will block the camera and the corresponding key hole image cannot be obtained. Some solutions use 3D laser scanning to guide the grabbing and releasing of boxes by analyzing point cloud data, which results in high system overhead and cannot guarantee operating efficiency.
[0004] Therefore, the present invention proposes an automatic pick-and-place system and operation method based on machine vision. Summary of the invention
[0005] In view of this, the present invention proposes a machine vision-based automatic grasping and releasing system and operation method for a device with good real-time performance, high accuracy and strong robustness.
[0006] The technical solution of the present invention is achieved in this way:
[0007] In one aspect, the present invention provides a machine vision-based automatic container grabbing and placing system, comprising a gantry, a spreader, a ground identification unit (100), an image acquisition module (200), an image processing module (300) and a control module (400); the gantry is provided with a spreader;
[0008] A ground marking unit (100) comprises a plurality of ground markings, each of which is arranged in a linear array on the ground of the yard and fixed on the ground, and adjacent ground markings are arranged at intervals; each ground marking corresponds to at least an end point position of a container placement position;
[0009] The image acquisition module (200) comprises a first industrial camera (21), a second industrial camera (22) and four pinhole cameras, wherein the first industrial camera (21) and the industrial camera are centrally symmetrically arranged at the end positions of the two long sides of the spreader in the extending direction, and the first industrial camera (21) and the second industrial camera (22) respectively capture images of ground markings at both ends of the container placement position; each pinhole camera is respectively arranged at the locks on the two short sides of the spreader, and each pinhole camera captures an image of a lock hole or a corner of a container below the spreader;
[0010] The image processing module (300) receives images of the ground mark taken by the first industrial camera (21) and the second industrial camera (22), and keyhole images or box corner images of the container under the spreader taken by the pinhole cameras, extracts geometric features from the images of the ground mark, the keyhole images or the box corner images, obtains the position of the spreader in space and the position of the spreader relative to the ground mark or the container in the yard based on an N-point perspective algorithm, and obtains an angle offset and a position offset that need to be adjusted for the spreader, and the image processing module (300) sends the angle offset and the position offset to the control module (400);
[0011] A control module (400) changes the position of the spreader according to the angle offset and the position offset sent by the image processing module (300), or sends a control signal for grabbing and releasing a container, so as to adjust the position of the spreader and the container or to grab and release a container;
[0012] When it is necessary to execute a box grabbing and releasing action, the spreader moves to the target container placement position, the image processing module (300) obtains images of the ground mark taken by the first industrial camera (21) and the second industrial camera (22), and based on the calibrated camera internal parameters and distortion coefficients, uses an N-point perspective algorithm to solve the external parameters of the ground mark in the camera, and then based on the calibrated relative posture between the first industrial camera, the second industrial camera and the spreader, converts the current posture of the spreader in the world coordinate system to obtain the angle offset and position offset that need to be adjusted for the spreader; the control module (400) makes the spreader located directly above the container or the container placement position corresponding to the ground mark according to the angle offset and position offset sent by the image processing module (300);
[0013] The image processing module (300) also selectively obtains container corner images through each pinhole camera to extract and calculate the container corners, so that the spreader or the container on the spreader is further aligned with the ground mark or the four corners of the placed container and then lowered. Based on the above technical solution, preferably, the ground marks are respectively distributed in a linear array along the horizontal movement direction of the gantry trolley and the horizontal movement direction of the gantry trolley, and at least two different ground marks can determine a container placement position.
[0014] Further preferably, the established world coordinate system takes the horizontal movement direction of the gantry trolley as the X-axis, the horizontal movement direction of the gantry trolley as the Y-axis, the vertical direction as the Z-axis, and the morphological center of each ground marker as the zero point, to establish a world coordinate system based on each ground marker.
[0015] Further preferably, the ground marking is a group of centrosymmetric patterns, each pattern corresponding to an end point of a container placement position.
[0016] More preferably, the pattern of the ground marking is rectangular, L-shaped, fan-shaped or circular.
[0017] Further preferably, it also includes a third industrial camera (23) and a fourth industrial camera (24), and the third industrial camera (23) and the fourth industrial camera (24) are respectively arranged at the center of the two long sides of the sling, and the third industrial camera (23) and the fourth industrial camera (24) selectively aim at the ground markings spanning between the two ends of the container and obtain images of the ground markings there.
[0018] In addition, the present invention also provides an operation method of a container automatic pick-and-place system based on machine vision, which is characterized by comprising the following steps:
[0019] S100: The aforementioned automatic container pick-and-place system based on machine vision is configured on the gantry;
[0020] S200: performing a single camera calibration on each industrial camera and pinhole camera to obtain an intrinsic parameter matrix and distortion parameters of each industrial camera and pinhole camera;
[0021] S300: Calibrate each industrial camera: Use an actual lifting box to assist in the calibration, establish a world coordinate system based on each ground mark, and unify the coordinates of the first industrial camera (21) and the second industrial camera (22); manually lift the container to the center of a container placement position, so that the four end points of the container close to the ground are at the same distance from the center of the ground marks of the corresponding end points; respectively photograph the corresponding ground marks and measure the coordinate values of the first industrial camera (21) and the second industrial camera (22) in the world coordinate system of the ground marks photographed by them, then unlock the lifting device and raise it to a certain height, and when the lifting device is still, the first industrial camera (21) and the second industrial camera (22) continue to photograph the images of the corresponding ground marks, and the image processing module (300) uses the distortion parameters obtained in the previous step, Correct the distortion of the first industrial camera (21) and the second industrial camera (22), then calculate the rotation matrix and translation matrix of the first industrial camera (21) and the second industrial camera (22) according to the camera internal parameters by applying the N-point perspective algorithm, that is, the posture of the first industrial camera (21) and the second industrial camera (22); then obtain the two-dimensional rotation angle and axial position of the first industrial camera (21) or the second industrial camera (22) in three axes in space according to the rotation matrix, and save the initial posture of the first industrial camera (21) and the second industrial camera (22); repeat the above steps at different hoist heights; fit the initial posture of the first industrial camera (21) and the second industrial camera (22) when the hoist is at different heights, and eliminate the influence of the installation deviation of the first industrial camera (21) and the second industrial camera (22);
[0022] S400: for the relative positions of the four pinhole cameras, using the obtained distortion parameters of each pinhole camera, when the sling is unlocked and raised, the images of each pinhole camera are acquired at the height, the feature points of the keyhole or the box corner of the keyhole image or the box corner image are extracted, the pixel coordinate values at the height are saved, and the above steps are repeated at different sling heights; the corresponding relationship between the pixel coordinates of the four pinhole cameras and the feature points of the four keyholes or the box corners is fitted;
[0023] S500: When there is no container placed at the corresponding container placement position on the ground and a container needs to be placed: the spreader drives the container to move to the container placement position, the image processing module (300) processes the images of the ground mark taken by the first industrial camera (21) and the second industrial camera (22), respectively calculates the current rotation matrix and translation matrix of the first industrial camera (21) and the second industrial camera (22) in the world coordinate system, converts the rotation matrix and translation matrix into two-dimensional rotation angle offsets and axial offsets of three axes in the space relative to the initial posture, and sends the two-dimensional rotation angle offsets and axial offsets of the three axes to the control module (400), respectively. The control module (400) first adjusts the posture of the spreader and the container according to the rotation angle offset and the axial offset. During the spreader lowering process, the image acquisition module (200) and the image processing module (300) detect in real time that the current spreader posture is kept flush with the relative position of the container placement position determined by the ground mark, thereby realizing the container placement action;
[0024] S600: When there is a container placed at the corresponding container placement position on the ground and the containers need to be stacked and placed further: the sling brings the container to the top of the target container placement position, the image processing module (300) processes the current rotation matrix and translation matrix of the first industrial camera (21) and the second industrial camera (22) in the world coordinate system, obtains the two-dimensional rotation angle offset and the axial offset of the three axes in the space, and sends the two-dimensional rotation angle offset and the axial offset of the three axes to the control module (400) respectively. The control module (400) first adjusts the posture of the sling and the container according to the rotation angle offset and the axial offset to ensure that the edge of the container lifted by the sling is parallel to the edge of the container placed below. During the process of lowering the sling, the image processing module (300) also adjusts the posture of the sling according to the images of the lock hole or the images of the box corners obtained by the four pinhole cameras. The control module (400) makes corresponding adjustments to ensure that the container lifted by the sling is aligned with the four box corners of the placed container, and completes the container placement action of the stacked container;
[0025] S700: When a container is placed at a corresponding container placement position on the ground and an empty sling is required to grab it: the empty sling reaches the top of the target container and is lowered, the image processing module (300) processes the images of the ground mark obtained by the first industrial camera (21) and the second industrial camera (22), obtains three-axis two-dimensional rotation angle offsets and position offsets, and sends the angle offsets and axial offsets to the control module (400). The control module (400) adjusts the posture of the sling and lowers it. The image processing module (300) also adjusts the posture of the sling according to the images of the lock hole or the images of the box corners obtained by the four pinhole cameras, so that when the sling falls vertically, the lock on the sling can directly fall into the lock hole of the container to realize the container grabbing action.
[0026] Further preferably, the individual camera calibration for each industrial camera and pinhole camera is performed by presetting a chessboard with a known size, using each industrial camera to respectively shoot chessboard images in different postures, reading the corner points of the chessboard images, and obtaining the camera intrinsic parameter matrix and distortion parameters through the Zhang Zhengyou calibration method.
[0027] Further preferably, the step S300 also includes a step of unifying the coordinates of the third industrial camera (23) and the fourth industrial camera (24) with the first industrial camera (21) and the second industrial camera (22): when the container has a ground mark across it, the third industrial camera (23) and the fourth industrial camera (24) are respectively aligned with the image of the ground mark across which the container is across, and the coordinate values of the third industrial camera (23) and the fourth industrial camera (24) in the world coordinate system of the corresponding ground mark are obtained; after the sling is unlocked and raised, the above steps are repeated to correct the distortion of the third industrial camera (23) and the fourth industrial camera (24), and the two-dimensional rotation angles and axial translation amounts of the third industrial camera (23) or the fourth industrial camera (24) in three axes in space are obtained, the initial postures of the third industrial camera (23) and the fourth industrial camera (24) are saved, and the influence of the installation deviation of the third industrial camera (23) and the fourth industrial camera (24) is eliminated.
[0028] More preferably, in step S500 or S700, when there is a ground mark across the container placement position, the image processing module (300) also simultaneously processes the images of the ground mark taken by the third industrial camera (23) and the fourth industrial camera (24), and respectively calculates the postures of the third industrial camera (23) and the fourth industrial camera (24) in the world coordinate system established by the corresponding ground mark, so that during the lowering process of the sling, the image acquisition module (200) and the image processing module (300) can detect in real time the relative position of the current container posture relative to the container placement position determined by the ground mark, and the subsequent process is the same as step S500 or S700.
[0029] The present invention has the following beneficial effects compared with the prior art:
[0030] (1) The present invention is based on machine vision to achieve framing for fixed ground marks, obtain the current position of the spreader in real time, and use the N-point perspective algorithm to calculate the adjustment parameters of the spreader, so that the control module can achieve closed-loop control and accurately realize the automatic box grabbing and releasing action with good real-time performance;
[0031] (2) Multiple industrial cameras are used to extract geometric features of different ground signs, and multiple pinhole cameras are used to shoot and align the edges of containers. Cameras of the same type can verify each other, and the system has good redundancy and strong robustness.
[0032] (3) The ground marker is fixed on the ground with a specific pattern, which can indicate the end point of the container close to the ground, and can also assist in locating the outline of the container and defining the initial pose and real-time pose of each camera in the world coordinate system;
[0033] (4) An N-point perspective algorithm is used to solve the rotation matrix and translation matrix of the corresponding camera to achieve the mapping of the 3D object to the 2D pixel coordinate system. Combined with the pre-performed camera calibration, the corresponding camera pose can be quickly calculated;
[0034] (5) According to the different usage conditions of the container placement position, the system can stably realize the action of grabbing or releasing the container, thereby improving the reliability of grabbing and releasing the container. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0036] Figure 1 It is a top view of the combined arrangement state of a ground identification unit, an image acquisition module, an image processing module and a control module of a machine vision-based automatic grasping and releasing system and an operation method of the present invention;
[0037] Figure 2 A top view of the layout state of a ground marking unit of a machine vision-based automatic grab and place system and operation method of the present invention;
[0038] Figure 3 It is a top view of the arrangement state of the image acquisition module of the automatic grasping and placing system and operation method of the device based on machine vision of the present invention on the spreader;
[0039] Figure 4 The present invention is an operation flow chart of an automatic pick-and-place system and operation method based on machine vision. DETAILED DESCRIPTION
[0040] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] The technical solution of the present invention is achieved in this way:
[0042] On the one hand, if Figure 1 As shown in FIG. 3 , the present invention provides a machine vision-based automatic container grabbing and placing system, including a gantry and a spreader, wherein the gantry is arranged in a yard; the spreader is arranged on the gantry, and a pair of long sides and a pair of short sides are symmetrically arranged on the spreader, the short sides are arranged in parallel between the two long sides, and locks are arranged on the two short sides respectively, and the gantry includes a large vehicle for driving the gantry to move along the ground and a small vehicle for driving the spreader to move along the span direction of the gantry; and further includes a ground identification unit 100, an image acquisition module 200, an image processing module 300 and a control module 400;
[0043] The ground marking unit 100 includes a plurality of ground markings, each of which is arranged in a linear array on the ground of the yard and fixed on the ground, and adjacent ground markings are arranged at intervals; each ground marking corresponds to at least an end point position of a container placement position.
[0044] The image acquisition module 200 includes a first industrial camera 21, a second industrial camera 22 and four pinhole cameras. The first industrial camera 21 and the industrial camera are symmetrically arranged at the end positions of the two long sides. The first industrial camera 21 and the second industrial camera 22 respectively take images of the ground marks at both ends of the container placement position; each pinhole camera is respectively arranged at the locks on the two short sides of the spreader, and each pinhole camera takes images of the lock holes or box corners of the container under the spreader. The pinhole cameras are shown as reference numerals 25, 26, 27 and 28.
[0045] The image processing module 300 receives the images of the ground markings taken by the first industrial camera 21 and the second industrial camera 22, as well as the keyhole images or the box corner images of the container under the spreader taken by each pinhole camera, extracts the geometric features in the images of the ground markings, the keyhole images or the box corner images, obtains the position of the spreader in space and the position of the spreader relative to the ground markings or the containers in the yard based on the N-point perspective algorithm, and obtains the angle offset and position offset that need to be adjusted for the spreader. The image processing module 300 sends the angle offset and the position offset to the control module 400.
[0046] In the field of machine vision, the N-point perspective algorithm is a common method for solving the camera pose. OpenCV already has a function SlovePnP for solving the algorithm. Usually, the position of four known points can be obtained to solve a relatively accurate camera pose. For specific function contents, please refer to GitHub or the official OpenCV documentation, which will not be repeated here. The images of the ground mark taken by the first industrial camera 21 and the second industrial camera 22 can easily obtain the coordinates of the four known points corresponding to the ground mark.
[0047] The control module 400 changes the position of the spreader according to the angle offset and position offset sent by the image processing module 300, or sends a control signal for grabbing and releasing a container, so as to adjust the position of the spreader and the container or to grab and release a container.
[0048] When the present invention is working: when it is necessary to execute the action of grabbing and releasing the container, the spreader moves to the target container placement position, the image processing module 300 obtains the image of the ground mark taken by the first industrial camera 21 and the second industrial camera 22, and based on the calibrated camera internal parameters and distortion coefficients, the N-point perspective algorithm is applied to solve the external parameters of the ground mark in the camera, and then based on the calibrated relative posture between the first industrial camera, the second industrial camera and the spreader, the current posture of the spreader in the world coordinate system is converted, that is, the current posture in the coordinate system corresponding to the ground mark is established to obtain the angle offset and position offset that need to be adjusted for the spreader; the control module 400 makes the spreader located directly above the container placement position corresponding to the container or the ground mark according to the angle offset and position offset sent by the image processing module 300; the image processing module 300 also selectively obtains the container corner image through each pinhole camera to extract and calculate the corner, so that the spreader or the container on the spreader is further aligned with the ground mark or the four corners of the placed container and then lowered.
[0049] like Figure 2 As shown, in order to better define the first container placement position 1 or the second container placement position 2 so that the movement direction of the container coincides with the ground mark, the ground marks of the ground mark unit 100 can be distributed in a linear array along the horizontal movement direction of the gantry's trolley and the horizontal movement direction of the gantry's trolley. Figure 2 The row of one group of ground markings 3 and 4 and the row of another group of ground markings 5, 6 and 7 form the outline of a number of first container placement positions 1; and the row of one group of ground markings 5, 6 and 7 and the row of ground marking 8 form a number of second container placement positions 2. The ground markings can be made of reflective materials. The ground markings can be composed of a group of centrally symmetrical patterns, each pattern corresponding to the end point of the container placement position. The pattern of the ground marking is rectangular, L-shaped, fan-shaped or circular. Figure 2 For example, the ground marking shown in the figure is a cross-shaped ground marking composed of four centrosymmetrical L-shaped patterns, and the turning point of each L-shaped pattern corresponds to the end point of a container placement position.
[0050] Each ground mark corresponds to at least one endpoint of a container placement location; this has two meanings: four adjacent containers on the same plane share different patterns of a ground mark; and stacked containers of the same specifications share the same pattern in a ground mark. At least two different ground marks can determine a container placement location.
[0051] Likewise Figure 2 As shown, due to the different sizes of containers, the first container placement position 1 or the second container placement position 2 can correspond to containers of different specifications, such as a 20-foot container or a 40-foot container. In order to better determine the outline of the second container placement position 2, a group of ground marks are set across the container body there, and the group of ground marks can indicate the midpoint of the outline of the container in the length direction, such as the column where the ground mark 9 is located in the figure.
[0052] In the above system, the corresponding world coordinate system is established based on each ground mark, with the horizontal movement direction of the gantry trolley as the X-axis, the horizontal movement direction of the gantry trolley as the Y-axis, the vertical direction as the Z-axis, and the morphological center of each ground mark as the zero point to establish a world coordinate system based on each ground mark.
[0053] In order to better observe the relative relationship between the length direction of the container and the ground mark, the system of the present invention also includes a third industrial camera 23 and a fourth industrial camera 24, which are respectively set at the center of the two long sides of the sling, and the third industrial camera 23 and the fourth industrial camera 24 selectively aim at the ground mark across the two ends of the container, and obtain the image of the ground mark there. In order to improve the imaging quality of each industrial camera, an infrared light source 29 can be set at each industrial camera to better obtain the image of the ground mark for the industrial camera.
[0054] like Figure 4 As shown, on the other hand, the present invention also provides an operation method of a container automatic pick-and-place system based on machine vision, comprising the following steps:
[0055] S100: The aforementioned automatic container pick-and-place system based on machine vision is configured on the gantry;
[0056] S200: performing a single camera calibration on each industrial camera and pinhole camera to obtain an intrinsic parameter matrix and distortion parameters of each industrial camera and pinhole camera;
[0057] S300: Calibrate each industrial camera: Use the actual hoisting box to assist in the calibration, establish the world coordinate system based on each ground mark, and unify the coordinates of the first industrial camera 21 and the second industrial camera 22; manually lift the container to the center of a container placement position, so that the four end points of the container close to the ground are at the same distance from the center of the ground marks of the corresponding end points; respectively photograph the corresponding ground marks and measure the coordinate values of the first industrial camera 21 and the second industrial camera 22 in the world coordinate system of the ground marks photographed by them, then unlock the hoist and raise it to a certain height, and when the hoist is still, the first industrial camera 21 and the second industrial camera 22 continue to photograph the images of the corresponding ground marks, and the image processing module 300 obtains the image of the ground mark according to the image processing module 300. Take the distortion parameters, correct the distortion of the first industrial camera 21 and the second industrial camera 22, and then use the N-point perspective algorithm to calculate the rotation matrix and translation matrix of the first industrial camera 21 and the second industrial camera 22 according to the camera internal parameters, that is, the position and posture of the first industrial camera 21 and the second industrial camera 22; then obtain the two-dimensional rotation angle and axial position of the first industrial camera 21 or the second industrial camera 22 in three axes in space according to the rotation matrix, and save the initial position and posture of the first industrial camera 21 and the second industrial camera 22; repeat the above steps at different hoist heights; fit the initial position and posture of the first industrial camera 21 and the second industrial camera 22 when the hoist is at different heights, and eliminate the influence of the installation deviation of the first industrial camera 21 and the second industrial camera 22;
[0058] S400: for the relative positions of the four pinhole cameras, by using the distortion parameters of each pinhole camera that have been obtained, when the spreader is unlocked and raised, the images of each pinhole camera are obtained at the height, the feature points of the keyhole image or the box corner image are extracted, the pixel coordinate values at the height are saved, and the above steps are repeated at different spreader heights; the corresponding relationship between the pixel coordinates of the four pinhole cameras and the feature points of the four keyholes or box corners is fitted; the subsequent steps S500-S700 correspond to different yard conditions respectively, and different container grabbing and releasing action strategies are executed;
[0059] S500: When there is no container at the corresponding container placement position on the ground and a container needs to be placed: the spreader drives the container to move to the container placement position, the image processing module 300 processes the images of the ground mark taken by the first industrial camera 21 and the second industrial camera 22, respectively calculates the current rotation matrix and translation matrix of the first industrial camera 21 and the second industrial camera 22 in the world coordinate system, converts the rotation matrix and the translation matrix into a two-dimensional rotation angle offset and an axial offset of three axes in the space relative to the initial posture, and sends the two-dimensional rotation angle offset and the axial offset of the three axes to the control module 400 respectively. The control module 400 first adjusts the posture of the spreader and the container according to the rotation angle offset and the axial offset. During the lowering process of the spreader, the image acquisition module 200 and the image processing module 300 detect in real time that the posture of the current spreader is kept flush with the relative position of the container placement position determined by the ground mark, thereby realizing the container placement action;
[0060] S600: When there are containers placed at the corresponding container placement position on the ground and the containers need to be stacked and placed further: the sling brings the container to the top of the target container placement position, the image processing module 300 processes the current rotation matrix and translation matrix of the first industrial camera 21 and the second industrial camera 22 in the world coordinate system, obtains the two-dimensional rotation angle offset and the axial offset of the three axes in the space, and sends the two-dimensional rotation angle offset and the axial offset of the three axes to the control module 400 respectively. The control module 400 first adjusts the posture of the sling and the container according to the rotation angle offset and the axial offset to ensure that the edge of the container lifted by the sling is parallel to the edge of the container placed below. During the lowering process of the sling, the image processing module 300 also adjusts the posture of the sling according to the images of the keyhole or the images of the box corners obtained by the four pinhole cameras. The control module 400 makes corresponding adjustments to ensure that the container lifted by the sling is aligned with the four box corners of the placed container, and the container placement action of the stacked container is completed;
[0061] S700: When there is a container placed at the corresponding container placement position on the ground and an empty sling is needed to grab it: the empty sling reaches above the target container and is lowered, the image processing module 300 processes the images of the ground marks obtained by the first industrial camera 21 and the second industrial camera 22, and obtains the two-dimensional rotation angle offset and position offset of the three axes, and sends the angle offset and the axial offset to the control module 400, the control module 400 adjusts the posture of the sling and lowers it, the image processing module 300 also adjusts the posture of the sling according to the images of the keyhole or the images of the box corners obtained by the four pinhole cameras, so that when the sling falls vertically, the lock on the sling can directly fall into the keyhole of the container to realize the container grabbing action.
[0062] In the above step S200, a single camera calibration is performed on each industrial camera and pinhole camera, which is to preset a chessboard with known size, use each industrial camera to shoot chessboard images of different postures, read the corner points of the chessboard images, and obtain the camera intrinsic parameter matrix and distortion parameters through Zhang Zhengyou calibration method. The following method can be used for transformation:
[0063] Internal parameter matrix where f x is the normalized focal length on the u-axis of the pixel coordinate system, f y is the normalized focal length on the v-axis of the pixel coordinate system; u0 and v0 are the center coordinates in the pixel coordinate system, in pixels; let
[0064]
[0065]
[0066] Thus, the internal parameter matrix M is obtained;
[0067] The distortion parameters include radial distortion parameters k1, k2, k3 and k4 and tangential distortion parameters p1 and p2. Let (x0, y0) be the original position of the distortion point in the image coordinate system, (x, y) be the position of the distortion point in the image coordinate system after correction, r 2 =x 2 +y 2 ; The distortion parameters are obtained by the following formula:
[0068]
[0069] In order to better confirm the real-time posture of larger-sized containers, the above step S300 also includes a step of unifying the coordinates of the third industrial camera 23 and the fourth industrial camera 24 with the first industrial camera 21 and the second industrial camera 22: when the container has a ground mark across it, the third industrial camera 23 and the fourth industrial camera 24 are respectively aligned with the images of the ground mark across which the container is across, and the coordinate values of the third industrial camera 23 and the fourth industrial camera 24 in the world coordinate system of the corresponding ground mark are obtained. After the sling is unlocked and raised, the above steps are repeated to correct the distortion of the third industrial camera 23 and the fourth industrial camera 24, and the two-dimensional rotation angles and axial translations of the three axes of the third industrial camera 23 or the fourth industrial camera 24 in space are obtained, and the initial postures of the third industrial camera 23 and the fourth industrial camera 24 are saved, so as to eliminate the influence of the installation deviation of the third industrial camera 23 and the fourth industrial camera 24 caused by the installation.
[0070] In the above-mentioned step S500 or S700, when there is a ground mark across the container placement position, the image processing module 300 also processes the images of the ground mark taken by the third industrial camera 23 and the fourth industrial camera 24 at the same time, and respectively calculates the postures of the third industrial camera 23 and the fourth industrial camera 24 in the world coordinate system established by the corresponding ground mark, so that during the lowering process of the sling, the image acquisition module 200 and the image processing module 300 can detect in real time the relative position of the current container's posture relative to the container placement position determined by the ground mark, and the subsequent process is the same as step S500 or S700.
[0071] In the above steps S500-S700, according to the current situation of the yard, corresponding container grabbing or directional actions are performed to implement different countermeasures for different container placement positions. The container corresponding to S700 has a cross-mounted ground mark, that is, the length of the container is relatively large, such as the corresponding 40-foot cabinet. The cross-mounted ground marks corresponding to the third industrial camera 23 and the fourth industrial camera 24 can indicate the edge position of the middle part of its length direction, which can help improve the efficiency of grabbing and releasing boxes and shorten the posture adjustment time. The image processing module 300 can also obtain the execution status of the current sling's grabbing and releasing box action, return to step S500 or S700, and prepare for subsequent further actions.
[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A container automatic pick-and-place system based on machine vision, characterized by: It comprises a gantry, a lifting device, a ground marking unit (100), an image acquisition module (200), an image processing module (300) and a control module (400); the gantry is provided with a lifting device; A ground marking unit (100) comprises a plurality of ground markings, each of which is arranged in a linear array on the ground of the yard and fixed on the ground, and adjacent ground markings are arranged at intervals; each ground marking corresponds to at least an end point position of a container placement position; The image acquisition module (200) comprises a first industrial camera (21), a second industrial camera (22) and four pinhole cameras, wherein the first industrial camera (21) and the industrial camera are centrally symmetrically arranged at the end positions of the two long sides of the spreader in the extending direction, and the first industrial camera (21) and the second industrial camera (22) respectively capture images of ground markings at both ends of the container placement position; and each pinhole camera is respectively arranged at the locks on the two short sides of the spreader, and each pinhole camera captures an image of a lock hole or a corner of a container below the spreader; The image processing module (300) receives images of the ground mark taken by the first industrial camera (21) and the second industrial camera (22), and images of the keyhole or the corner of the container under the spreader taken by the pinhole cameras, extracts geometric features from the images of the ground mark, the keyhole or the corner of the container, obtains the position of the spreader relative to the ground mark or the container in the yard based on an N-point perspective algorithm, and obtains an angle offset and a position offset that need to be adjusted for the spreader, and the image processing module (300) sends the angle offset and the position offset to the control module (400); The control module (400) changes the position of the spreader according to the angle offset and the position offset sent by the image processing module (300), or sends a control signal for grabbing and releasing a container, so as to adjust the position of the spreader and the container or to achieve a grabbing and releasing action; When it is necessary to execute a container grabbing and releasing action, the spreader moves to the target container placement position, the image processing module (300) obtains images of the ground mark taken by the first industrial camera (21) and the second industrial camera (22), and based on the calibrated camera internal parameters and distortion coefficients, uses an N-point perspective algorithm to solve the external parameters of the ground mark in the camera, and then based on the calibrated relative posture between the first industrial camera, the second industrial camera and the spreader, converts the current posture of the spreader in the world coordinate system to obtain the angle offset and position offset that need to be adjusted for the spreader; the control module (400) makes the spreader located directly above the container or the container placement position corresponding to the ground mark according to the angle offset and position offset sent by the image processing module (300); The image processing module (300) also selectively obtains container corner images through each pinhole camera to extract and calculate the container corners, so that the spreader or the container on the spreader is further aligned with the ground mark or the four corners of the placed container before being lowered; The ground markings are respectively distributed in a linear array along the horizontal movement direction of the gantry trolley and the horizontal movement direction of the gantry trolley, and at least two different ground markings can determine a container placement position; The established world coordinate system takes the horizontal movement direction of the gantry trolley as the X-axis, the horizontal movement direction of the gantry trolley as the Y-axis, the vertical direction as the Z-axis, and the shape center of each ground mark as the zero point to establish a world coordinate system based on each ground mark; The ground markings are a group of centrosymmetric patterns, each pattern corresponding to an end point of a container placement position; The container further comprises a third industrial camera (23) and a fourth industrial camera (24). The third industrial camera (23) and the fourth industrial camera (24) are respectively arranged at the center of the two long sides of the sling. The third industrial camera (23) and the fourth industrial camera (24) selectively aim at the ground mark arranged across the two ends of the container and obtain the image of the ground mark there.
2. The automatic container pick-and-place system based on machine vision according to claim 1, characterized in that: The pattern of the ground mark is rectangular, L-shaped, fan-shaped or circular.
3. An operating method of a container automatic pick-and-place system based on machine vision, characterized in that: The steps include: S100: configuring the automatic container grabbing and placing system based on machine vision as described in any one of claims 1 to 2 on the gantry; S200: performing a single camera calibration on each industrial camera and pinhole camera to obtain an intrinsic parameter matrix and distortion parameters of each industrial camera and pinhole camera; S300: Calibrate each industrial camera: Use an actual lifting box to assist in the calibration, establish a world coordinate system based on each ground mark, and unify the coordinates of the first industrial camera (21) and the second industrial camera (22); manually lift the container to the center of a container placement position, so that the four end points of the container close to the ground are at the same distance from the center of the ground marks of the corresponding end points; respectively photograph the corresponding ground marks and measure the coordinate values of the first industrial camera (21) and the second industrial camera (22) in the world coordinate system of the ground marks photographed by them, then unlock the lifting device and raise it to a certain height, and when the lifting device is stationary, the first industrial camera (21) and the second industrial camera (22) continue to photograph the images of the corresponding ground marks, and the image processing module (300) uses the distortion parameters obtained in the previous step, Correct the distortion of the first industrial camera (21) and the second industrial camera (22), then calculate the rotation matrix and translation matrix of the first industrial camera (21) and the second industrial camera (22), i.e., the position and posture of the first industrial camera (21) and the second industrial camera (22) by applying the N-point perspective algorithm according to the camera internal parameters; then obtain the two-dimensional rotation angle and axial position of the first industrial camera (21) or the second industrial camera (22) in three axes in space according to the rotation matrix, and save the initial position and posture of the first industrial camera (21) and the second industrial camera (22); repeat the above steps at different hoist heights; fit the initial position and posture of the first industrial camera (21) and the second industrial camera (22) when the hoist is at different heights, and eliminate the influence of the installation deviation of the first industrial camera (21) and the second industrial camera (22); S400: for the relative positions of the four pinhole cameras, using the obtained distortion parameters of each pinhole camera, when the sling is unlocked and raised, the images of each pinhole camera are acquired at the height, the feature points of the keyhole or the box corner of the keyhole image or the box corner image are extracted, the pixel coordinate values at the height are saved, and the above steps are repeated at different sling heights; the corresponding relationship between the pixel coordinates of the four pinhole cameras and the feature points of the four keyholes or the box corners is fitted; S500: When there is no container placed at the corresponding container placement position on the ground and a container needs to be placed: the spreader drives the container to move to the container placement position, the image processing module (300) processes the images of the ground mark taken by the first industrial camera (21) and the second industrial camera (22), respectively calculates the current rotation matrix and translation matrix of the first industrial camera (21) and the second industrial camera (22) in the world coordinate system, converts the rotation matrix and translation matrix into two-dimensional rotation angle offsets and axial offsets of three axes in the space relative to the initial posture, and sends the two-dimensional rotation angle offsets and axial offsets of the three axes to the control module (400), respectively. The control module (400) first adjusts the posture of the spreader and the container according to the rotation angle offset and the axial offset. During the spreader lowering process, the image acquisition module (200) and the image processing module (300) detect in real time that the posture of the current spreader is kept flush with the relative position of the container placement position determined by the ground mark, thereby realizing the container placement action; S600: When there is a container placed at the corresponding container placement position on the ground and the containers need to be stacked: the spreader brings the container to the top of the target container placement position, the image processing module (300) processes the current rotation matrix and translation matrix of the first industrial camera (21) and the second industrial camera (22) in the world coordinate system, obtains the two-dimensional rotation angle offset and the axial offset of the three axes in the space, and sends the two-dimensional rotation angle offset and the axial offset of the three axes to the control module (400), the control module (400) first adjusts the posture of the spreader and the container according to the rotation angle offset and the axial offset, to ensure that the edge of the container lifted by the spreader is parallel to the edge of the container placed below, during the process of lowering the spreader, the image processing module (300) also adjusts the posture of the spreader according to the images of the keyhole or the images of the box corners obtained by the four pinhole cameras, and the control module (400) makes corresponding adjustments to ensure that the container lifted by the spreader is aligned with the four box corners of the placed container, and completes the container placement action of the stacked container; S700: When a container is placed at a corresponding container placement position on the ground and an empty spreader is required to grab it: the empty spreader reaches the top of the target container and is lowered, the image processing module (300) processes the images of the ground mark obtained by the first industrial camera (21) and the second industrial camera (22), obtains three-axis two-dimensional rotation angle offsets and position offsets, and sends the angle offsets and axial offsets to the control module (400). The control module (400) adjusts the posture of the spreader and lowers it. The image processing module (300) also adjusts the posture of the spreader according to the images of the lock hole or the images of the box corners obtained by the four pinhole cameras, so that when the spreader falls vertically, the lock on the spreader can directly fall into the lock hole of the container to realize the container grabbing action.
4. The method for operating a container automatic pick-and-place system based on machine vision according to claim 3 is characterized in that: The individual camera calibration for each industrial camera and pinhole camera is to preset a chessboard with a known size, use each industrial camera to shoot chessboard images in different postures, read the corner points of the chessboard images, and obtain the camera intrinsic parameter matrix and distortion parameters through the Zhang Zhengyou calibration method.
5. The method for operating a container automatic pick-and-place system based on machine vision according to claim 3, characterized in that: Step S300 also includes a step of unifying the coordinates of the third industrial camera (23) and the fourth industrial camera (24) with the first industrial camera (21) and the second industrial camera (22): when the container has a ground mark across it, the third industrial camera (23) and the fourth industrial camera (24) are respectively aligned with the image of the ground mark across which the container is across, and the coordinate values of the third industrial camera (23) and the fourth industrial camera (24) in the world coordinate system of the corresponding ground mark photographed are obtained. After the sling is unlocked and raised, the above steps are repeated to correct the distortion of the third industrial camera (23) and the fourth industrial camera (24), and the two-dimensional rotation angles and axial translation amounts of the third industrial camera (23) or the fourth industrial camera (24) in three axes in space are obtained, and the initial postures of the third industrial camera (23) and the fourth industrial camera (24) are saved to eliminate the influence of the installation deviation of the third industrial camera (23) and the fourth industrial camera (24).
6. The method for operating a container automatic pick-and-place system based on machine vision according to claim 4, characterized in that: In step S500 or S700, when there is a ground mark across the container placement position, the image processing module (300) also simultaneously processes the images of the ground mark taken by the third industrial camera (23) and the fourth industrial camera (24), and respectively calculates the postures of the third industrial camera (23) and the fourth industrial camera (24) in the world coordinate system established by the corresponding ground mark, so that during the lowering process of the sling, the image acquisition module (200) and the image processing module (300) can detect in real time the relative position of the posture of the current container relative to the container placement position determined by the ground mark, and the subsequent process is the same as step S500 or S700.
