Automatic container stacking method, system, equipment and medium based on visual inspection

Abstract

The present application provides a method, system, equipment and medium for automatic stacking of containers based on visual detection, which is applied to the technical field of automatic stacking of containers in ports. By adding customized road spike markers and using ordinary high-definition cameras to capture images of ground markers, no high-performance industrial computer is required, which significantly reduces the overall cost. Through deep learning, the container position markers and customized road spike markers are identified, so as to fit the center coordinates of the container position, and then the spreader is controlled to automatically stack the containers according to the center coordinates. This is not only simple and reliable to implement, but also has relatively low hardware costs. Moreover, with the assistance of road spike markers, accurate identification can be achieved both during the day and at night, solving the problem of marker identification at night / in rainy and foggy days, and realizing efficient and accurate stacking operations.

Description

Technical Field

[0001] The present application relates to the technical field of automatic stacking of containers in ports, and in particular to a method, system, equipment and medium for automatic stacking of containers based on visual inspection. Background Art

[0002] Container stacking is a critical and frequent operation in container terminal operations. With the continuous growth of port business, higher requirements are being placed on the efficiency and accuracy of container stacking. Traditional manual operation methods are not only inefficient but also prone to errors, making them unable to meet the requirements of fast and efficient operations in modern ports. Therefore, achieving automated container stacking has become a key approach to improving the efficiency and quality of port operations.

[0003] Currently, mainstream automatic container stacking technology relies mainly on lidar, 3D cameras, and traditional image processing technologies. Although automation has been achieved to a certain extent, many problems exist. For example, the lidar-based system is expensive and sensitive to environmental interference; the 3D camera-based system has strict requirements on lighting conditions; and the traditional image processing-based system has low detection accuracy and cannot meet the requirements of high-precision stacking. This limits the promotion and application of existing technologies and affects the further improvement of port operation efficiency.

[0004] Based on this, a new solution for automatic container stacking is needed. Summary of the Invention

[0005] In view of this, an embodiment of this specification provides a method for automatic container stacking based on visual detection, which uses a camera installed on a spreader to capture images of ground markers, and uses deep learning to identify ground container position markers and customized road spike markers in the image, thereby fitting the target container position posture, and finally controlling the spreader according to the target container position posture to accurately perform the container stacking operation.

[0006] The embodiments of this specification provide the following technical solutions:

[0007] The embodiments of this specification provide a method for automatically stacking containers based on visual inspection, including:

[0008] Use the camera installed on the spreader to collect images of ground markers;

[0009] Based on deep learning, identifying ground bay markers and road stud markers in the ground marker image;

[0010] Obtaining a target bay position according to the ground bay mark and the road spike mark, wherein the target bay position includes: a bay center coordinate and a bay rotation angle;

[0011] According to the target bay position and posture, combined with the spreader posture feedback from the spreader detection system, a posture deviation between the spreader posture and the target bay position and posture is obtained;

[0012] The spreader movement is controlled, and when the posture deviation is less than a preset threshold, the box stacking operation is performed.

[0013] The embodiment of this specification also provides a container automatic stacking system based on visual detection, and the container automatic stacking system based on visual detection includes:

[0014] The identification acquisition module is used to collect ground identification images using a camera installed on the spreader; based on deep learning, it identifies ground bay markers and road stud markers in the ground identification images;

[0015] A posture acquisition module is used to obtain a target bay position posture according to the ground bay mark and the road spike mark, wherein the target bay position posture includes: bay center coordinates and bay rotation angle;

[0016] a deviation analysis module, configured to obtain a posture deviation between the spreader posture and the target spreader posture based on the target spreader posture and the spreader posture fed back by the spreader detection system;

[0017] The control module is used to control the movement of the spreader and perform the box stacking operation when the posture deviation is less than a preset threshold.

[0018] An embodiment of this specification also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned method for automatic container stacking based on visual inspection is implemented.

[0019] The embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the automatic container stacking method based on visual inspection as described above is implemented.

[0020] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0021] 1. This application adds ground identification objects. Even at night and in rainy conditions, or when the bay is obstructed, it can identify road spikes and calculate the target bay position, including center coordinates and rotation angle, thereby achieving accurate box stacking operations.

[0022] 2. This application does not rely on expensive lidar or 3D cameras, nor does it require high-performance industrial computers to process large amounts of data. This not only significantly reduces the hardware cost of the system, but also reduces maintenance difficulty and simplifies the overall complexity of the system.

[0023] 3. This application does not need to process large amounts of data like lidar or 3D cameras, which enables faster response and execution, significantly improving operational efficiency and helping to meet the port's demand for efficient operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 This is a flow chart of a method for automatic container stacking based on visual inspection in this application;

[0026] Figure 2 This is a schematic diagram of image stitching in this application;

[0027] Figure 3 It is the identification diagram of the corner points of the road spikes in the ground space in this application;

[0028] Figure 4 This is the splicing effect diagram after the angle compensation of the corner points of the spikes in this application;

[0029] Figure 5 It is the container yard identification map in this application;

[0030] Figure 6 This is a schematic diagram of the container bottom opening operation in this application;

[0031] Figure 7 This is a schematic diagram of the container stacking operation in this application;

[0032] Figure 8 This is the architecture diagram of the container automatic stacking system in this application;

[0033] Figure 9 is a schematic diagram of the structure of the electronic device in this application;

[0034] In the figure, 1. Ground level marker; 2. Road spike marker; 3. Container; 4. Stacked containers; 5. Spreader; 6. Spreader detection system; 7. Trolley. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0036] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0037] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0038] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0039] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples, however, one skilled in the art will appreciate that the examples can be practiced without these specific details.

[0040] With the increasing demand for logistics and port automation, automatic container stacking technology has become the key to improving efficiency. Traditional methods rely on manual operations, which are inefficient and prone to errors, and cannot meet the requirements of efficient operations in modern ports.

[0041] There are several existing automatic container stacking solutions:

[0042] First, the automatic container stacking solution based on lidar uses lidar to scan containers and yard environments, generate three-dimensional point cloud data, and combine point cloud processing algorithms to realize container positioning and posture estimation. It is suitable for complex environments. Although it can achieve high-precision three-dimensional positioning, the lidar equipment is expensive and requires a matching high-performance industrial machine to process large amounts of data, which significantly increases the overall cost of the system. Moreover, the point cloud data processing algorithm is complex and difficult to develop and maintain. In addition, lidar is sensitive to environmental interference such as dust, rain and snow, and its performance will degrade in bad weather.

[0043] Second, there is the automatic stacking solution based on 3D cameras. This solution obtains three-dimensional information of containers through 3D cameras and uses image processing algorithms to realize container positioning and posture estimation. It is suitable for complex environments. Although it can achieve high-precision three-dimensional positioning, the 3D camera equipment is expensive and requires a matching high-performance industrial machine to process large amounts of data, which significantly increases the overall cost of the system. Moreover, the three-dimensional reconstruction algorithm is complex and difficult to develop and maintain. In addition, 3D cameras have strict requirements on lighting conditions, and their performance will degrade at night or in low light.

[0044] Third, the automatic stacking solution based on traditional image processing uses ordinary cameras to obtain two-dimensional images of containers, and combines traditional image processing algorithms (such as edge detection and template matching) to achieve container positioning and posture estimation. Although the hardware cost is low and the implementation is simple, the detection accuracy is low and it is difficult to meet the needs of high-precision stacking, especially when the ground and container surfaces are damaged or the yard environment is complex. In addition, it is sensitive to environmental conditions such as light and weather, and its performance is significantly reduced at night or in bad weather.

[0045] In view of this, the inventors discovered through research and improvement exploration that although traditional image processing solutions are low-cost, they have obvious deficiencies in accuracy and cannot meet the requirements of high-precision container stacking, especially when the ground or container surface is damaged, and when the yard environment is relatively complex, its performance is even more difficult to achieve ideal. The solutions using lidar or 3D cameras, although excellent in some aspects, the lidar, 3D camera itself and the required high-performance industrial control computer greatly increase the overall cost of the system. In addition, the development and maintenance of point cloud and three-dimensional reconstruction algorithms are difficult, and the system integration process is also relatively complex.

[0046] Moreover, existing solutions have poor adaptability to the environment and are sensitive to conditions such as light, rain, snow, and dust. Their performance will drop sharply at night or in bad weather.

[0047] Based on this, the embodiment of this specification proposes a method for automatic stacking of containers based on visual detection: the overall idea is: adding customized road spike markers, using ordinary high-definition cameras to capture images of ground markers, without the need for high-performance industrial computers, significantly reducing the overall cost, and through deep learning, identifying the container position markers and customized road spike markers, thereby fitting the center coordinates of the container position, and then controlling the spreader to automatically stack the containers according to this center coordinate. This is not only simple and reliable to implement, but also has relatively lower hardware costs. Moreover, with the assistance of road spike markers, accurate identification can be achieved both day and night, solving the problem of marker identification at night / rainy and foggy days, and realizing efficient and accurate stacking operations.

[0048] The following describes the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown, the embodiment of this specification provides a method for automatically stacking containers based on visual inspection, including:

[0050] Using the camera installed on the spreader 5 to collect images of ground markers;

[0051] Based on deep learning, identifying ground bay markers and road stud markers in the ground marker image;

[0052] Obtaining a target bay position according to the ground bay mark and the road spike mark, wherein the target bay position includes: a bay center coordinate and a bay rotation angle;

[0053] According to the target bay position and posture, combined with the spreader posture feedback from the SDS spreader posture detection system 6, the posture deviation between the spreader posture and the target bay position and posture is obtained;

[0054] The spreader movement is controlled, and when the posture deviation is less than a preset threshold, the box stacking operation is performed.

[0055] Specifically, a camera is installed at each of the four corners of the spreader 5, with the lens pointing downward to ensure that the image of the ground marker can be clearly captured, such as Figure 5 As shown, the ground markers include a ground bay position marker 1 and a customized road spike marker 2, wherein the ground bay position marker 1 is used to mark the ground position of the container in the container yard, and the container 3 is usually placed in the center of the area; the road spike marker 2 is used to assist in calculating the center coordinates and posture of the bay.

[0056] It should be noted that the size and spacing of ground markers are designed according to actual operational requirements. Taking into account the particularity of the port operating environment, high-resolution, high-frame-rate HD surveillance cameras can be selected.

[0057] During implementation, the main control system controls the spreader to move to a specified position above the target bay according to the operation instructions. When the spreader moves to the specified position, the camera is triggered to collect images. The camera triggering can be completed automatically by the main control system according to the position information of the spreader, or through specific conditions detected by the sensor, which is not limited here.

[0058] The collected images are recognized using a pre-trained deep learning algorithm to identify ground bend marks and road spike marks. Based on the recognized ground bend marks and road spike marks, the center coordinates and rotation angle of the target bend are calculated. The main control system calculates the posture deviation (including position deviation and angle deviation) between the spreader posture and the target bend posture based on the spreader posture (including position and posture) fed back by the SDS spreader posture detection system. Based on the calculated posture deviation, the main control system adjusts the position and posture of the spreader to align it with the posture of the target bend. When the posture deviation is less than the preset threshold, the box stacking operation is performed.

[0059] like Figure 6 As shown, if there is no container at the target bay, the spreader is controlled to place the target container on the ground, ensuring that the center coordinates of the spreader and the center coordinates of the target bay are as close as possible to each other.

[0060] like Figure 7 As shown, if there is already a container at the target bay, the spreader is controlled to stack the target container on the lower container (stacked container 4), and it is also necessary to ensure that the center coordinates of the spreader and the center coordinates of the target bay are as close as possible.

[0061] Through the above steps, the present invention can realize high-precision automatic stacking operation of containers, which not only improves the efficiency and precision of stacking, but also reduces the cost of the system, enhances the environmental adaptability of the system, and has broad application prospects.

[0062] It should be noted that the method in this application is also applicable to the box grabbing scenario.

[0063] In some embodiments, obtaining the target bay position and posture according to the ground bay mark and the road spike mark includes:

[0064] Extracting the corner coordinates of the road spike marker;

[0065] Based on the physical size of the road stud marker and the camera parameters, a homography matrix is ​​constructed to map the corner point coordinates to the ground coordinate system. The formula is:

[0066] ;

[0067] Where s represents the scale factor; u and v represent the coordinates of the point on the ground marker image plane; K represents the camera parameters; X and Y are the coordinates of the point on the ground coordinate system; represents the rotation vector; t represents the translation vector;

[0068] According to the mapping results, the initial mapping coordinates are obtained;

[0069] Obtaining the camera's installation angle deviation around the Z axis based on the corner point coordinates;

[0070] Performing rotation compensation on the initial mapping coordinates according to the installation angle deviation to obtain compensated mapping coordinates;

[0071] The target position and posture are obtained according to the compensated mapping coordinates.

[0072] During implementation, the Z-axis of the ground coordinate system is defined as vertically upward. Due to the inherent deviation of mechanical installation, the optical axes of the cameras at the four corners of the sling may not be completely perpendicular to the ground, and each camera has an independent rotation angle offset around the Z-axis. The image homography transformation can be used to map a point in an image to a corresponding point on another plane, thereby establishing a mapping relationship between the image plane and the ground coordinate system, that is, a transformation relationship from one plane to another.

[0073] Specifically, the correspondence between the XY point on the ground coordinate system and the image plane point uv is described by Formula 1, which is as follows:

[0074] (1)

[0075] Where s represents the scale factor, which is the inverse of the depth from the camera to the ground; K represents the camera internal parameters.

[0076] make That is the homography matrix of the two planes, which is affected by the camera parameters and camera and ground coordinate system rotations and the effect of translation t.

[0077] When the point on the image plane is known When you can use Get the corresponding coordinates of each image point on the image in ground coordinates Given 4 or more pairs of image points and corresponding points in the ground coordinate system, the homography matrix H can be obtained using functions in toolboxes such as openCV. In practical applications, the four corner points of the road spikes can be selected as corresponding points to accurately map the points on the image to the ground coordinate system.

[0078] After obtaining the H matrix based on the determined features, all points on the image plane can be mapped to the ground, thereby obtaining a space that satisfies the Euclidean distance metric. Since the installation position and angle of each camera may be different, each camera will generate an independent H matrix. If four cameras are directly spliced, the mapping result is as follows: Figure 2 As shown, the edge of the spike will not be parallel to the coordinate system ( Figure 2 (where 1 to 4 are image numbers and A to D are corresponding road stud images) and the deviation needs to be corrected:

[0079] The rotation offset is calculated based on the corner coordinates of the spike marker, specifically, Figure 3 As shown, let the coordinates of the corner points be to , then, according to and , using the function , get the angle between one of the sides of the spike mark and the ground coordinate system.

[0080] The initial mapping coordinates are rotated so that the edge of the road stud is parallel to the ground coordinate system. The stitching result of the four camera images is as follows: Figure 4 As shown, the effect caused by camera installation deviation is effectively corrected.

[0081] Through the above steps, the position and posture of the target bay can be stably extracted based on the ground bay mark and the road spike mark, including the bay center coordinates and bay rotation angle, providing a reliable basis for subsequent precise box stacking operations.

[0082] In some embodiments, when constructing the homography matrix, the translation vector is obtained based on the distance between the camera and the road stud marker, including:

[0083] Use laser ranging to obtain the distance between the camera and the road stud mark;

[0084] Alternatively, the distance from the camera to the road spike marker is obtained based on the size of the road spike marker in the ground marker image and the physical size of the road spike marker.

[0085] In the implementation, the camera height changes continuously due to the lifting of the sling, so the homography matrix in , Nothing changes, only the translation vector t needs to be updated in real time. The following method can be used to dynamically obtain the translation vector t:

[0086] You can use the pre-calibrated physical size of the road stud and the camera internal parameter matrix , the distance information from the camera to the road spike is obtained through the length and width of the road spike in the image, thereby obtaining the translation vector t.

[0087] The precise height of the camera from the ground can also be obtained in real time through the laser sensor carried by the sling, thereby obtaining the translation vector t.

[0088] In some embodiments, obtaining the installation angle deviation of the camera around the Z axis according to the corner point coordinates includes:

[0089] Obtaining the angles between each side of the road spike marker and the ground coordinate system based on the corner point coordinates;

[0090] The angles of each side are weighted to obtain the actual deflection angle of the camera around the Z axis.

[0091] In combination with the above embodiment, in order to improve the robustness of the estimation, the angle of each edge can be calculated and weighted to obtain the angle of the spike (90 degrees should be subtracted when weighting the vertical edge angle).

[0092] In some embodiments, obtaining the target position and posture according to the compensated mapping coordinates includes:

[0093] Obtaining the center coordinates of each road spike marker according to the compensated mapping coordinates;

[0094] When all road spike markers are visible, obtaining the center coordinates of the bay position according to the center coordinates of each road spike marker;

[0095] Obtaining the angle between the direction vector of each side of the bay and the ground coordinate system based on the center coordinates of each road spike marker; weighting the angle of each side to obtain the rotation angle of the bay relative to the ground coordinate system;

[0096] When two road spikes on the same long side are visible, the long side of the bay and the angle of the long side of the bay are obtained according to the center coordinates of the road spikes;

[0097] Obtaining the center coordinates of the bay position according to the bay position long side and the bay position fixed aspect ratio;

[0098] Obtaining a rotation angle of the bay position relative to a ground coordinate system according to the angle of the long side of the bay position;

[0099] When two road spikes on the same short side are visible, the short side of the bay and the angle of the short side of the bay are obtained according to the center coordinates of the road spikes.

[0100] Obtaining the center coordinates of the bay position according to the bay position short side and the bay position fixed aspect ratio;

[0101] According to the angle of the short side of the bay position, a rotation angle of the bay position relative to the ground coordinate system is obtained;

[0102] When the diagonal spike point is visible, the center coordinates of the bay position and the diagonal angle of the bay position are obtained according to the center coordinates of the spike mark;

[0103] According to the bay position diagonal angle and the bay position fixed aspect ratio, the rotation angle of the bay position relative to the ground coordinate system is obtained.

[0104] In implementation, such as Figure 4 As shown, the center positions of A~D can be easily obtained, and these center positions are A1~D1 respectively.

[0105] When all the spikes can be detected, the center point of the long side of the bay can be used When the spike detection is missing, the center point of the bay is obtained based on a single long side, and the center point of the short side can be obtained similarly.

[0106] The angle of the long side of the bay can be obtained using the function atan2.

[0107] If all four road spikes are detectable, the weighted angle information can be obtained by using the method of obtaining the center point of the bay. If only the points on the same long side are obtained, the function , get the long side angle.

[0108] Similarly, the angle of the short side can be obtained. Since the two angles differ by 90 degrees, they can be weighted to obtain a more stable result.

[0109] If only the diagonal spike points are detected, the angle is estimated using the following formula:

[0110] Assume the length and width of the bay are W and H respectively, then its diagonal angle is , this value is fixed for the same specification of the bay. Note that the angle of the other diagonal line is The actual detected diagonal angle is T1, and the angle difference between T1 and the corresponding diagonal is the angle of the bay position.

[0111] In some embodiments, the road spike markers are deployed at the centers of adjacent bay vertices, and the physical size and ground coordinate positions of the road spike markers are pre-calibrated.

[0112] During implementation, by deploying road spike markers at the centers of adjacent bay corners, uniform coverage of the entire yard can be achieved. This not only improves positioning accuracy, but also ensures that the camera can clearly capture these markers during the descent of the spreader. Even if some road spike markers are blocked or damaged, positioning and posture estimation can still be performed through other road spike markers.

[0113] In some embodiments, the method for automatic container stacking based on visual inspection further includes:

[0114] Assign a unique number to each bay in the container yard;

[0115] When it is necessary to generate the position and posture of the target bay position, if the ground mark and the road spike mark corresponding to the target bay position are identified, the position and posture of the target bay position are generated based on the identified ground mark and the road spike mark, the spreader is controlled to perform the box stacking operation, and the generated position and posture of the target bay position and the number corresponding to the bay position are stored in the database;

[0116] If the ground mark or spike mark corresponding to the target bay position is not identified, the historical bay position posture corresponding to the target bay position number is extracted from the database, and the spreader is controlled to perform the box stacking operation based on the historical bay position posture.

[0117] In practice, each bay in the yard is assigned a unique number and stored in a database.

[0118] When the spreader moves above the target bay, the camera is triggered to capture images of ground markers and road spike markers, and the deep learning algorithm is used to identify the ground markers and road spike markers in the image. If the recognition is successful, the position and posture of the target bay are calculated based on these markers, and the posture and bay number are stored in the database. The movement of the spreader is controlled according to the calculated posture to perform the box stacking operation.

[0119] If the ground sign or road spike sign of the target bay cannot be identified, the historical bay position posture corresponding to the target bay number is extracted from the database, and the movement of the spreader is controlled according to the historical bay position posture to perform the stacking operation. Even if the ground sign or road spike sign is blocked, damaged, or the light is extremely bad and cannot be identified, the system can still use historical data to complete the stacking operation. As the operation progresses, more and more bay position posture data are accumulated in the database. These data can be used to further optimize the performance of the system, thereby improving the accuracy of positioning and posture estimation, and the efficiency of stacking will become higher and higher in the later stage.

[0120] It should be noted that the bottom opening action (empty storage location, first case placement) in this application aims to align the center of the container and rotate the container around the z-axis to zero. As long as the subsequent case stacking action is aligned with the center and angle of the container, each stacked case will maintain the same position and posture. This application is also applicable to case grabbing scenarios. As long as the control detects the container position and controls the spreader to coincide with the container position, accurate case grabbing can be achieved.

[0121] Here is another example, which is a schematic illustration formed by combining the above examples.

[0122] like Figure 8As shown, the main control system receives the stacking instruction, controls the trolley to move to the specified position, and then moves the trolley 7 to the top of the container 3 to be grabbed. The main control system controls the lowering of the spreader 5. During the lowering process of the spreader 5, the SDS spreader posture detection system 6 continuously calculates the position, posture and vertical distance of the spreader 5 in real time, and transmits the calculation results to the main control system.

[0123] When the spreader 5 reaches the designated position (the spreader's XY coordinates and the calculated center coordinates of the ground container or the center coordinates of the lower container are within 2 cm, and the deflection angle is less than 0.2 degrees), it automatically grabs the target container and automatically pulls it to the top. It should be noted that after the container is detected, the spreader's posture is adjusted based on the position of the container (the container), thereby achieving container grip and landing.

[0124] The trolley 7 moves to the position where the target container needs to be stacked according to the instruction of the main control system, and starts to control the spreader to be lowered to the ground or the lower container.

[0125] For operations involving placing containers on the ground, known as bottom opening operations, the sub-control system uses image recognition technology to calculate the center coordinates of the current container position based on the ground container position markers and road spike markers captured by the spreader's four corner cameras. The main control system then controls the spreader, which has already grasped the target container, to lower it to the ground based on the spreader's position and posture information fed back by the SDS system, ensuring that the spreader's center coordinates coincide as closely as possible with the calculated ground container position center coordinates. For stacking containers on the second or higher levels, since the system has already calculated the center coordinates of the current container position through image recognition, it only needs to continue to control the spreader to stack the target container on the lower container according to this center coordinate, in conjunction with the spreader's position and posture output by the SDS system.

[0126] Because each bay has a fixed and unique number, the center coordinates of each bay can be stored in the master control system's database during operation, which will increase the efficiency of subsequent stacking. Moreover, even if the ground markings are damaged or blocked, or in extremely poor lighting conditions, the bay center coordinates stored in the database can still be used for automated stacking operations.

[0127] Through actual box stacking operations at the work site, the present invention has been verified to achieve an offset of less than 2 cm between adjacent boxes and an overall offset of less than 5 cm for 5 layers of stacked boxes. The time for a single box stacking operation is between 2 minutes and 40 minutes and 3 minutes.

[0128] The following are examples of automatic stacking performance under different environmental conditions:

[0129]

[0130] Under different test environments, the algorithm recognition accuracy is ≥98%.

[0131] This application uses high-definition surveillance cameras installed at the four corners of the spreader to collect images of ground markers, and through deep learning, identifies ground bay markers and customized road spike markers, thereby fitting the center coordinates of the container bays, and then controls the tire crane trolley and spreader to automatically stack containers based on this center coordinate. Compared with existing products that use technologies such as lidar and 3D cameras, this application is not only simple and reliable to implement, but also has relatively lower hardware costs. Moreover, with the assistance of road spike markers, it can accurately identify both day and night, achieving efficient and accurate stacking operations. During the stacking operation, accurate detection of the spreader position and posture and supporting precise motion control are also required. This function is implemented by the SDS spreader posture detection system 6 (also called SDS system or spreader detection system). By identifying the LED target on the top of the spreader, the SDS system can accurately calculate the XY coordinates and rotation angles of the spreader. Through the supporting laser ranging radar, the height information of the spreader can be accurately measured, thereby providing support for precise control of the spreader for stacking operations.

[0132] This application identifies the four corners of the spikes, and then combines the specific bay position information (obtained based on the parking position of the truck, and the precise position of the truck is corrected in combination with the magnetic scale positioning data), calculates the coordinate information of the four corners in the reference coordinate system, and then calculates the center coordinates of the bay, thereby achieving precise box stacking operations.

[0133] This application can achieve an offset of <2cm between adjacent boxes, an overall offset of <5cm for 5-layer stacked boxes, and a single stacking operation time of between 2 minutes and 40 minutes and 3 minutes.

[0134] Based on the same inventive concept, the present application also provides an automatic container stacking system based on visual detection, the automatic container stacking system based on visual detection comprising:

[0135] The identification acquisition module is used to collect ground identification images using a camera installed on the spreader; based on deep learning, it identifies ground bay markers and road stud markers in the ground identification images;

[0136] A posture acquisition module is used to obtain a target bay position posture according to the ground bay mark and the road spike mark, wherein the target bay position posture includes: bay center coordinates and bay rotation angle;

[0137] a deviation analysis module for obtaining a posture deviation between the spreader posture and the target spreader posture based on the target spreader posture and the spreader posture fed back by the SDS spreader posture detection system 6;

[0138] The control module is used to control the movement of the spreader and perform the box stacking operation when the posture deviation is less than a preset threshold.

[0139] It should be noted that the settings of the unit module functions and the number of modules in the automatic container stacking system based on visual inspection can be made according to the aforementioned method embodiments, and will not be further explained here.

[0140] Based on the same inventive concept, the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute: the automatic container stacking method based on visual detection as described in any embodiment of the present application.

[0141] like Figure 9 As shown, the present application also provides a structural diagram of an electronic device, which shows the structure of the electronic device 500. The electronic device 500 here is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0142] The electronic device 500 may include: at least one processor 510; and a memory 520 communicatively connected to the at least one processor; wherein the memory 520 stores instructions that can be executed by the at least one processor 510, and the instructions are executed by the at least one processor 510 to enable the at least one processor 510 to execute: the automatic container stacking method based on visual detection described in any embodiment of the present application.

[0143] It should be noted that the electronic device 500 may be in the form of a general-purpose computing device, for example, it may be a server device.

[0144] In implementation, the components of the electronic device 500 may include but are not limited to: the above-mentioned at least one processor 510, the above-mentioned at least one memory 520, and a bus 530 connecting different system components (including the memory 520 and the processor 510), wherein the bus 530 may include a data bus, an address bus, and a control bus.

[0145] In implementation, the memory 520 may include a volatile memory, such as a random access memory (RAM) 5201 and / or a cache memory 5202 , and may further include a read-only memory (ROM) 5203 .

[0146] The memory 520 may also include a program tool 5205 having a set (at least one) of program modules 5204, such program modules 5204 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0147] The processor 510 executes various functional applications and data processing by running computer programs stored in the memory 520 .

[0148] The electronic device 500 can also communicate with one or more external devices 540 (e.g., a keyboard, pointing device, etc.). This communication can be performed via an input / output (I / O) interface 550. Furthermore, the electronic device 500 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 560. The network adapter 560 communicates with other modules in the electronic device 500 via a bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 500, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.

[0149] Based on the same inventive concept, the present application also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for automatically stacking containers based on visual inspection provided in any embodiment of the present application is implemented.

[0150] Specifically, the readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0151] In a possible embodiment, the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the automatic container stacking method based on visual detection provided by any embodiment of the present application.

[0152] The program code for executing the present invention may be written in any combination of one or more programming languages, and the program code may be executed entirely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0153] In this specification, the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments described later, the description is relatively simple, and the relevant parts can be referred to the partial description of the previous embodiments.

[0154] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for automatic container stacking based on visual inspection, characterized in that: include: Use the camera installed on the spreader to collect images of ground markers; Based on deep learning, identifying ground bay markers and road stud markers in the ground marker image; Obtaining a target bay position according to the ground bay marker and the road spike marker, including: constructing a homography matrix based on the corner point coordinates of the road spike marker, in combination with the physical size of the road spike marker and camera parameters, and mapping the corner point coordinates to a ground coordinate system to obtain initial mapping coordinates; obtaining an installation angle deviation of the camera around a Z axis according to the corner point coordinates, and performing rotation compensation on the initial mapping coordinates according to the installation angle deviation to obtain compensated mapping coordinates; Obtaining the center coordinates of each road spike marker according to the compensated mapping coordinates, and obtaining the target position and posture based on the center coordinates of the road spike marker and the visibility of the road spike; The visibility of the road spikes includes: all road spikes are visible, two road spikes on the same long side are visible, two road spikes on the same short side are visible, or diagonal road spike points are visible; When two road spikes on the same long side are visible, the long side of the bay and the angle of the long side of the bay are obtained according to the center coordinates of the road spikes; Obtaining the center coordinates of the bay position according to the bay position long side and the bay position fixed aspect ratio; Obtaining a rotation angle of the bay position relative to a ground coordinate system according to the angle of the long side of the bay position; When two road spikes on the same short side are visible, the short side of the bay and the angle of the short side of the bay are obtained according to the center coordinates of the road spikes. Obtaining the center coordinates of the bay position according to the bay position short side and the bay position fixed aspect ratio; According to the angle of the short side of the bay position, a rotation angle of the bay position relative to the ground coordinate system is obtained; When the diagonal spike point is visible, the center coordinates of the bay position and the diagonal angle of the bay position are obtained according to the center coordinates of the spike mark; Obtaining a rotation angle of the bay position relative to a ground coordinate system according to the bay position diagonal angle and the bay position fixed aspect ratio; According to the target bay position and posture, combined with the spreader posture feedback from the spreader detection system, a posture deviation between the spreader posture and the target bay position and posture is obtained; The spreader movement is controlled, and when the posture deviation is less than a preset threshold, the box stacking operation is performed.

2. The method for automatic container stacking based on visual inspection according to claim 1, characterized in that: When constructing the homography matrix, the translation vector is obtained based on the distance from the camera to the road stud marker, including: Use laser ranging to obtain the distance between the camera and the road stud mark; Alternatively, the distance from the camera to the road spike marker is obtained based on the size of the road spike marker in the ground marker image and the physical size of the road spike marker.

3. The automatic container stacking method based on visual inspection according to claim 1 is characterized in that: Obtaining the installation angle deviation of the camera around the Z axis according to the corner point coordinates includes: Obtaining the angles between each side of the road spike marker and the ground coordinate system based on the corner point coordinates; The angles of each side are weighted to obtain the actual deflection angle of the camera around the Z axis.

4. The automatic container stacking method based on visual inspection according to claim 1 is characterized in that: The road spike markers are deployed at the centers of the vertex corners of adjacent bays, and the physical size and ground coordinate positions of the road spike markers are pre-calibrated.

5. The automatic container stacking method based on visual inspection according to claim 1 is characterized in that: The automatic container stacking method based on visual detection also includes: Assign a unique number to each bay in the container yard; When it is necessary to generate the position and posture of the target bay position, if the ground mark and the road spike mark corresponding to the target bay position are identified, the position and posture of the target bay position are generated based on the identified ground mark and the road spike mark, the spreader is controlled to perform the box stacking operation, and the generated position and posture of the target bay position and the number corresponding to the bay position are stored in the database; If the ground mark or spike mark corresponding to the target bay position is not identified, the historical bay position posture corresponding to the target bay position number is extracted from the database, and the spreader is controlled to perform the box stacking operation based on the historical bay position posture.

6. A container automatic stacking system based on visual inspection, characterized in that: The automatic container stacking system based on visual detection includes: The identification acquisition module is used to collect ground identification images using a camera installed on the spreader; based on deep learning, it identifies ground bay markers and road stud markers in the ground identification images; a posture acquisition module for obtaining a target bezel posture based on the ground bezel marker and the road spike marker, comprising: constructing a homography matrix based on the corner point coordinates of the road spike marker, in combination with the physical size of the road spike marker and camera parameters, mapping the corner point coordinates to a ground coordinate system to obtain initial mapping coordinates; obtaining an installation angle deviation of the camera around the Z axis based on the corner point coordinates, and performing rotation compensation on the initial mapping coordinates based on the installation angle deviation to obtain compensated mapping coordinates; Obtaining the center coordinates of each road spike marker according to the compensated mapping coordinates, and obtaining the target position and posture based on the center coordinates of the road spike marker and the visibility of the road spike; The visibility of the road spikes includes: all road spikes are visible, two road spikes on the same long side are visible, two road spikes on the same short side are visible, or diagonal road spike points are visible; When two road spikes on the same long side are visible, the long side of the bay and the angle of the long side of the bay are obtained according to the center coordinates of the road spikes; Obtaining the center coordinates of the bay position according to the bay position long side and the bay position fixed aspect ratio; Obtaining a rotation angle of the bay position relative to a ground coordinate system according to the angle of the long side of the bay position; When two road spikes on the same short side are visible, the short side of the bay and the angle of the short side of the bay are obtained according to the center coordinates of the road spikes. Obtaining the center coordinates of the bay position according to the bay position short side and the bay position fixed aspect ratio; According to the angle of the short side of the bay position, a rotation angle of the bay position relative to the ground coordinate system is obtained; When the diagonal spike point is visible, the center coordinates of the bay position and the diagonal angle of the bay position are obtained according to the center coordinates of the spike mark; Obtaining a rotation angle of the bay position relative to a ground coordinate system according to the bay position diagonal angle and the bay position fixed aspect ratio; a deviation analysis module, configured to obtain a posture deviation between the spreader posture and the target spreader posture based on the target spreader posture and the spreader posture fed back by the spreader detection system; The control module is used to control the movement of the spreader and perform the box stacking operation when the posture deviation is less than a preset threshold.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for automatic container stacking based on visual inspection as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatic container stacking based on visual inspection as described in any one of claims 1 to 5 is implemented.

Images