Visual positioning method for grabbing pose of port unmanned crane

Abstract

The invention relates to the technical field of port automation, and discloses a port unmanned crane grabbing pose visual positioning method. The invention aims to solve the technical problems of high system cost, poor environmental adaptability and insufficient positioning robustness caused by excessive dependence of a port unmanned crane on a multi-source expensive sensor, complex calibration deployment process and easiness in signal interference by the environment. According to the method, a double-coordinate reference frame in which a site fixed coordinate system and a crane moving space coordinate system are linked is constructed, and full-process autonomous closed-loop control from path planning to accurate grabbing and unloading is realized by deeply fusing multi-dimensional visual image information. The accurate pose of the crane is calculated through a real-time visual self-calibration mechanism; and a multi-stage visual servo strategy is adopted to guide the grabbing unit to complete high-precision operation. By constructing a set of visual positioning and control system with low cost and high robustness, the autonomy level, the environmental adaptability and the economic efficiency of the system are greatly enhanced.

Description

Technical Field

[0001] This invention relates to the field of port automation technology, and more specifically, to a visual positioning method for grasping pose of an unmanned crane in a port. Background Technology

[0002] As the core equipment for container loading and unloading operations in ports, the operating efficiency of gantry cranes directly affects the lifeline of the entire terminal's operation. Traditional manned crane operations heavily rely on operators performing long hours of repetitive, high-intensity work in the operator's cab, making them highly susceptible to human factors such as fatigue and misjudgment. To achieve continuous, efficient, and safe operations in ports around the clock, the development of unmanned crane (unmanned crane) technology, replacing manual labor with automated systems, has become an inevitable trend in the construction of smart ports worldwide.

[0003] Current mainstream unmanned crane (UGC) technologies rely heavily on complex systems composed of multiple, expensive sensors for high-precision positioning and sensing. The most common approach is a combination of differential GPS and lidar. However, these technologies exhibit a series of inherent and insurmountable drawbacks in practical applications: differential GPS is highly susceptible to signal blockage and multipath interference in dense container yards or near tall ships, leading to a sharp drop in positioning accuracy or even temporary failure. While lidar offers high accuracy, its equipment is expensive, and its performance is significantly affected by adverse weather conditions common in ports, such as rain, fog, and dust, reducing its reliability. More importantly, the high procurement, deployment, and maintenance costs of a complete UGC system incorporating multi-source sensor fusion can even exceed the traditional model of ensuring operational safety through increased manpower and optimized scheduling. This significantly hinders the large-scale economic adoption of this technology.

[0004] This multi-sensor fusion approach not only increases the hardware cost and maintenance complexity of the entire system, but more importantly, it results in lower overall system robustness and environmental adaptability. Therefore, developing a lower-cost, more robust positioning method that can adapt to the complex and dynamic environment of ports, rather than relying on expensive and environmentally sensitive sensors, has become a key technological bottleneck for the further development and widespread adoption of unmanned crane technology. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, this invention provides a visual positioning method for grasping pose of unmanned cranes in ports, which solves the technical pain points of existing unmanned cranes in ports that rely too much on expensive multi-source sensors, and that are subject to complex calibration and deployment processes, high system costs, poor environmental adaptability, and insufficient positioning robustness due to the susceptibility of signals to environmental interference.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a visual positioning method for the grasping pose of an unmanned crane in a port, which includes the following steps: S1: Establish a reference coordinate system: Establish the absolute coordinate system of the work site and the relative coordinate system of the crane structure as a reference frame for positioning and control; S2: Positioning and Grabbing: Identify the initial pose of the target container in the absolute coordinate system, and plan the motion path by combining the initial pose of the target with the initial position coordinates of the crane; drive the crane to move along the motion path and complete the placement; during the grabbing stage, use the absolute coordinate system and the relative coordinate system to guide and control the grabbing unit until the grabbing is identified as complete. S3: Target transfer: Identify the position of the target placement platform in the absolute coordinate system and plan the transportation path, drive the crane to transport the grabbed container to the vicinity of the target placement platform; S4: Alignment and Unloading: The spatial geometric contours of the container bottom surface and the bearing plane of the target placement platform are analyzed; during the unloading process, the spatial geometric contours are dynamically aligned, and the lowering trajectory of the gripping unit is controlled until the bottom surface and the bearing plane are spatially coincident.

[0008] As a preferred embodiment of the port unmanned crane grasping pose visual positioning method of the present invention, the method is executed by analyzing image information collected by multiple sets of cameras fixedly installed on the crane. The multiple sets of cameras, depending on their function and installation location, specifically include: Positioning and calibration camera group: Fixedly installed at the four corners of the support columns on both sides of the crane, near the bottom, with the viewing angle facing the ground; used to identify the landing point of the ground marker during the positioning guidance process, and to provide image information for calculating the physical distance between the crane support column and the landing point of the ground marker; Global observation camera group: Located on a fixed structure at a high position of the crane, it provides three-dimensional observation of the work site from multiple different directions and heights; the acquired image information can be used to calculate the absolute coordinate system position of the target container and the target placement platform; Local alignment camera group: installed on the body of the gripping unit; after the gripping unit approaches the container, the local coordinate system of the gripping unit is established using image information in order to calculate the relative pose between the operating end of the gripping unit and the corner piece of the container; Opposing monitoring camera group: respectively fixed on the inner sidewalls of the support columns on both sides of the crane, with the shooting optical axes pointing towards each other, so as to jointly monitor the loading and unloading area located between the two columns; and provide image information of the coordinate position of the structure in the loading and unloading area during the loading process.

[0009] As a preferred embodiment of the port unmanned crane grasping pose visual positioning method of the present invention, the reference coordinate system specifically includes: the absolute coordinate system pre-stores known three-dimensional coordinates of all fixed facilities in the site, the landing points of the ground markers, and all landing areas; the relative coordinate system takes one corner of the bottom surface of the crane support column as the origin, and pre-stores the relative coordinates of the key structural points of the crane itself, as well as the pre-calibrated three-dimensional spatial pose information of each installed camera relative to the origin of the coordinate system.

[0010] As a preferred embodiment of the port unmanned crane grasping pose visual positioning method of the present invention, the following is provided: Before identifying the target pose in step S2 positioning and grasping, it is necessary to convert the coordinates in the relative coordinate system to the coordinates in the current absolute coordinate system. The specific process includes: The system identifies the base angle of the support column (serving as the origin of the relative coordinate system) and the nearest ground marker landing point in the image captured by the positioning and calibration camera group. Image analysis is used to statistically determine the pixel distance between the base angle of the support column and the ground marker landing point in two orthogonal directions. Using a preset scale corresponding to the fixed height of the positioning and calibration camera group, the coordinate difference between the two relative positions in the physical world, represented by the pixel distance, is calculated. This coordinate difference is then vector-synthesized with the known coordinates of the ground marker landing point pre-stored in the absolute coordinate system to calculate the initial position coordinates of the current origin of the relative coordinate system in the absolute coordinate system, establishing an initial mapping relationship between the relative and absolute coordinate systems.

[0011] As a preferred embodiment of the port unmanned crane grasping pose visual positioning method of the present invention, the process of identifying the initial pose of the target in step S2 specifically includes the following steps: from the continuous images collected by the global observation camera group, select valid images containing the target container and the landing point of nearby ground markers; extract the visible two-dimensional contour features of the target container from the valid images; then, based on the known pose of the global observation camera group in the absolute coordinate system, back-project the two-dimensional contour features to form a spatial ray beam that originates from the camera focus and passes through the two-dimensional contour features; determine the three-dimensional spatial position of the target container in the absolute coordinate system by calculating the intersection area of ​​multiple spatial ray beams in three-dimensional space; and perform secondary verification by performing the same calculation on the landing point of the nearby ground markers.

[0012] As a preferred embodiment of the port unmanned crane grasping pose visual positioning method of the present invention, the process of planning the motion path in step S2 specifically includes the following steps: planning multiple initial motion path sets connecting the starting point and the ending point based on the digital scene map pre-stored in the absolute coordinate system, the initial position coordinates, and the target initial pose; identifying temporary obstacles on the paths in the initial motion path set by analyzing the real-time images collected by the global observation camera group, and adding the additional time cost required for detours to the paths with identified obstacles; then querying the dispatch center data to determine whether the paths in the initial motion path set have spatial and temporal overlap with routes already occupied by other equipment, and adding the waiting time cost required for waiting or avoidance to the paths with overlap; calculating the total time spent on different paths and selecting the shortest time as the motion path, and uploading it to the dispatch center.

[0013] As a preferred embodiment of the port unmanned crane grasping posture visual positioning method of the present invention, the crane moves according to the movement path. During the movement, after the boundary of the target landing area is identified in the image collected by the positioning calibration camera group, the landing process of the crane begins: by calculating the distance between the corner point of the landing area appearing in the image and the bottom corner of the support column, and combining the absolute coordinate information of the target landing area and the relative coordinate information of the bottom corner of the crane support column, a control command is generated and the crane is guided to make fine adjustments to its position until all positioning calibration cameras can simultaneously identify the corresponding corner point of the landing area in their respective fields of view. After the landing is completed, a coordinate system transformation is performed to establish the transportation and landing mapping relationship between the relative coordinate system and the absolute coordinate system.

[0014] As a preferred embodiment of the port unmanned crane grasping pose visual positioning method of the present invention, wherein: the guidance and control in step S2 first involves a coarse alignment process, specifically including: By using a group of opposing surveillance cameras for collaborative observation, physical feature points on the target container are identified and extracted from the image. Based on the known pose of each camera in the relative coordinate system, the physical feature points in the image are back-projected to form a spatial straight line in three-dimensional space that starts from the focal point of the camera and passes through the physical feature points. By calculating the intersection or closest point of these spatial straight lines, the three-dimensional spatial coordinates of the physical feature points in the relative coordinate system are determined, and the position of the target container is obtained. The current coordinates of the gripping unit in the relative coordinate system are continuously determined using the same method, and the real-time positional deviation between the gripping unit and the target hovering point on the target container is calculated. If the real-time position deviation is greater than a preset rapid movement threshold, the grasping unit is driven to move at high speed to quickly approach the target hovering point; Once the real-time position deviation is less than the threshold, the position of the gripping unit is fine-tuned at a low speed until each local alignment camera group installed on the gripping unit can stably capture the corner image of the target container in its field of view.

[0015] As a preferred embodiment of the port unmanned crane grasping pose visual positioning method of the present invention, wherein: after completing the coarse alignment process, the guidance and control in step S2 will perform a fine alignment process, specifically including: Using one of the cameras in the local alignment camera group as the origin, a local coordinate system for the grasping unit is established based on the pre-calibrated relative positions between the cameras; During the grasping process, the local alignment camera group is used for collaborative observation to calculate the three-dimensional coordinates of the grasping unit's operating end and the target container corner opening in the local coordinate system in real time. Based on the relative pose deviation between the operating end and the corner opening, closed-loop control is performed to guide the operating end to align and move towards the corner opening. After the image from the local alignment camera group confirms that the operating end has passed through the corner opening, the operating end is further controlled to advance a preset distance along the insertion direction to ensure complete positioning. Then, the grasping unit is driven to perform a locking action. Throughout the entire process of the crane lifting the container until the handling is completed, the images from the localized aligned camera group will be continuously monitored to ensure the reliability of the grasping status.

[0016] As a preferred embodiment of the port unmanned crane grasping posture visual positioning method of the present invention, the process of moving the container to the vicinity of the target placement platform in step S3 is the same as the identification and movement placement process in step S2. After the placement is completed and the placement mapping relationship is generated, step S4 alignment and unloading is performed. The alignment and unloading further includes: calculating the geometric center coordinates of the container and the bearing plane of the target placement platform according to the spatial geometric contours of the parsed container bottom surface and the target placement platform; and during the lowering process, taking the vertical projection point of the geometric center of the target container bottom surface and the geometric center of the platform as the control target, and performing auxiliary correction on the lowering trajectory to ensure that the weight of the container can be evenly applied to the target placement platform, so as to reduce the risk of off-center loading or overturning caused by the center of gravity shift.

[0017] The beneficial effects of this invention are as follows: By constructing a dual-coordinate reference frame that links a fixed site coordinate system with a crane's moving spatial coordinate system, and deeply integrating visual image information from multiple groups and multi-view cameras, this invention achieves fully autonomous closed-loop control from path planning to precise grasping and unloading. A real-time visual self-calibration mechanism calculates the crane's precise pose in the site; and a multi-stage visual servoing strategy guides the grasping unit to complete high-precision operations. This invention, by constructing a low-cost, highly robust pure vision positioning and control system, significantly improves the accuracy of unmanned crane positioning and grasping, as well as the reliability of all-weather operation, and greatly enhances the system's autonomy, environmental adaptability, and economic efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 Flowchart of a visual positioning method for capturing the pose of an unmanned crane in a port.

[0020] Figure 2 This is a flowchart of the crane movement and positioning process.

[0021] Figure 3 Flowchart for capturing unit alignment.

[0022] Figure 4 Flowchart for confirming safe uninstallation. Detailed Implementation

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0025] Secondly, the term "one embodiment" or "example" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. The appearance of an embodiment in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that mutually excludes other embodiments.

[0026] Example 1 Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a visual positioning method for grasping pose of an unmanned crane in a port, comprising the following steps: S1. Establish a reference coordinate system: Establish the absolute coordinate system of the work site and the relative coordinate system of the crane structure as a reference frame for positioning and control; This method is implemented by analyzing image information captured by multiple sets of cameras fixedly installed on the crane. These multiple sets of cameras, depending on their function and installation location, specifically include: Positioning and calibration camera group: Fixedly installed at the four corners of the support columns on both sides of the crane, near the bottom, with the viewing angle facing the ground; used to identify the landing point of the ground marker during the positioning guidance process, and to provide image information for calculating the physical distance between the crane support column and the landing point of the ground marker; Global observation camera group: Located on a fixed structure high up on the crane, it provides three-dimensional observation of the work site from multiple different directions and heights; the acquired image information can be used to calculate the absolute coordinate system position of the target container and the target placement platform; Local alignment camera group: installed on the body of the gripping unit; after the gripping unit approaches the container, the local coordinate system of the gripping unit is established using image information in order to calculate the relative pose between the operating end of the gripping unit and the corner piece of the container. Opposing monitoring camera group: fixed on the inner sidewalls of the support columns on both sides of the crane, with their shooting optical axes pointing towards each other, to jointly monitor the loading and unloading area located between the two columns; providing image information of the coordinate position of the structure within the loading and unloading area during the loading process; The reference coordinate system and the absolute coordinate system pre-store the known three-dimensional coordinates of all fixed facilities, ground marker landing points and landing areas within the site; the relative coordinate system takes one corner of the bottom surface of the crane support column as the origin and pre-stores the relative coordinates of the key structural points of the crane itself, as well as the pre-calibrated three-dimensional spatial pose information of each installed camera relative to the origin.

[0027] Before the unmanned crane system is put into operation, a series of initial configuration and calibration work needs to be carried out to establish a precise and unified digital foundation for all subsequent automated operation processes.

[0028] The absolute coordinates of the work site are established by on-site surveyors using total stations or RTK-GPS equipment. First, a fixed corner point of the port terminal (e.g., a reference stake for a berth) is used as the origin of the absolute coordinate system, and the directions of the three orthogonal axes X, Y, and Z are defined (e.g., the X-axis along the quayline, the Y-axis perpendicular to the quayline pointing inland, and the Z-axis vertically upwards). Then, the precise 3D contours of all large fixed facilities (such as quay cranes, yard crane tracks, and buildings) that affect subsequent operations, the ground markers for crane positioning (e.g., specific crosshairs, T-shaped marks, or crane positioning areas resembling parking spaces painted on the ground), and the precise corner coordinates of all standard container stacking areas and vehicle parking areas are measured and entered into the system database, creating a high-precision digital scene map.

[0029] The origin of the crane's relative coordinate system is set at the vertex of a bottom corner of one of the crane's support columns (e.g., the bottom corner of the right front support column facing the land). Then, technicians will accurately measure and record the three-dimensional coordinates of all key structural points on the crane (e.g., the bottom corner vertices of the other support columns, the geometric center of the crane's crossbeam, and the coordinates of the return point of the grabbing unit in the retracted state) relative to this origin.

[0030] Then, precise internal and external parameter calibration and pose calibration are performed on the cameras installed at various locations on the crane. Specifically, for the four positioning calibration cameras fixed at the base of the support column, the three-dimensional coordinates (X, Y, Z) of their optical centers relative to the origin of the relative coordinate system and the attitude angles (pitch, yaw, roll) of their optical axes relative to the coordinate system axes must be accurately measured. This process ensures that the system can accurately calculate the three-dimensional spatial relationship of any two-dimensional pixel seen by any camera in the image to its position relative to the crane's coordinate system. The calibration information of all cameras, along with the coordinates of the crane's own structural points, is then embedded in the crane's control system.

[0031] Thus, a globally unified, static absolute coordinate system and a relative coordinate system that moves with the vehicle but has a constant internal structure have been constructed. Together, they form the reference framework for the entire visual positioning and control method.

[0032] S2. Positioning and Grabbing: Identify the initial pose of the target container in the absolute coordinate system, and plan the motion path by combining the initial pose of the target with the initial position coordinates of the crane; drive the crane to move along the motion path and complete the placement; in the grabbing stage, use the absolute coordinate system and the relative coordinate system to guide and control the grabbing unit until the grabbing is identified as complete. Before identifying the target pose during positioning and grasping, it is necessary to convert the coordinates in the relative coordinate system to the coordinates in the current absolute coordinate system. In the image captured by the positioning calibration camera group, the bottom angle of the support column, which serves as the origin of the relative coordinate system, and the nearest ground marker landing point are identified. Through image analysis, the pixel distance between the bottom angle of the support column and the ground marker landing point in the image and in two orthogonal directions are calculated. Using the preset scale corresponding to the fixed height of the positioning calibration camera group, the coordinate difference between the two relative positions in the physical world, represented by the pixel distance, is calculated. The coordinate difference is then vector-synthesized with the known coordinates of the ground marker landing point pre-stored in the absolute coordinate system to calculate the initial position coordinates of the current relative coordinate system origin in the absolute coordinate system, thus establishing the initial mapping relationship between the relative coordinate system and the absolute coordinate system. The initial pose of the target is identified by filtering out valid images containing the target container and the landing points of nearby ground markers from continuous images acquired by the global observation camera group. The visible two-dimensional contour features of the target container are extracted from the valid images. Then, based on the known pose of the global observation camera group in the absolute coordinate system, the two-dimensional contour features are back-projected to form a spatial ray beam that originates from the camera focus and passes through the two-dimensional contour features. By calculating the intersection area of ​​multiple spatial ray beams in three-dimensional space, the three-dimensional spatial position of the target container in the absolute coordinate system is determined. The same calculation is performed on the landing points of nearby ground markers for secondary verification. The system plans motion paths based on a pre-stored digital scene map in the absolute coordinate system, initial position coordinates, and target initial pose, creating multiple sets of initial motion paths connecting the start and end points. By analyzing real-time images captured by the global observation camera group, temporary obstacles on the paths in the initial motion path set are identified, adding additional time costs for detours to the paths with identified obstacles. Then, data from the dispatch center is queried to determine if the paths in the initial motion path set overlap spatially and temporally with routes already occupied by other devices, adding waiting time costs for waiting or avoidance to the paths with overlaps. After calculating the total time spent on different paths, the path with the shortest time is selected as the motion path and uploaded to the dispatch center. The crane moves according to the motion path. During the movement, after the boundary of the target landing area is identified in the images collected by the positioning and calibration camera group, the landing process of the crane begins. By calculating the distance between the corner point of the landing area and the bottom corner of the support column that should be supported in the image, and combining the absolute coordinate information of the target landing area and the relative coordinate information of the bottom corner of the crane support column, control commands are generated and the crane is guided to make fine adjustments to its position until all positioning and calibration cameras can simultaneously identify the corresponding corner point of the landing area in their respective fields of view. After the landing is completed, a coordinate system transformation is performed to establish the transportation and landing mapping relationship between the relative coordinate system and the absolute coordinate system. The guidance and control process begins with a coarse alignment process. This involves collaborative observation using opposing surveillance cameras to identify and extract physical feature points on the target container from the images. Based on the known pose of each camera in the relative coordinate system, the physical feature points in the images are projected back to form a straight line in three-dimensional space that originates from the camera's focal point and passes through the physical feature points. By calculating the intersection or closest point of these straight lines, the three-dimensional spatial coordinates of the physical feature points in the relative coordinate system are determined, thus obtaining the position of the target container. The current coordinates of the gripping unit in the relative coordinate system are continuously determined using the same method, and the real-time positional deviation between the gripping unit and the target hovering point on the target container is calculated. If the real-time position deviation is greater than a preset fast movement threshold, the grasping unit is driven to move at high speed to quickly approach the target hovering point. Once the real-time position deviation is less than the threshold, the position of the gripping unit is fine-tuned at a low speed until each local alignment camera group installed on the gripping unit can stably capture the corner image of the target container in its field of view. After the coarse alignment process, the guidance and control will carry out a fine alignment process, taking one of the cameras in the local alignment camera group as the coordinate origin, and establishing a local coordinate system of the grasping unit based on the pre-calibrated relative positions between the cameras; During the grasping process, a local alignment camera group is used for collaborative observation to calculate the three-dimensional coordinates of the grasping unit's operating end and the corner opening of the target container in the local coordinate system in real time. Closed-loop control is performed based on the relative pose deviation between the operating end and the corner opening to guide the operating end to align and move towards the corner opening. After the image from the local alignment camera group confirms that the operating end has passed through the corner opening, the operating end is controlled to advance a preset distance along the insertion direction to ensure complete positioning, and then the grasping unit is driven to perform a locking action.

[0033] Upon receiving a new capture task instruction, the crane itself is first positioned to determine the starting point for path planning. The process begins by analyzing the ground image stream acquired by the positioning calibration cameras installed at the base corners of the crane's four support columns. The position of the support column base corner representing the origin of the relative coordinate system is then identified from the image stream, and the nearest ground marker landing point within the field of view is located. Image analysis measures the horizontal and vertical pixel distances between the support column base corner and the marker landing point on the image. Using a pre-calibrated precise scale (e.g., 0.5 cm per pixel for a height of 5 meters), the actual distance difference between the two is calculated. Vector synthesis is then used to calculate the current absolute coordinates of the crane's relative coordinate system origin. Adding these absolute coordinates to all points in the relative coordinate system transforms them into points in the absolute coordinate system.

[0034] in, It is the position vector of the crane relative to the origin of the coordinate system in the absolute coordinate system; It is the known position vector of the ground marker landing point in the absolute coordinate system; It is a vector of actual coordinate differences calculated based on pixel distance.

[0035] Then, from the video stream of the global observation camera group, image frames that clearly show both the target container and the landing point of a nearby ground marker are selected, and the visible top outline of the target container is extracted. Although modern cameras often use multiple chips to collaboratively image, an entire lens module can obtain its equivalent lens optical center by connecting an equivalent imaging point to the actual object's imaging position. All imaging is achieved by light emitted from the object passing through this equivalent optical center. In other words, all positions on the straight line connecting a point outside the lens and the imaging area to the optical center will eventually be imaged at this point. The absolute coordinates of each global observation camera after transformation are known, so the position of this straight line in the absolute coordinate system is also determined. The same structure will necessarily be in different positions in different images. Connecting it to the optical center of the corresponding camera can determine multiple light paths. By calculating the intersection area of ​​the light paths, the absolute coordinate position of the target structure in the absolute coordinate system can be obtained. The same process is then performed on the markers near the target container, and the coordinates are compared with those stored internally. Since the purpose of this process is to roughly determine a movement path, the fine-tuning of the landing point is the content of subsequent steps, and only the deviation needs to be within a controllable range.

[0036] in, It represents the position coordinates of a feature point on the target container in the absolute coordinate system; it represents the position coordinates of the feature point on the target container in the absolute coordinate system. The optical center coordinates of each camera The spatial straight line jointly determined by its image and feature points; Representing a spatial point to the straight line The coordinates of the feature point are the coordinates of the point whose vertical distance to all lines is the shortest.

[0037] After determining the approximate locations of the crane and the target container, multiple paths are planned using a digital scene map. Obstacles on all possible paths are then identified using a global observation camera system. Paths containing obstacles are then modified to avoid them. Next, an application is submitted to the port's central dispatch center to obtain all currently uploaded and saved routes for the current period. Since these routes are already submitted and saved, subsequent cranes must pass through intersection points sequentially to avoid management confusion. This necessitates adding additional waiting time costs, including braking, waiting, and restarting, for paths with intersection points. After modification, the time required for different routes is calculated, and the shortest route is selected as the optimal path and uploaded to the dispatch center to lock the right-of-way, preparing to drive the crane to move.

[0038] in, It is the base travel time for the route; It is the sum of the additional time costs incurred in avoiding temporary obstacles; It is the sum of waiting time costs caused by overlapping paths with other devices; It represents the total time spent on the path.

[0039] During the crane's movement, the positioning and calibration camera group verifies the path based on the coordinates of the captured positioning points to prevent deviations. Once the crane reaches a position near the target landing point, it slows down until the marker points near the target landing point appear in the image, entering a fine-tuning phase. First, the relative coordinates of the support column angles captured by the cameras that obtained the marker point image information, as well as the absolute coordinates of the marker points and the target landing area, are confirmed. This allows for the calculation of the absolute coordinates of the geometric center of the crane's bottom surface and the geometric center of the target area. Using the coincidence of these geometric centers as the target, the crane is controlled to move at a low speed for fine-tuning during movement. Only after all positioning and calibration cameras can recognize the positioning markers in the target area does the crane stop its landing movement and re-perform the absolute coordinate transformation of the origin, establishing a new mapping relationship.

[0040] Then, using the images of the loading and unloading area obtained by the opposing monitoring camera group, a more accurate absolute coordinate position of the target container is obtained. The coordinates of its top corner fitting and the default return coordinates of the crane grabbing unit are used to control the grabbing unit to fall. First, it moves at high speed to quickly reach the vicinity of the top of the container, and then uses the image information to move at a low speed and adjust until the corner fitting opening is identified in multiple images of the local aligned camera group, and then the movement stops.

[0041] Each grasping unit requires three or more cameras at different angles to obtain a sufficiently three-dimensional local spatial image. Since the units of relative and absolute coordinate systems are too large compared to the local range of motion of the grasping unit, and the local alignment camera group cannot effectively obtain usable reference points during the movement of the grasping unit, it is necessary to select one camera as the origin of the local coordinate system for this corner piece, and determine the corresponding coordinate positions of the other cameras operating in the local area of ​​this corner piece in the local coordinate system. Then, using the same calculation principle, the relative coordinates between the end of the grasping unit and the corner piece opening in the regional coordinate system are calculated. The grasping unit is then driven into the corner piece opening and pushed forward a certain distance before locking.

[0042] S3, Target Transfer: Identify the position of the target placement platform in the absolute coordinate system and plan the transport path, then drive the crane to move the grabbed container to the vicinity of the target placement platform; The process of moving the object to the vicinity of the target placement platform is identified as the same as the relocation and placement process. Throughout the entire process of the crane lifting the container until the handling is completed, the images from the localized camera array will be continuously monitored to ensure the reliability of the capture status.

[0043] The entire target transfer process is logically identical to identifying the target container and completing its placement; only the objects and targets being processed have changed.

[0044] After the acquisition is complete, the target placement platform in the task command is identified from the image data. Its absolute coordinates are then calculated and verified using nearby markers. Finally, the path is planned and the platform is placed. Compared to no-load operation, the drive power and acceleration / deceleration curves need to be adjusted appropriately during the handling process to ensure smooth operation.

[0045] Throughout the entire process, from the moment the crane picks up the container and begins transporting it until it is finally positioned on the target platform, a group of localized cameras mounted on the gripping unit continuously monitors the engagement status of the rotary lock head and the container corner fittings. If any signs of unlocking or loosening due to violent shaking or accidental impact are detected, an emergency brake is immediately triggered and an alarm is sounded, maximizing safety during the heavy-duty handling process.

[0046] S4. Alignment and Unloading: The spatial geometric contours of the container bottom and the bearing plane of the target placement platform are analyzed; during the unloading process, the spatial geometric contours are dynamically aligned, and the lowering trajectory of the grabbing unit is controlled until the bottom and the bearing plane are spatially coincident. After the container is positioned and the positioning mapping relationship is generated, the unloading is performed. Based on the spatial geometric contours of the bottom surface of the container and the bearing plane of the target positioning platform, the geometric center coordinates of each are calculated. During the lowering process, the vertical projection point of the geometric center of the bottom surface of the target container coincides with the geometric center of the platform as the control target. The lowering trajectory is assisted and corrected to ensure that the weight of the container is evenly applied to the target positioning platform, thereby reducing the risk of off-center loading or overturning that may be caused by the center of gravity shift.

[0047] After placement, the relative coordinates of the four corners of the container's bottom surface and the four corners of the target placement platform within the loading / unloading area are calculated using a group of opposing monitoring cameras. The geometric centers of both planes are then calculated. During the lowering process, the calculated coordinates of the four corners of the bottom surface are used to assess the container's descent attitude, allowing for timely adjustments in case of deviation. While maintaining the container's attitude, the vertical projection of the container's bottom surface center coordinates is ensured to coincide with the platform's center coordinates as much as possible. This ensures that when the container finally contacts the platform, its weight is evenly and stably distributed in the central area of ​​the platform, effectively reducing the risk of rollover or structural damage due to severe uneven loading.

[0048] During the lowering process, the gap between the container's bottom and the platform is monitored in real time. Once the gap disappears, it indicates that the two are physically aligned. However, at this point, most of the container's weight is still borne by the taut lifting ropes. Lowering continues until the lifting ropes sag naturally with a slight curve. Then, the rotary lock head of the gripping unit rotates 90° in the opposite direction to unlock, and the gripping unit is lifted to complete the unloading task. The entire unmanned crane then enters standby mode, ready to receive the next round of work instructions.

[0049] In summary, this invention constructs a dual-coordinate reference frame that links a fixed site coordinate system with a crane's moving spatial coordinate system. It deeply integrates visual image information from multiple groups and multi-view cameras to achieve fully autonomous closed-loop control from path planning to precise grasping and unloading. A real-time visual self-calibration mechanism calculates the crane's precise pose within the site, and a multi-stage visual servo strategy guides the grasping unit to complete high-precision operations. By constructing a low-cost, highly robust pure vision-based positioning and control system, this invention significantly improves the accuracy of unmanned crane positioning and grasping, enhances the reliability of all-weather operation, and greatly strengthens the system's autonomy, environmental adaptability, and economic efficiency.

[0050] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. The grabbed container is moved to the vicinity of the target landing platform; S4: Positioning unloading: the spatial geometric profile of the container bottom surface and the bearing plane of the target landing platform is analyzed; during the unloading process, dynamic alignment is performed on the spatial geometric profile to control the lowering trajectory of the grabbing unit until the bottom surface and the bearing plane are spatially coincident.

2. The grabbed container is moved to the vicinity of the target landing platform; S4: Positioning unloading: the spatial geometric profile of the container bottom surface and the bearing plane of the target landing platform is analyzed; during the unloading process, dynamic alignment is performed on the spatial geometric profile to control the lowering trajectory of the grabbing unit until the bottom surface and the bearing plane are spatially coincident.

3. The vision-based positioning method of claim 2, wherein, The reference coordinate system specifically includes: pre-stored known three-dimensional coordinates of all fixed facilities in the site, the ground marker landing point and all landing areas in the absolute coordinate system; the relative coordinate system has the relative coordinates of the key structure points of the crane itself and the pre-calibrated three-dimensional space pose information of each installed camera relative to the coordinate origin, with one corner of the bottom surface of the crane support column as the coordinate origin.

4. The vision-based positioning method of a port unmanned crane's grasping pose according to claim 3, characterized in that, Before identifying the target pose in the positioning and grabbing of step S2, the coordinates in the relative coordinate system need to be converted into the coordinates in the current absolute coordinate system, and the specific process includes: The support column bottom corner and the nearest ground marker landing point in the image captured by the positioning calibration camera group are identified as the origin of the relative coordinate system, and the pixel distances of the support column bottom corner and the ground marker landing point in two orthogonal directions in the image are calculated through image analysis. The coordinate difference of the relative positions of the two in the physical world is calculated using the preset scale corresponding to the fixed height of the positioning calibration camera group. The coordinate difference and the known coordinates of the ground marker landing point in the absolute coordinate system are vector synthesized to calculate the initial position coordinates of the relative coordinate system origin in the absolute coordinate system, and the initial mapping relationship between the relative coordinate system and the absolute coordinate system is established.

5. The vision-based positioning method of claim 4, wherein, The process of identifying the target initial pose in step S2 specifically includes the following steps: filtering out the effective images containing the target container and the nearby ground marker landing point from the continuous images collected by the global observation camera group, extracting the visible two-dimensional contour features of the target container from the effective images, and then inversely projecting the two-dimensional contour features according to the known pose of the global observation camera group in the absolute coordinate system to form a spatial ray bundle from the camera focal point through the two-dimensional contour features. The three-dimensional space position of the target container in the absolute coordinate system is determined by calculating the intersection region of multiple spatial ray bundles in three-dimensional space; and the same calculation is performed on the nearby ground marker landing point for secondary verification.

6. The vision-based positioning method of claim 5, wherein, The process of planning the motion path in step S2 specifically includes the following steps: planning a plurality of initial motion paths connecting the starting point and the ending point according to the pre-stored digital scene map in the absolute coordinate system, the initial position coordinates and the target initial pose; identifying temporary obstacles on the paths in the initial motion path set by analyzing real-time images collected by the global observation camera group, and adding additional time cost required for detouring to the paths where obstacles are identified; then querying the dispatch center data to determine whether the paths in the initial motion path set coincide with routes occupied by other devices in space and time, and adding waiting time cost required for waiting or avoiding to the paths where coincidences exist; and selecting the path with the shortest time consumption as the motion path after calculating the total time consumption of different paths, and uploading the path to the dispatch center.

7. The vision-based positioning method of claim 6, wherein, After the boundaries of the target landing area are identified in the images collected by the positioning calibration camera group during the movement of the crane according to the motion path, the landing process of the crane starts: the distance between the landing area corner points appearing in the images and the bottom corners of the support columns is calculated, the absolute coordinate information of the target landing area and the relative coordinate information of the bottom corners of the support columns of the crane are combined, control instructions are generated and the crane is guided for position fine-tuning until all the positioning calibration cameras can simultaneously identify the corresponding landing area corner points in their respective fields of view, and after landing, a coordinate system transformation is performed to establish the carrying and landing mapping relationship between the relative coordinate system and the absolute coordinate system.

8. The vision-based positioning method of a port unmanned crane's grasping pose according to claim 3, characterized in that, The guiding and control in step S2 first performs a coarse alignment process, which specifically includes: The target container is identified and the physical feature points thereon are extracted in the images by using the opposite monitoring camera group for collaborative observation; the physical feature points in the images are inversely projected according to the known poses of each camera in the relative coordinate system to form a spatial straight line in three-dimensional space from the camera focal point to the physical feature points; the three-dimensional space coordinates of the physical feature points in the relative coordinate system are determined by calculating the intersection points or the closest points of these spatial straight lines to obtain the position of the target container; The current coordinates of the grabbing unit in the relative coordinate system are continuously determined in the same way, and the real-time positional deviation between the grabbing unit and the target hovering point on the target container is calculated; In the case where the real-time positional deviation is greater than a preset fast movement threshold, the grabbing unit is driven to move at high speed to quickly approach the target hovering point; When the real-time positional deviation is less than the threshold, the position of the grabbing unit is fine-tuned at a lower speed until each local alignment camera group mounted on the grabbing unit can stably capture the corner piece images of the target container in its field of view.

9. The vision-based positioning method of claim 8, wherein, After the coarse alignment process is completed, the guiding and control in step S2 performs a fine alignment process, which specifically includes: Taking one camera in the local alignment camera group as the coordinate origin, a local coordinate system of the grabbing unit is established based on the relative positions of the cameras previously calibrated. During the grabbing process, the local alignment camera group is used for cooperative observation to respectively calculate the three-dimensional coordinates of the operation end of the grabbing unit and the opening of the target container corner fitting in the local coordinate system in real time; closed-loop control is performed based on the relative pose deviation between the operation end and the opening of the corner fitting to guide the operation end to align and move to the opening of the corner fitting; after the image of the local alignment camera group confirms that the operation end has passed through the opening of the corner fitting, the operation end is controlled to continue advancing in the insertion direction by a preset distance to ensure complete positioning, and then the grabbing unit is driven to perform a locking action; During the whole process of lifting the container by the crane until the handling is completed, the image of the local alignment camera group is continuously monitored to ensure the reliability of the grabbing state.

10. The vision-based positioning method of claim 7, wherein, The process of handling the container to the vicinity of the target landing platform in step S3 is the same as the identification and moving landing process in step S2, and after the landing is completed and the placement landing mapping relationship is generated, the alignment unloading in step S4 is performed, which further includes: calculating the geometric center coordinates of the container bottom surface and the target landing platform bearing plane respectively according to the analyzed spatial geometric contour; and in the lowering process, the vertical projection point of the geometric center of the target container bottom surface and the geometric center of the platform are kept coincident as the control target to assist in correcting the lowering track, so as to ensure that the weight of the container can be evenly applied on the target landing platform, thereby reducing the risk of unbalanced load or overturning caused by the deviation of the center of gravity.

Images