Pose alignment method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to a pose alignment method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: acquiring image data of a container at multiple visual angles, and inputting the image data into a pre-trained rotating target detection neural network model; utilizing the rotating target detection neural network model to respectively detect a rotating bounding box and a semantic key point in the image data of the container at each view angle; carrying out binocular stereo matching on the semantic key points under the multiple visual angles to obtain a key point matching result, and solving by utilizing the key point matching result in combination with a preset gantry crane kinematics constraint and the rotation bounding box to obtain a container pose; and a motion control instruction is generated according to the position and posture of the container and the current position and posture of the lifting appliance, and the motion control instruction is executed to adjust the position and posture of the lifting appliance to be aligned with the position and posture of the container. By adopting the method, the robustness and the pose alignment precision can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular, to a pose alignment method and device, a computer device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] How to use the railway yard portal crane to automatically grasp the container is the key to improving the efficiency and safety of the railway yard container loading and unloading operation. At present, the main methods are based on laser scanning and contour matching, and based on traditional visual positioning to identify the position of the container, and then control the alignment of the spreader of the portal crane and the container, and automatically grasp the container by the spreader.

[0003] However, when using the method based on laser scanning and contour matching, there are problems of weak environmental interference resistance and poor robustness. For example, rain, snow, fog, sand and other bad weather can strongly scatter or absorb laser beams, causing point cloud data missing, noise increasing and even complete failure. The surface emission characteristics have a great influence, and the surface material (such as rust, oil stains, water accumulation) and color (dark color) of the container can cause a huge difference in laser echo intensity. Strong reflection areas may produce "ghost" or saturation, while weak reflection areas have weak signals and are difficult to detect, resulting in incomplete or deformed contour extraction. Strong sunlight directly hitting the laser receiver can drown out the weak laser signal, reducing the signal-to-noise ratio.

[0004] In addition, due to the huge amount of calculation of large-scale point cloud data filtering, segmentation, registration (ICP algorithm) and other operations, the point cloud processing takes a long time. At the same time, the matching process between the real-time scanned contour and the preset container model (especially when there is occlusion or deformation) is an iterative optimization process with high computer complexity, and there is a problem of slow matching process. When the spreader moves at high speed, it requires extremely high processing speed to ensure real-time guidance, which requires high processor performance.

[0005] When using the method based on traditional visual positioning, it is difficult to accurately estimate the attitude parameters such as the pitch angle and roll angle of the container, which can easily lead to low alignment accuracy between the spreader and the container. And in complex lighting conditions such as rain, fog and backlight, the image acquisition accuracy and image processing accuracy of traditional visual processing will be affected, leading to deviation in container positioning.

[0006] Therefore, there is an urgent need for a pose alignment method with stronger robustness and higher accuracy in the portal crane scene to realize accurate alignment of the spreader and the container. SUMMARY

[0007] Therefore, it is necessary to provide a high-precision pose alignment method and device with improved robustness, a computer device, a computer readable storage medium, and a computer program product to solve the technical problems.

[0008] In a first aspect, the present application provides a pose alignment method, comprising:

[0009] obtaining image data of a container under multiple views, and inputting the image data into a pre-trained rotating target detection neural network model;

[0010] detecting a rotating bounding box and semantic key points in the image data of the container under each view by using the rotating target detection neural network model;

[0011] performing binocular stereo matching on the semantic key points under multiple views to obtain a key point matching result, and solving a container pose by using the key point matching result in combination with a preset gantry crane kinematic constraint and the rotating bounding box;

[0012] generating a motion control instruction according to the container pose and a current spreader pose, and adjusting the spreader pose to align with the container pose by executing the motion control instruction.

[0013] In one of the example embodiments, before the image data is input into the pre-trained rotating target detection neural network model, the method further comprises:

[0014] obtaining image acquisition device parameters corresponding to each view, wherein the image acquisition device parameters include intrinsic parameters and extrinsic parameters;

[0015] performing intrinsic calibration on the image acquisition device corresponding to each view according to the intrinsic parameters;

[0016] performing binocular extrinsic calibration on the image acquisition devices under different views according to the extrinsic parameters under different views;

[0017] establishing a conversion relationship between a three-dimensional coordinate system of the image acquisition device under each view and a preset gantry crane world coordinate system.

[0018] In one of the example embodiments, the binocular stereo matching on the semantic key points under multiple views to obtain a key point matching result, and solving a container pose by using the key point matching result in combination with a preset gantry crane kinematic constraint and the rotating bounding box, comprises:

[0019] mapping the key point matching result and the rotating bounding box to the gantry crane world coordinate system by using the conversion relationship, the extrinsic parameters, and the intrinsic parameters, and solving in combination with the gantry crane kinematic constraint to obtain the container pose.

[0020] In one of the example embodiments, the generating the motion control instruction according to the container pose and the current spreader pose, and executing the motion control instruction to adjust the spreader pose to align with the container pose, comprises:

[0021] determining a translation vector and a rotation angle of the spreader pose relative to the container pose;

[0022] generating the motion control instruction according to the translation vector and the rotation angle;

[0023] issuing the motion control instruction to the portal crane, the trolley, and the spreader slewing mechanism to execute to adjust the spreader pose to align with the container pose.

[0024] In one of the example embodiments, the method further comprises:

[0025] acquiring lifting image data collected for the spreader and the container during lifting of the spreader;

[0026] calculating a distance between the spreader and the container according to the lifting image data, the distance being used to adjust lifting motion parameters of the spreader.

[0027] In one of the example embodiments, the method further comprises:

[0028] when the distance reaches a preset distance threshold, performing target detection on the lifting image data to obtain a buckle position of the container;

[0029] adjusting the current spreader pose to align with the current container pose according to the buckle position.

[0030] In a second aspect, the present application further provides a pose alignment device, comprising:

[0031] an image acquisition module configured to acquire image data of a container under multiple perspectives, and input the image data to a pre-trained rotation target detection neural network model;

[0032] a target detection module configured to detect a rotation bounding box and semantic key points in the image data of the container under each perspective by using the rotation target detection neural network model;

[0033] a pose estimation module configured to perform binocular stereo matching on the semantic key points under multiple perspectives to obtain key point matching results, and solve a container pose by using the key point matching results in combination with a preset portal crane kinematic constraint and the rotation bounding box;

[0034] The pose alignment module is configured to generate a motion control instruction according to the container pose and a current crane pose, and execute the motion control instruction to adjust the crane pose to align with the container pose.

[0035] In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the pose alignment method in any one of the embodiments of the first aspect.

[0036] In a fourth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the pose alignment method in any one of the embodiments of the first aspect.

[0037] In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program. The computer program is executed by a processor to implement the pose alignment method in any one of the embodiments of the first aspect.

[0038] The pose alignment method, device, computer device, computer readable storage medium and computer program product can obtain image data of a container in multiple perspectives, input the image data into a pre-trained rotating target detection neural network model, detect a rotating bounding box and semantic key points in the image data of the container in each perspective by using the rotating target detection neural network model, perform binocular stereo matching on the semantic key points in the multiple perspectives to obtain a key point matching result, solve the container pose by using the key point matching result in combination with a preset gantry crane kinematic constraint and the rotating bounding box, generate a motion control instruction according to the container pose and a current crane pose, and execute the motion control instruction to adjust the crane pose to align with the container pose. The method can increase data richness by using multi-perspective image data, obtain depth information by using binocular stereo matching, and narrow the solution space by introducing the gantry crane kinematic constraint, thereby improving the accuracy of container pose estimation and the robustness of the pose estimation algorithm. In turn, the container and the crane can be aligned with high precision by using the container pose and the crane pose with higher accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0040] Figure 1An application environment diagram of a pose alignment method in an embodiment;

[0041] Figure 2 A flowchart of a pose alignment method in an embodiment;

[0042] Figure 3 A flowchart of a device calibration step in an embodiment;

[0043] Figure 4 A flowchart of a motion control instruction generation step in an embodiment;

[0044] Figure 5 A flowchart of a pose alignment method in another embodiment;

[0045] Figure 6 A structural block diagram of a pose alignment device 600 in an embodiment;

[0046] Figure 7 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0047] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0048] It should be noted that the data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data need to comply with relevant regulations.

[0049] The pose alignment method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 The container 10 and the portal crane 20. The portal crane 20 (i.e. a portal crane) can grab the container 10 through the spreader 21. The spreader 21 is provided with multiple image acquisition devices 22 at both ends, which can acquire images of the container 10 from multiple perspectives. The image acquisition device 22 can be a camera, an image sensor or the like. The multiple image acquisition devices 22 with different perspectives can constitute a multi-view vision system. Figure 1 Only one way of arranging four image acquisition devices 22 at both ends of the spreader 21 is shown. Those skilled in the art can understand that the arrangement position or the arrangement number of the image acquisition device 22 can be flexibly adjusted according to the accuracy requirement in the actual portal crane scene.

[0050] Optionally, in an exemplary embodiment, a server, a terminal or a system composed of a server and a terminal can be employed to control the lifting of the spreader 21 and process the image data of the container 10 collected by the image collection device 22. As shown in Figure 2 A pose alignment method is provided, which is described by taking the server as an example and includes the following steps S202 to S208. Among them:

[0051] In step S202, image data of the container under multiple viewing angles is obtained, and the image data is input into a pre-trained rotation target detection neural network model.

[0052] The rotation target detection neural network model can be a model pre-trained by using rotation target samples to pre-train a target detection model (such as a YOLO system model). For example, in the model pre-training stage, image samples of the container 10 under different viewing angles can be collected, and rotation bounding box annotation information and semantic key point annotation information can be added in the image samples under each viewing angle as rotation target samples. The rotation target samples are divided into a training set and a test set. The initial target detection model is trained by using the training set, and the trained target detection model is used to predict the rotation bounding box information and the semantic key point information in the test set. The model parameters of the target detection model are adjusted according to the loss value between the prediction result of the trained target detection model and the annotation information of the test set until the adjusted loss value meets a preset condition or the preset number of iterations is reached, and the training is stopped. The target detection model at this time is used as the pre-trained rotation target detection neural network model used in the application stage.

[0053] Exemplarily, the server can obtain image data of the container 10 under multiple viewing angles from multiple image collection devices 22, and input the image data under each viewing angle into the pre-trained rotation target detection neural network model. Optionally, in some embodiments, the image data can be synchronously collected by performing an image collection operation on the container 10 by each image collection device 22 every preset collection period. Alternatively, in some other embodiments, the image data can also be data at the same time extracted from video stream data obtained by video recording the container 10 by each image collection device 22.

[0054] In step S204, the rotation target detection neural network model is used to detect the rotation bounding box and the semantic key point in the image data of the container under each viewing angle, respectively.

[0055] In the portal crane scenario of the present application, the container 10 is prone to rotation during the lifting process by the spreader 21 due to factors such as gravity, air resistance, wind speed, etc. The rotated bounding box is a bounding box that can adapt to the direction of the target object and rotate at any angle to more compactly wrap the target object. Therefore, the embodiments of the present application use the rotated bounding box target detection method to identify the detection box in which the container 10 is located in the image data.

[0056] The semantic key points can be used to represent key points with obvious features in the container 10, such as the four corner points at the top or bottom of the container 10, etc.

[0057] Exemplarily, the server can use the rotated target detection neural network model to perform rotated target detection on the image data of the container 10 at each view based on the pre-trained model parameters, to obtain the rotated bounding box and the semantic key points in the image data of the container 10 at each view.

[0058] In step S206, the semantic key points at multiple views are subjected to binocular stereo matching to obtain key point matching results, and the container pose is solved by using the key point matching results in combination with the preset portal crane kinematic constraints and the rotated bounding box.

[0059] The key point matching results can include point pairs composed of key point positions of the same semantic key point at different views.

[0060] The portal crane kinematic constraints can include constraints such as the activity angle, the motion speed limit value, and the anti-collision safety distance of each part mechanism in the portal crane.

[0061] Exemplarily, the server can perform semantic matching on the semantic key points at each pair of views, and in the case of consistent semantic matching, a binocular stereo matching correspondence relationship between the semantic key points at different views is established. For example, in the case where the semantic key point a at view 1 and the semantic key point a' at view 2 are both corner points A of the container 10, a correspondence relationship between the semantic key point a and the semantic key point a' can be established, forming the key point matching results of the corner point A at view 1 and view 2.

[0062] The server can use the key point matching results at multiple views to estimate the pose of the container 10, and at the same time, in the pose estimation process, the preset portal crane kinematic constraints and the rotated bounding box are introduced as constraint conditions for solving, to obtain the optimal six-degree-of-freedom pose of the container 10 as the container pose.

[0063] In step S208, a motion control instruction is generated according to the container pose and the current spreader pose, and the spreader pose is adjusted to align with the container pose by executing the motion control instruction.

[0064] For example, the server can obtain the current spreader pose of spreader 21. It compares the spreader pose with the container pose and generates motion control commands based on the comparison result. The motion control commands are executed to adjust the spreader pose and container pose to align, ensuring that the spreader 21 maintains the same horizontal position and rotation angle as the container 10 after execution, thus facilitating subsequent precise grasping.

[0065] In the above pose alignment method, image data of the container from multiple perspectives is acquired and input into a pre-trained rotating target detection neural network model. The rotating target detection neural network model is used to detect the rotating bounding box and semantic key points of the container in the image data of each perspective. Binocular stereo matching is performed on the semantic key points from multiple perspectives to obtain key point matching results. The container pose is obtained by combining the key point matching results with preset gantry crane kinematic constraints and rotating bounding boxes. Motion control commands are generated based on the container pose and the current spreader pose. The motion control commands are executed to adjust the spreader pose and the container pose for alignment. This method can increase the data richness by using multi-view image data, obtain depth information by using binocular stereo matching, and reduce the solution space by introducing gantry crane kinematic constraints, thereby improving the accuracy of container pose estimation and the robustness of the pose estimation algorithm. Furthermore, high-precision alignment of the container and spreader can be achieved using higher-precision container pose and spreader pose.

[0066] In one exemplary embodiment, such as Figure 3 As shown, the pose alignment method provided in this application may further include the following steps S302 to S308. Wherein:

[0067] Step S302: Obtain the image acquisition device parameters corresponding to each viewpoint. The image acquisition device parameters include intrinsic parameters and extrinsic parameters.

[0068] Step S304: Perform intrinsic parameter calibration operation on the image acquisition device corresponding to each viewpoint according to the intrinsic parameter parameters.

[0069] Step S306: Perform binocular extrinsic parameter calibration on the image acquisition device under different viewing angles according to the extrinsic parameters under different viewing angles.

[0070] For example, the server can obtain the image acquisition device parameters of the image acquisition device 22 corresponding to each viewpoint. Based on the intrinsic parameters of the image acquisition device 22 corresponding to each viewpoint, an intrinsic parameter calibration operation is performed on the image acquisition device 22 under each viewpoint. Based on the extrinsic parameters of the image acquisition devices 22 under each pair of viewpoints, a binocular extrinsic parameter calibration operation is performed on the image acquisition devices 22 under each pair of viewpoints to determine the relative pose between the image acquisition devices 22.

[0071] Step S308: Establish the transformation relationship between the three-dimensional coordinate system of the image acquisition device under each viewpoint and the preset world coordinate system of the gantry crane.

[0072] Among them, the three-dimensional coordinate system can be used to characterize the camera coordinate system of the image acquisition device.

[0073] For example, the server can acquire calibration images of preset calibration points in the gantry crane 20 from each viewpoint, captured by the image acquisition device 22. The server identifies the calibration location of the preset calibration point in each viewpoint's calibration image. Using the intrinsic and extrinsic parameters of the image acquisition device 22 for each viewpoint, the calibration location is converted into three-dimensional calibration coordinates in the three-dimensional coordinate system of each image acquisition device 22. The server obtains the gantry crane calibration coordinates of the preset calibration point in a preset gantry crane world coordinate system. Using the three-dimensional calibration coordinates and the gantry crane calibration coordinates, the server establishes the transformation relationship between the three-dimensional coordinate system of the image acquisition device for each viewpoint and the preset gantry crane world coordinate system.

[0074] In this embodiment, by performing intrinsic parameter calibration on the image acquisition device and binocular extrinsic parameter calibration on each pair of image acquisition devices, a transformation relationship between the three-dimensional coordinate system of the image acquisition device and the world coordinate system of the gantry crane is established, which can improve the efficiency and accuracy of subsequent cross-view matching.

[0075] In an exemplary embodiment, step S206 may include: mapping the key point matching results and the rotated bounding box to the gantry crane world coordinate system using transformation relations, extrinsic parameters and intrinsic parameters, and solving the kinematic constraints of the gantry crane to obtain the container pose.

[0076] For example, the server can use intrinsic parameters to convert the pixel coordinates of semantic keypoints in the image data into 3D semantic coordinates of the semantic keypoints in the 3D coordinate system of the image acquisition device 22. Then, through the transformation relationship between the 3D coordinate system of the image acquisition device 22 and the gantry crane world coordinate system, the 3D semantic coordinates are converted into gantry crane semantic coordinates in the gantry crane world coordinate system. Similarly, the rotation bounding box in the image data is converted into a gantry crane bounding box in the gantry crane world coordinate system using intrinsic parameters, extrinsic parameters, and transformation relationships. Based on the keypoint matching results, a correspondence between the gantry crane semantic coordinates from different perspectives is established, forming multiple point pairs in the gantry crane world coordinate system. Container pose estimation is performed using multiple point pairs. In the pose estimation process, the gantry crane bounding box and gantry crane kinematic constraints are introduced as constraints, thereby solving for the optimal six-degree-of-freedom pose as the container pose.

[0077] In this embodiment, by utilizing the transformation relationship between the three-dimensional coordinate system of the image acquisition device and the world coordinate system of the gantry crane, the key point matching results and the rotated bounding box are mapped to the world coordinate system of the gantry crane. Combined with the kinematic constraints of the gantry crane, the coordinate dimensions can be unified, thereby improving the accuracy of container pose estimation.

[0078] In one exemplary embodiment, such as Figure 4 As shown, step S208 may include steps S402 to S406. Wherein:

[0079] Step S402: Determine the translation vector and rotation angle of the spreader pose relative to the container pose.

[0080] For example, the server can perform calculations using the position information in the spreader pose and the position information in the container pose to obtain the translation vector of the spreader pose relative to the container pose. Then, it can perform calculations using the angle information in the spreader pose and the angle information in the container pose to obtain the rotation angle of the spreader pose relative to the container pose.

[0081] Step S404: Generate motion control commands based on the translation vector and rotation angle.

[0082] Step S406: The motion control command is sent to the gantry crane trolley, gantry crane trolley and spreader slewing mechanism to adjust the spreader position and align it with the container position.

[0083] For example, the server can generate motion control commands based on the translation vector and rotation angle, and send the motion control commands to the gantry crane 20's trolley and crane ...

[0084] In this embodiment, by comparing the estimated container pose with the current spreader pose, the translation vector and rotation angle that the spreader needs to move are calculated, and corresponding motion control commands are generated. The motion control commands are executed to guide the gantry crane trolley, gantry crane carriage and spreader slewing mechanism to work together, so that the spreader and the container maintain the same horizontal position and rotation angle, which can improve the accuracy of pose alignment and work efficiency.

[0085] In an exemplary embodiment, the pose alignment method provided in this application may further include: acquiring lifting image data of the spreader and the container during the lifting process; calculating the distance between the spreader and the container based on the lifting image data, wherein the distance is used to adjust the lifting motion parameters of the spreader.

[0086] For example, during the lifting and lowering process of the spreader 21, the server can continue to acquire lifting image data collected by various image acquisition devices 22 regarding the lifting and lowering of the spreader 21 and the container 10. A depth estimation algorithm is used to perform real-time calculations on the lifting and lowering image data to determine the real-time distance between the spreader 21 and the container 10. This enables real-time monitoring of the lifting and lowering process of the spreader 21. Optionally, in some embodiments, the lifting speed, lifting amplitude, and other lifting motion parameters of the spreader 21 can be adjusted based on the distance between the spreader 21 and the container 10.

[0087] In this embodiment, by using a depth estimation algorithm to calculate the distance between the spreader and the container in real time during the spreader lifting process, the safety and accuracy of the spreader lifting control can be improved, and the risk of collision between the spreader and the container can be avoided.

[0088] In an exemplary embodiment, the pose alignment method provided in this application may further include: when the distance reaches a preset distance threshold, performing target detection on the lifting image data to obtain the buckle position of the container; and adjusting the current spreader pose to align with the current container pose according to the buckle position.

[0089] For example, when the distance between the spreader 21 and the container 10 reaches a preset distance threshold, the server can perform target detection on the lifting image data to locate the latch positions at the four corners of the container 10, thus obtaining the four latch positions of the container 10. Based on the detected four latch positions, the pose of the spreader 21 can be further refined to achieve precise alignment between the spreader pose and the container pose.

[0090] In this embodiment, when the spreader approaches the container, the buckles at the four corners of the container are detected and located. The spreader's posture is then further refined based on the buckle positions, which improves the posture alignment accuracy.

[0091] In one exemplary embodiment, such as Figure 5 As shown, a pose alignment method is also provided, including the following steps S502 to S512. Wherein:

[0092] Step S502: Obtain the image acquisition device parameters corresponding to each viewpoint, perform intrinsic parameter calibration and binocular extrinsic parameter calibration, and establish the transformation relationship between the three-dimensional coordinate system of the image acquisition device and the preset gantry crane world coordinate system.

[0093] Step S504: Use a rotating target detection neural network model to detect the rotating bounding box and semantic key points in the image data of the container from each viewpoint.

[0094] Step S506: Perform stereo matching on semantic key points from multiple perspectives to obtain key point matching results. Use transformation relations, extrinsic parameters, and intrinsic parameters to map the key point matching results and the rotated bounding box to the gantry crane world coordinate system. Combine the kinematic constraints of the gantry crane to solve the problem and obtain the container pose.

[0095] Step S508: Determine the translation vector and rotation angle of the spreader's position relative to the container's position, generate motion control commands based on the translation vector and rotation angle, and send the motion control commands to the gantry crane trolley, gantry crane trolley, and spreader slewing mechanism for execution, so as to adjust the spreader's position to align with the container's position.

[0096] Step S510: During the lifting and lowering process of the spreader, acquire lifting image data collected on the spreader and the container, and calculate the distance between the spreader and the container based on the lifting image data.

[0097] Step S512: When the distance reaches the preset distance threshold, target detection is performed on the lifting image data to obtain the buckle position of the container, and the current spreader pose is adjusted to be aligned with the current container pose according to the buckle position.

[0098] In this embodiment, semantic key points are obtained through rotational target detection, container pose is obtained through binocular stereo vision matching, the distance between the spreader and the container is obtained through depth estimation, and precise pose alignment between the spreader and the container is achieved through container latch recognition. This improves the accuracy of container pose estimation, reduces the data processing cost of automated processing, and enhances the algorithm's robustness to harsh environments, as well as its ability to handle and adapt to complex scenarios. This enables fully automated and highly reliable container grasping in gantry crane scenarios, improving the automation and efficiency of container grasping in gantry crane scenarios and reducing the rate of manual intervention.

[0099] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0100] Based on the same inventive concept, this application also provides a pose alignment device for implementing the pose alignment method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more pose alignment device embodiments provided below can be found in the limitations of the pose alignment method described above, and will not be repeated here.

[0101] In one exemplary embodiment, such as Figure 6 As shown, a pose alignment device 600 is provided, including: an image acquisition module 602, a target detection module 604, a pose estimation module 606, and a pose alignment module 608, wherein:

[0102] The image acquisition module 602 is used to acquire image data of the container from multiple perspectives and input the image data into a pre-trained rotating target detection neural network model.

[0103] The target detection module 604 is used to detect the rotated bounding box and semantic key points of the container in the image data of each view using a rotated target detection neural network model.

[0104] The pose estimation module 606 is used to perform stereo matching of semantic key points from multiple perspectives to obtain key point matching results. The container pose is obtained by combining the key point matching results with preset gantry crane kinematic constraints and rotating bounding boxes.

[0105] The pose alignment module 608 is used to generate motion control commands based on the container pose and the current spreader pose, and execute the motion control commands to adjust the spreader pose and the container pose for alignment.

[0106] In an exemplary embodiment, the pose alignment device 600 further includes a calibration module for acquiring image acquisition device parameters corresponding to each viewpoint, the image acquisition device parameters including intrinsic parameters and extrinsic parameters; performing intrinsic parameter calibration operations on the image acquisition device corresponding to each viewpoint according to the intrinsic parameter parameters; performing binocular extrinsic parameter calibration operations on the image acquisition device under different viewpoints according to the extrinsic parameters under different viewpoints; and establishing the transformation relationship between the three-dimensional coordinate system of the image acquisition device under each viewpoint and the preset gantry crane world coordinate system.

[0107] In an exemplary embodiment, the pose estimation module 606 is further configured to map the key point matching results and the rotated bounding box to the gantry crane world coordinate system using transformation relations, extrinsic parameters and intrinsic parameters, and solve the problem in conjunction with the gantry crane kinematic constraints to obtain the container pose.

[0108] In an exemplary embodiment, the pose alignment module 608 is further configured to determine the translation vector and rotation angle of the spreader pose relative to the container pose; generate motion control commands based on the translation vector and rotation angle; and send the motion control commands to the gantry crane trolley, the gantry crane trolley and the spreader slewing mechanism for execution, so as to adjust the spreader pose to align with the container pose.

[0109] In an exemplary embodiment, the pose alignment device 600 further includes a lifting monitoring module, used to acquire lifting image data collected on the spreader and the container during the lifting process; and to calculate the distance between the spreader and the container based on the lifting image data, the distance being used to adjust the lifting motion parameters of the spreader.

[0110] In an exemplary embodiment, the pose alignment module 608 is further configured to perform target detection on the lifting image data when the distance reaches a preset distance threshold, to obtain the buckle position of the container; and adjust the current spreader pose to be aligned with the current container pose according to the buckle position.

[0111] Each module in the aforementioned pose alignment device 600 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0112] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores image data, rotated bounding boxes, semantic keypoints, keypoint matching results, container poses, spreader poses, motion control commands, and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a pose alignment method.

[0113] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0114] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0115] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0116] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A pose alignment method, characterized in that, The method includes: Image data of the container from multiple perspectives is acquired, and the image data is input into a pre-trained rotating target detection neural network model; The rotating target detection neural network model is used to detect the rotated bounding box and semantic key points in the image data of the container from each viewpoint. Binocular stereo matching is performed on semantic key points from multiple perspectives to obtain key point matching results. The container pose is obtained by combining the key point matching results with the preset kinematic constraints of the gantry crane and the rotating bounding box. Based on the container pose and the current spreader pose, a motion control command is generated, and the motion control command is executed to adjust the spreader pose to align with the container pose.

2. The method according to claim 1, characterized in that, Before inputting the image data into the pre-trained rotating target detection neural network model, the method further includes: Obtain the image acquisition device parameters corresponding to each viewpoint, including intrinsic parameters and extrinsic parameters; Perform intrinsic parameter calibration operations on the image acquisition device corresponding to each viewpoint according to the intrinsic parameter parameters; Perform binocular extrinsic parameter calibration on image acquisition devices from different perspectives based on extrinsic parameter values ​​from different perspectives; Establish the transformation relationship between the three-dimensional coordinate system of the image acquisition device and the preset world coordinate system of the gantry crane from each perspective.

3. The method according to claim 2, characterized in that, The process of performing binocular stereo matching on semantic key points from multiple perspectives to obtain key point matching results, and then using these key point matching results in conjunction with preset gantry crane kinematic constraints and the rotated bounding box to solve for the container pose, includes: The key point matching results and the rotated bounding box are mapped to the gantry crane world coordinate system using the transformation relationship, the extrinsic parameters, and the intrinsic parameters, and then solved in combination with the gantry crane kinematic constraints to obtain the container pose.

4. The method according to claim 1, characterized in that, The step of generating motion control commands based on the container pose and the current spreader pose, and executing the motion control commands to adjust the spreader pose to align with the container pose, includes: Determine the translation vector and rotation angle of the spreader pose relative to the container pose; The motion control command is generated based on the translation vector and the rotation angle; The motion control command is sent to the gantry crane trolley, gantry crane carriage, and spreader slewing mechanism for execution, so as to adjust the spreader's posture to align with the container's posture.

5. The method according to claim 1, characterized in that, The method further includes: During the lifting and lowering of the spreader, image data of the lifting and lowering of the spreader and the container are acquired; The distance between the spreader and the container is calculated based on the lifting image data, and the distance is used to adjust the lifting motion parameters of the spreader.

6. The method according to claim 5, characterized in that, The method further includes: When the distance reaches a preset distance threshold, target detection is performed on the lifting image data to obtain the buckle position of the container; Adjust the current spreader position to align with the current container position based on the latch position.

7. A pose alignment device, characterized in that, The device includes: The image acquisition module is used to acquire image data of the container from multiple perspectives and input the image data into a pre-trained rotating target detection neural network model. The target detection module is used to detect the rotated bounding box and semantic key points in the image data of the container from each viewpoint using the rotated target detection neural network model. The pose estimation module is used to perform stereo matching of semantic key points from multiple perspectives to obtain key point matching results. The container pose is obtained by combining the key point matching results with preset gantry crane kinematic constraints and the rotated bounding box. The pose alignment module is used to generate motion control commands based on the container pose and the current spreader pose, and execute the motion control commands to adjust the spreader pose and the container pose to be aligned.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images