Method, device, storage medium and program product for motion control of a tyre crane

Abstract

The application provides a tire crane cart motion control method, device, storage medium and program product, relates to the field of port machinery automation, and the method comprises the following steps: acquiring a first image of a yard based on a camera; identifying a bay number and its position in the first image; determining whether the bay number is a target bay number; if so, controlling the tire crane cart to move to a bay corresponding to the bay number based on the position of the bay number; acquiring a second image of the yard based on the camera, the second image comprising the bay number, and the distance between the camera and the bay number when the second image is acquired is less than the corresponding distance when the first image is acquired; identifying the center position of the bay number in the second image, and controlling the tire crane cart to move to the center position of the bay based on the center position. The application solves the problems of high deployment cost, difficult maintenance, poor positioning stability and easy environmental influence of the positioning of the tire crane cart.

Description

Technical Field

[0001] This application relates to the field of port machinery automation, and in particular to motion control methods, equipment, storage media and program products for tire crane trolleys. Background Technology

[0002] In automated container terminals, the positioning accuracy of rubber-tyred gantry cranes (RTGs) directly determines the loading and unloading efficiency and safety. The higher the positioning accuracy, the higher the motion control accuracy of the RTG.

[0003] The following two methods are commonly used in the positioning of tire crane trolleys in the existing technology:

[0004] The first approach is an infrastructure-dependent solution: using magnetic nails or radio frequency identification (RFID) tags to locate the rubber-tired crane. Specifically, magnetic nails or RFID tags need to be pre-buried in the yard ground, and corresponding magnetic sensors or RFID readers need to be installed on the rubber-tired crane. When the crane travels over the magnetic nails or tags, the sensor reads the pre-stored location code information to obtain the crane's real-time position. However, this solution requires large-scale infrastructure modifications before initial use because it necessitates laying magnetic nails or RFID tags on the ground. Furthermore, dynamic changes to the terminal yard, such as replanning bay layouts, adjusting container stacking areas, or expanding or reducing the yard area, require corresponding adjustments to the positions of the magnetic nails or RFID tags. Therefore, this solution suffers from high deployment costs and maintenance difficulties.

[0005] The second approach is a GPS-based positioning scheme: This involves using a lidar system integrated with GPS, combined with real-time kinematic (RTK) technology, to locate the tire crane. However, this scheme suffers from several drawbacks. In densely packed container yards, GPS signals are easily blocked, and GPS positioning performance drops drastically in foggy weather. Furthermore, lidar is expensive. Therefore, this approach suffers from high deployment costs, poor positioning stability, and susceptibility to environmental influences. Summary of the Invention

[0006] In view of this, this application provides a motion control method, device, storage medium, and program product for a tire crane trolley, which can solve the problems of high deployment cost, difficult maintenance, poor positioning stability, and susceptibility to environmental influences in tire crane trolley positioning schemes.

[0007] The present application is described below from multiple aspects, and the implementation methods and beneficial effects of these aspects can be referred to each other.

[0008] In a first aspect, this application provides a motion control method for a tire-mounted crane, comprising: acquiring a first image of a stockyard using a camera, the first image including a bay number; identifying the bay number and its position in the first image based on a first recognition model; determining whether the bay number is a target bay number; if the bay number is a target bay number, controlling the tire-mounted crane to move to the bay corresponding to the bay number based on the position of the bay number; if the tire-mounted crane has moved to the bay corresponding to the bay number, acquiring a second image of the stockyard using a camera, the second image including the bay number, and the distance between the camera and the bay number when acquiring the second image is less than the distance between the camera and the bay number when acquiring the first image; identifying the center position of the bay number in the second image based on a second recognition model; and adjusting the position of the tire-mounted crane based on the center position of the bay number to move the tire-mounted crane to the center position of the bay.

[0009] According to this solution, firstly, there is no need to deploy additional physical markers such as magnetic nails or RFID tags, reducing deployment and maintenance costs. Secondly, replacing lidar with a vision-based solution improves positioning stability and reduces susceptibility to yard environment and weather conditions. Thirdly, based on visual recognition, the approximate location of the bay number is first roughly determined, followed by precise positioning of the bay number's center position. This staged processing improves positioning accuracy, making the motion control process of the tire crane more stable and reliable.

[0010] In one possible implementation of the first aspect above, adjusting the position of the tire crane based on the center position of the berth number includes: evaluating the confidence level of the second recognition model using an adaptive Kalman filter; and adjusting the position of the tire crane based on the odometer and / or inertial measurement unit of the tire crane if the confidence level is lower than a first preset threshold; wherein the odometer and inertial measurement unit are used to record the moving distance and direction of the tire crane.

[0011] According to this scheme, the confidence level of the second recognition model is evaluated in real time using an adaptive Kalman filter, thereby effectively determining whether the current visual positioning result is reliable. When the position number is obscured by dirt, there are drastic changes in lighting, or camera shake, resulting in a low confidence level for the second recognition model, the system will not blindly trust erroneous visual information. Instead, it will switch to a dead reckoning mode based on odometry or inertial measurement units, using short-term, high-precision relative motion information to continue position adjustment. This avoids control interruptions or malfunctions caused by visual failures, ensuring that the tire crane can still stably complete alignment under complex working conditions.

[0012] In one possible implementation of the first aspect above, identifying the berth number and its location in the first image based on the first recognition model includes: identifying the berth number in the first image based on the first recognition model; retrieving the coordinate range of the berth number from a pre-constructed electronic map based on the berth number; and determining the location of the berth number based on the coordinate range of the berth number.

[0013] According to the scheme, the electronic map provides a stable spatial reference. By retrieving the known coordinate range of the identified bay number in the electronic map, the visual recognition results can be fused with the actual geographical location to obtain the position of the bay number in the yard, which makes it easier for the rubber-tired crane to move accurately to the corresponding bay.

[0014] In one possible implementation of the first aspect above, identifying the center position of the berth number in the second image based on the second recognition model includes: identifying the outline of the berth number in the second image based on the second recognition model; and determining the center position of the berth number based on the outline of the berth number.

[0015] According to this scheme, the geometric center of the physical mark of the berth number is accurately reflected by extracting the complete outline of the berth number and calculating the center based on the geometry of the outline.

[0016] In one possible implementation of the first aspect above, determining the center position of the bit number based on the outline of the bit number includes: determining the coordinates of the center position of the bit number in the pixel coordinate system based on the outline of the bit number; and converting the coordinates from the pixel coordinate system to the world coordinate system to determine the center position of the bit number.

[0017] According to this scheme, the center position identified in the image is mapped to the actual physical position through coordinate transformation, thereby providing the required data for the motion control of the tire crane.

[0018] In one possible implementation of the first aspect described above, the camera is mounted on a tire crane, and the first and second images are extracted from video captured by the camera.

[0019] According to the scheme, based on the method of extracting images from video, electronic devices can extract the first and second images from the video captured by the camera, realize real-time visual feedback in the process of large vehicle motion control, and facilitate smooth motion adjustment.

[0020] In one possible implementation of the first aspect above, acquiring a first image of the stockyard based on a camera includes: extracting video frames from a video captured by the camera to obtain a first initial image, and preprocessing the first initial image to obtain a first image; wherein the preprocessing includes at least one of distortion removal, grayscale conversion, and histogram equalization; or, acquiring a second image of the stockyard based on a camera includes: extracting video frames from a video captured by the camera to obtain a second initial image, and preprocessing the second initial image to obtain a second image.

[0021] According to this scheme, the preprocessed image features are clearer and more consistent, which reduces the processing difficulty of the first and second recognition models, thereby improving the success rate and accuracy of number recognition and center positioning, and indirectly enhancing the robustness of the entire motion control method.

[0022] In a second aspect, this application provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the method disclosed in the first aspect above and any possible implementation thereof.

[0023] Thirdly, this application provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the method disclosed in the first aspect above and any possible implementation thereof.

[0024] Fourthly, this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the methods disclosed in the first aspect and any possible implementation thereof.

[0025] It should be understood that the beneficial effects of the second to fourth aspects mentioned above can be referred to the beneficial effects described in the first aspect, and will not be repeated here. Attached Figure Description

[0026] Figure 1 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0027] Figure 2 A schematic flowchart illustrating the motion control method for a tire-mounted crane trolley provided in an embodiment of this application;

[0028] Figure 3 A schematic diagram illustrating the process of identifying the bayonet number and its position as provided in an embodiment of this application;

[0029] Figure 4 A flowchart illustrating the process of identifying the center position of the bay number provided in an embodiment of this application;

[0030] Figure 5 Scene diagrams from the camera's perspective provided in embodiments of this application;

[0031] Figure 6 A schematic diagram illustrating the process of adjusting the position of the tire crane trolley provided in an embodiment of this application;

[0032] Figure 7 A schematic diagram of the motion control device for the tire crane trolley provided in the embodiments of this application;

[0033] Figure 8 A block diagram of an electronic device provided in an embodiment of this application;

[0034] Figure 9 This is a block diagram of a SoC (system on chip) provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] First, the technical problems to be solved by the embodiments of this application will be explained.

[0037] In some embodiments, the positioning scheme for rubber-tired cranes adopts an infrastructure-dependent approach. Specifically, magnetic nails or RFID tags are pre-embedded in the yard ground, and corresponding magnetic induction sensors or RFID readers are installed on the rubber-tired cranes. Based on this, the crane's position can be obtained in real time. However, this scheme requires large-scale infrastructure modifications before its initial use, including pre-embedding magnetic nails or RFID tags. Furthermore, when the terminal yard undergoes dynamic changes, the positions of the magnetic nails or RFID tags need to be adjusted accordingly, resulting in high deployment costs and maintenance difficulties.

[0038] In other embodiments, a GPS-based positioning scheme is used for the positioning of the tire crane. Specifically, a lidar system integrating GPS is used in conjunction with RTK positioning technology to locate the tire crane. However, this scheme suffers from several drawbacks: GPS signals are easily blocked in densely packed container yards, GPS positioning performance drops sharply in foggy weather, and lidar is expensive, resulting in high deployment costs, poor positioning stability, and susceptibility to environmental influences.

[0039] In view of this, embodiments of this application provide a motion control method for a tire crane trolley. A first image of the bay number is acquired using a camera, and preliminary identification is performed on the first image to roughly locate the position of the number, allowing the tire crane trolley to move to the corresponding bay. Then, a second image of the bay number is acquired using a camera, where the distance between the camera and the bay number is closer than when the first image was acquired. A second identification is performed on the second image to finely locate the center position of the bay number, enabling the trolley to accurately move to the center position of the corresponding bay.

[0040] Therefore, the motion control method for the tire crane provided in this application uses visual recognition for positioning, eliminating the need for additional physical markers such as magnetic nails or RFID tags, and also eliminating the need for lidar. This solves the problems of high deployment costs, difficult maintenance, poor positioning stability, and susceptibility to environmental influences. Moreover, by first coarsely locating the approximate position of the bay number using visual recognition, and then finely locating the center position of the bay number, the accuracy of positioning is improved, making the motion control process of the tire crane more stable and reliable.

[0041] The motion control method for a tire crane trolley is described below with reference to specific embodiments and accompanying drawings.

[0042] Reference Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application.

[0043] like Figure 1 As shown, a camera 2 can be installed on the tire crane trolley 1. Camera 2 is specifically fixed to the center of the side of the tire crane trolley 1, 3.5m above the ground, with a 30° downward angle, ensuring that the camera 2's field of view covers the lane and adjacent bays. Camera 2 is powered by a 24V DC power supply and has wide dynamic range (WDR) functionality. Camera 2 can move with the tire crane trolley 1 and capture video in real time, eliminating the need for additional installation of a device to fix camera 2 in the yard, thus reducing deployment complexity and cost.

[0044] The motion control method for the tire crane trolley according to this application embodiment can be executed by an electronic device with processing capabilities. This electronic device is located in an electrical room and can be equipped with an embedded AI processor (such as Jetson Xavier), a custom Linux system, and algorithm software. It has a built-in neural network inference engine responsible for image preprocessing and AI model inference. On one hand, the electronic device can communicate with a camera via gigabit Ethernet to acquire video footage in real time and determine the position of the tire crane trolley. On the other hand, the electronic device can communicate with the main controller of the tire crane trolley via a switch, for example, using the Modbus TCP protocol or the Profinet protocol. Thus, the electronic device can send commands to the main controller based on the position of the tire crane trolley to control its movement distance and direction.

[0045] The motion control method for a tire crane provided in this application embodiment will be described below, taking an electronic device as the execution subject as an example.

[0046] Reference Figure 2 , Figure 2 This is a flowchart illustrating the motion control method for a tire crane trolley provided in an embodiment of this application.

[0047] like Figure 2 As shown, the motion control method for the tire crane trolley includes steps S210 to S270.

[0048] S210: Acquire the first image of the stockpile based on the camera.

[0049] The first image includes a bay number.

[0050] S220: Based on the first recognition model, identify the berth number and the position of the berth number in the first image.

[0051] S230: Determine whether the bay number is the target bay number.

[0052] S240: When the bay number is the target bay number, the tire crane trolley is moved to the bay number corresponding to the bay number based on the position control of the bay number.

[0053] For example, if the target bay number is 67, it means that the tire crane trolley needs to be moved to bay number 67.

[0054] S250: When the tire crane moves to the bay number corresponding to the bay, a second image of the yard is acquired based on the camera.

[0055] The second image includes a bit position number, and the distance between the camera and the bit position number when acquiring the second image is less than the distance between the camera and the bit position number when acquiring the first image.

[0056] S260: Based on the second recognition model, identify the center position of the berth number in the second image.

[0057] S270: Based on the center position of the bay number, adjust the position of the tire crane trolley so that the tire crane trolley moves to the center position of the bay.

[0058] Specifically, the center position of the corresponding bay can be determined first based on the center position of the bay number, then the deviation between the position of the tire crane trolley and the center position of the bay can be determined, and then the position of the tire crane trolley can be adjusted to move it to the center position of the bay.

[0059] Based on steps S210-S270, visual recognition is used to locate and control the rubber-tired crane to move to the center of the corresponding bay, facilitating subsequent loading and unloading operations. This method has several advantages: First, it eliminates the need for additional physical markers such as magnetic nails or RFID tags, reducing deployment and maintenance costs. Second, replacing lidar with a visual solution improves positioning stability and reduces susceptibility to yard conditions and weather. Third, based on visual recognition, the approximate location of the bay number is first roughly determined, followed by precise positioning of its center. This staged processing improves positioning accuracy, making the motion control of the rubber-tired crane smoother and more reliable, while maintaining recognition performance even with limited computing power in the embedded system.

[0060] In some embodiments, the process of acquiring a first image of the stockyard based on a camera as described in step S210 may include the following steps:

[0061] Video frames are extracted from the video captured by the camera to obtain a first initial image, and the first initial image is preprocessed to obtain a second image.

[0062] The process of acquiring a second image of the stockpile based on a camera, as mentioned in step S250, may include the following steps:

[0063] Video frames are extracted from the video captured by the camera to obtain a second initial image, and the second initial image is preprocessed to obtain a second image.

[0064] The preprocessing of the first or second initial image may include at least one of distortion removal, grayscale conversion, and histogram equalization. Distortion removal can eliminate geometric distortion caused by wide-angle lenses, making the outline and position of the berth number closer to the real geometric relationship. Grayscale conversion can remove unnecessary color information and reduce computational load. Histogram equalization can enhance image contrast and improve the recognizability of the berth number under backlight, shadow, or low-light conditions. The preprocessed image features are clearer and more consistent, reducing the processing difficulty of the first and second recognition models, thereby improving the success rate and accuracy of number recognition and center localization, and indirectly enhancing the robustness of the entire motion control method.

[0065] Understandably, before acquiring the first image of the stockyard based on the camera, the camera calibration process can be performed in advance using the Zhang Zhengyou method to obtain the camera's intrinsic parameter matrix and distortion system, which facilitates subsequent image preprocessing.

[0066] It is understood that the first image can be one or more images, and the second image can also be one or more images; this application does not limit this.

[0067] Based on the method of extracting images from videos, electronic devices can extract first and second images from videos captured by cameras, enabling real-time visual feedback during the motion control process of large vehicles, which is beneficial for achieving smooth motion adjustments.

[0068] Reference Figure 3 , Figure 3 This is a schematic diagram of the process for identifying the bit position number and the position of the bit position number provided in the embodiments of this application.

[0069] In some embodiments, such as Figure 3 As shown, the process of identifying the bit number and the position of the bit number in the first image based on the first recognition model in step S220 may include steps S310 to S330.

[0070] S310: Based on the first recognition model, identify the bit position number in the first image.

[0071] Specifically, the first recognition model can use a convolutional recurrent neural network model (CRNN model) or a transformer architecture.

[0072] The training process of this first recognition model can employ a combination of rule-based synthetic data pre-training and parameter adjustment using a small amount of real-world image data. Specifically, firstly, a computer program randomly generates bay numbering based on rules and renders it onto various random backgrounds (such as concrete textures, rust, and weeds). Simultaneously, random perspective transformations are applied to the bay numbering (simulating the visual effects viewed from different angles and distances), resulting in tens of thousands of synthetic images with random backgrounds and perspective transformations. These images are then used to pre-train the untrained first recognition model. Next, a small number of real images taken in the yard are used for secondary training of the pre-trained first recognition model. The model's parameters are adjusted to adapt it to the specific environment of a real yard, achieving a recognition rate of over 99% with extremely low annotation costs. This allows the model to be adapted only by collecting a small number of real-world samples for each yard when migrating and deploying it between different docks, significantly reducing data annotation costs.

[0073] S320: Based on the position number, retrieve the coordinate range of the position number from a pre-constructed electronic map.

[0074] Understandably, the electronic map includes each bay number and its corresponding coordinate range. During initial deployment, the electronic map is built by scanning the entire yard, thus supporting subsequent rapid matching. For example, if bay number 67 is identified, the corresponding coordinate range can be quickly retrieved using the electronic map. The error between this coordinate range and the actual coordinates of the bay number is within ±50cm.

[0075] S330: Determine the position of the bit number based on the coordinate range of the bit number.

[0076] Based on steps S310~S330, the electronic map provides a stable spatial reference. By retrieving the known coordinate range of the identified bay number in the electronic map, the visual recognition result can be fused with the actual geographical location to obtain the position of the bay number in the yard, which facilitates the accurate movement of the rubber-tired crane to the corresponding bay.

[0077] Reference Figure 4 , Figure 4 This is a flowchart illustrating the process of identifying the center position of the bit number in an embodiment of this application.

[0078] In some embodiments, such as Figure 4 As shown, the process of identifying the center position of the berth number in the second image based on the second recognition model in step S260 may include steps S410 to S420.

[0079] S410: Based on the second recognition model, identify the contour of the berth number in the second image.

[0080] Specifically, the second recognition model only needs to extract the region of interest (ROI) from the second image, that is, a region centered on the bay number. This region includes the bay number and its background, thus eliminating background interference and reducing computational load. Then, the second recognition model can use deep learning models such as U-Net and YOLOv8-seg to identify the contour of the bay number, achieving pixel-level recognition. Taking the YOLOv8-seg model as an example, improvements can be made to the YOLOv8-seg model by introducing a coordinate attention mechanism and cross-scale feature fusion (ASFF) technology.

[0081] The coordinate attention mechanism is embedded at the end of the model's backbone (responsible for feature extraction), allowing the model to focus more on positional and geometric information within the image. For morphological shading with bayonet numbers, this mechanism helps the model more accurately pinpoint edges and shapes. Figure 5 A scene diagram from the camera's perspective provided in an embodiment of this application, referring to... Figure 5 The background pattern of the serial number refers to the white area corresponding to serial number 67.

[0082] Cross-scale feature fusion technology is placed in the neck of the model (responsible for fusing features from different levels). When a camera shoots video, due to differences in distance and angle, the size of the bezel number will vary in the image due to perspective distortion and other scale changes. The role of ASFF is to enable the model to automatically learn to handle these changes in size and shape, accurately segmenting the bezel number regardless of whether it appears skewed or tilted in the image.

[0083] When performing the segmentation masking task, the model outputs a black and white image, where white represents the background area of ​​the B-number and black represents the background. Non-maximum suppression is applied to the black and white image output by the model, retaining only the black and white image with the best effect, and the Canny-Zernike moment subpixel edge detection algorithm is used to improve the edge localization accuracy of the B-number to the subpixel level (better than 0.1 pixels).

[0084] S420: Determine the center position of the bit number based on the outline of the bit number.

[0085] Reference Figure 5 The second image may include bay number 67, a tiled background, and yellow lane lines, etc. The outline of the bay number is also the outline of bay number 67.

[0086] Based on steps S410~S420, the complete outline of the shell number is extracted, and the center is calculated based on the geometry of the outline, thereby accurately reflecting the geometric center of the physical mark of the shell number.

[0087] In some embodiments, continue to refer to Figure 4 The process of determining the center position of the berth number based on the outline of the berth number mentioned in step S420 may include steps S421 to S422.

[0088] S421: Based on the contour of the bit number, determine the coordinates of the center position of the bit number in the pixel coordinate system.

[0089] S422: Convert the coordinates from the pixel coordinate system to the world coordinate system to determine the center position of the bit number.

[0090] Specifically, the Inverse Perspective Mapping (IPM) algorithm can be used, combined with pre-calibrated camera intrinsic and extrinsic parameters, for coordinate transformation. The formula for converting coordinates from pixel coordinates (u, v) to world coordinates (X, Y, Z) using the IPM algorithm is as follows:

[0091]

[0092] In the formula, K is the camera intrinsic parameter, and [R|t] is the camera extrinsic parameter, which is the position and angle of the camera mounted on the tire crane trolley.

[0093] After determining the center position of the bay number, and combining the physical dimensions of the bay number, the precise offset of the tire crane trolley relative to the bay position can be calculated, thereby achieving absolute positioning within ±3cm.

[0094] Based on steps S421~S422, pixel coordinates are usually not directly usable for controlling the movement of the tire crane. Through coordinate transformation, the center position identified in the image is mapped to the actual physical position, thereby providing the required data for the motion control of the tire crane.

[0095] Reference Figure 6 , Figure 6 This is a schematic diagram illustrating the process of adjusting the position of the tire crane trolley as provided in an embodiment of this application.

[0096] In some embodiments, such as Figure 6 As shown, the process of adjusting the position of the tire crane trolley based on the center position of the bay number mentioned in step S270 may include steps S610 to S620.

[0097] S610: An adaptive Kalman filter is used to evaluate the confidence level of the second recognition model.

[0098] For example, the confidence level of the second recognition model can be evaluated by splitting the IOY value and dynamically adjusting the observation noise covariance matrix R. If the value of R is large, the confidence level of the second recognition model can be considered low.

[0099] S620: When the confidence level is lower than a first preset threshold, adjust the position of the tire crane based on the odometer and / or inertial measurement unit of the tire crane.

[0100] For example, the first preset threshold can be 0.5.

[0101] The odometer and inertial measurement unit (IMU) are used to record the distance and direction of movement of the tire crane trolley. Additionally, the odometer and IMU can provide initial pose estimation, facilitating the control of the tire crane trolley to move to the vicinity of the target location before activating the camera and acquiring the first image, thereby improving motion control efficiency.

[0102] In another embodiment, when the confidence level is higher than a first preset threshold (e.g., 0.5) but lower than a second preset threshold (e.g., 0.95), the position of the tire crane can be adjusted based on data from the second recognition model, the odometer, and the inertial measurement unit. For example, when the vision system is occluded, the confidence level of the second recognition model decreases; in this case, its weight can be reduced, while the weights of the odometer and the inertial measurement unit can be increased. When the vision becomes clear, the confidence level of the second recognition model increases; in this case, its weight can be increased accordingly, while the weights of the odometer and the inertial measurement unit can be decreased, thus achieving dynamic adjustment of the weights.

[0103] Based on steps S610-S620, the confidence level of the second recognition model is evaluated in real time using an adaptive Kalman filter, thereby effectively determining whether the current visual positioning result is reliable. When the position number is obscured by dirt, there are drastic changes in lighting, or camera shake, resulting in a low confidence level for the second recognition model, the system will not blindly trust erroneous visual information. Instead, it will switch to a dead reckoning mode based on odometry or inertial measurement units, using short-term, high-precision relative motion information to continue position adjustment. This avoids control interruptions or malfunctions caused by visual failures, ensuring that the tire crane can still stably complete alignment under complex working conditions.

[0104] This application also provides a motion control device for a tire-mounted crane. This motion control device applies the motion control method for the tire-mounted crane described in the above embodiments.

[0105] Specifically, refer to Figure 7 , Figure 7 This is a schematic diagram of the motion control device for the tire crane trolley provided in the embodiments of this application, as shown below. Figure 7 As shown, the motion control device includes a camera drive module, a data acquisition module, a positioning algorithm module, a post-processing module, a business logic module, and a PLC communication module.

[0106] The camera driver module is used to drive the camera to start and capture video.

[0107] The data acquisition module is used to acquire the first image of the stockyard based on the camera. The first image includes the bay number.

[0108] The positioning algorithm module is used to identify the bay number and the position of the bay number in the first image based on the first recognition model, determine whether the bay number is the target bay number, and send a first instruction to the post-processing module if the bay number is the target bay number.

[0109] The post-processing module is also used to pass the first instruction to the business logic module.

[0110] The business logic module is used to send a first motion command to the main PLC of the tire crane trolley through the PLC communication module based on the first instruction, so that the main PLC controls the tire crane trolley to move to the position corresponding to the position number.

[0111] When the tire crane moves to the bay number corresponding to the bay number, the data acquisition module is used to acquire a second image of the yard based on the camera. The second image includes the bay number, and the distance between the camera and the bay number when acquiring the second image is less than the distance between the camera and the bay number when acquiring the first image.

[0112] The positioning algorithm module is used to identify the center position of the berth number in the second image based on the second recognition model, and send a second instruction to the post-processing module based on the center position of the berth number.

[0113] The post-processing module is also used to pass the second instruction to the business logic module.

[0114] The business logic module is used to send a second motion command to the main PLC of the tire crane trolley through the PLC communication module based on the second instruction, so that the main PLC adjusts the position of the tire crane trolley and moves the tire crane trolley to the center position of the trolley.

[0115] In some embodiments, refer to Figure 7 The motion control device may further include a preprocessing module, which is used to extract video frames from the video captured by the camera to obtain a first initial image, and to preprocess the first initial image to obtain a first image. It is also used to extract video frames from the video captured by the camera to obtain a second initial image, and to preprocess the second initial image to obtain a second image.

[0116] In some embodiments, refer to Figure 7 The motion control device may also include a WebUI module, and the post-processing module is used to control the WebUI module to display relevant information based on the first command or the second command.

[0117] In some embodiments, refer to Figure 7The business logic module also has a LOG recording function, which is used to record abnormal operating conditions of the tire crane trolley, including videos and records.

[0118] In some embodiments, the motion control device may further include a system communication interface module, through which the motion control device can communicate with other devices.

[0119] It is understood that the specific functions and corresponding technical effects of the motion control device for the tire crane can be found in the explanations in the embodiments corresponding to the motion control method for the tire crane mentioned above, and will not be repeated here.

[0120] This application also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program segment. The processor loads and executes the instruction or program segment to implement the motion control method for the tire crane described in the above embodiments. The specific functions and corresponding technical effects can be found in the explanations of the embodiments corresponding to the motion control method for the tire crane described above, and will not be repeated here.

[0121] Now for reference Figure 8 , Figure 8 This is a block diagram of an electronic device 1200 provided in an embodiment of this application. The electronic device 1200 may include one or more processors (corresponding to) coupled to a controller hub 1203. Figure 8 The first processor 1201 is described above. In at least one embodiment, the controller hub 1203 communicates with the first processor 1201 via a multi-branch bus such as a front-side bus (FSB), a point-to-point interface such as a quick-path interconnect (QPI), or a similar connection. The first processor 1201 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 1203 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on separate chips) (not shown), wherein the GMCH includes memory and a graphics controller and is coupled to the IOH.

[0122] Electronic device 1200 may also include a coprocessor coupled to controller hub 1203 (corresponding to...) Figure 8The processor 1202 and memory 1204 are integrated within the processor (as described in this application). Alternatively, one or both of the memory and GMCH can be integrated within the processor (as described in this application), with memory 1204 and the first coprocessor 1202 directly coupled to the first processor 1201 and the controller hub 1203, which is located on a single chip with the IOH. Memory 1204 can be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both. In one embodiment, the first coprocessor 1202 is a dedicated processor, such as a high-throughput MIC processor (many integerized core, MIC), a network or communication processor, a compression engine, a graphics processor, a general-purpose computing on GPU (GPGPU), or an embedded processor, etc. Optional properties of the first coprocessor 1202 are indicated by dashed lines. Figure 8 middle.

[0123] As a computer-readable storage medium, memory 1204 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, memory 1204 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device such as one or more hard-disk drives (HDDs), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0124] In one embodiment, electronic device 1200 may further include a network interface controller (NIC) 1206. Network interface 1206 may include a transceiver for providing a radio interface for electronic device 1200 to communicate with any other suitable device, such as a front-end module, antenna, etc. In various embodiments, network interface 1206 may be integrated with other components of electronic device 1200. Network interface 1206 can implement the functions of the communication unit in the above embodiments.

[0125] Electronic device 1200 may further include input / output (I / O) device 1205. I / O device 1205 may include: a user interface designed to enable a user to interact with electronic device 1200; a peripheral component interface designed to enable peripheral components to also interact with electronic device 1200; and / or sensors designed to determine environmental conditions and / or location information related to electronic device 1200.

[0126] It is worth noting that, Figure 8 This is merely an example. That is, although... Figure 8 The electronic device 1200 shown includes multiple devices such as a first processor 1201, a first coprocessor 1202, a controller hub 1203, and a memory 1204. However, in actual applications, devices using the methods of this application may include only a portion of the devices in the electronic device 1200. For example, it may include only the first processor 1201 and the network interface 1206. Figure 8 The properties of the optional devices are shown in dashed lines. According to some embodiments of this application, the memory 1204, which is a computer-readable storage medium, stores instructions that, when executed on a computer, cause the electronic device 1200 to perform the motion control method for the tire crane trolley according to the above embodiments. Specific details can be found in the methods described in the above embodiments, and will not be repeated here.

[0127] Now for reference Figure 9 , Figure 9 This is a block diagram of a SoC (system on chip) provided in an embodiment of this application. Figure 9 In the diagram, similar components share the same reference numerals. Additionally, dashed boxes are an optional feature for more advanced SoCs. Figure 9 In the SoC1300, interconnect unit 1350 is coupled to the processor (corresponding to...). Figure 9 The system includes a second processor 1310, a system agent unit 1380, a bus controller unit 1390, an integrated memory controller unit 1340, and one or more coprocessors (corresponding to...). Figure 9 The second coprocessor 1320 may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1330; and a direct memory access (DMA) unit 1360. In one embodiment, the second coprocessor 1320 includes a dedicated processor, such as a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor.

[0128] The static random access memory (SRAM) cell 1330 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of those instructions. These instructions may include, when executed by at least one unit in the processor, causing the SoC 1300 to perform the motion control method for the tire crane according to the above embodiments, as detailed in the methods described above, which will not be repeated here.

[0129] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0130] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0131] This application also provides a computer-readable storage medium storing at least one instruction or at least one program segment. The instruction or program segment is loaded and executed by a processor to implement the motion control method for a tire-mounted crane as described in the above embodiments. The specific functions and corresponding technical effects can be found in the explanations of the embodiments corresponding to the motion control method for the tire-mounted crane described above, and will not be repeated here.

[0132] This application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are loaded and executed by a processor, they implement the motion control method for the tire crane described in the above embodiments. The specific functions and corresponding technical effects can be found in the explanations of the embodiments corresponding to the motion control method for the tire crane described above, and will not be repeated here.

[0133] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A motion control method for a tire-mounted crane trolley, characterized in that, include: A first image of the stockyard is acquired using a camera, the first image including the bay number; Based on the first recognition model, identify the bay number and the position of the bay number in the first image; Determine whether the bit position number is the target bit position number; When the bay number is the target bay number, the tire crane trolley is moved to the bay number corresponding to the bay number based on the position of the bay number. When the tire crane moves to the bay number corresponding to the bay number, a second image of the yard is acquired based on the camera. The second image includes the bay number, and the distance between the camera and the bay number when the second image is acquired is less than the distance between the camera and the bay number when the first image is acquired. Based on the second recognition model, the center position of the berth number in the second image is identified; Based on the center position of the bay number, adjust the position of the tire crane trolley so that the tire crane trolley moves to the center position of the bay number.

2. The motion control method for a tire-mounted crane trolley according to claim 1, characterized in that, Adjusting the position of the tire crane trolley based on the center position of the aforementioned bay number includes: An adaptive Kalman filter is used to evaluate the confidence level of the second recognition model; If the confidence level is lower than a preset threshold, the position of the tire crane is adjusted based on the odometer and / or inertial measurement unit of the tire crane; wherein the odometer and the inertial measurement unit are used to record the moving distance and direction of the tire crane.

3. The motion control method for a tire-mounted crane trolley according to claim 1, characterized in that, Based on the first recognition model, the identification of the bay number and the position of the bay number in the first image includes: Based on the first recognition model, identify the bit position number in the first image; Based on the berth number, the coordinate range of the berth number is retrieved from a pre-constructed electronic map; The position of the bit number is determined based on the coordinate range of the bit number.

4. The motion control method for a tire-mounted crane trolley according to claim 1, characterized in that, Based on the second recognition model, identifying the center position of the berth number in the second image includes: Based on the second recognition model, the outline of the bay number in the second image is recognized; Based on the outline of the berth number, the center position of the berth number is determined.

5. The motion control method for a tire-mounted crane trolley according to claim 4, characterized in that, Determining the center position of the berth number based on its outline includes: Based on the outline of the bit number, determine the coordinates of the center position of the bit number in the pixel coordinate system; The coordinates are converted from the pixel coordinate system to the world coordinate system to determine the center position of the bit number.

6. The motion control method for a tire-mounted crane trolley according to claim 1, characterized in that, The camera is mounted on the tire crane trolley, and the first image and the second image are extracted from the video captured by the camera.

7. The motion control method for a tire-mounted crane trolley according to claim 6, characterized in that, Acquiring a first image of the storage yard based on a camera includes: extracting video frames from a video captured by the camera to obtain a first initial image, and preprocessing the first initial image to obtain the first image; wherein the preprocessing includes at least one of distortion correction, grayscale conversion, and histogram equalization; or, Acquiring a second image of the storage yard based on the camera includes: extracting video frames from the video captured by the camera to obtain a second initial image, and preprocessing the second initial image to obtain the second image.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the motion control method for a tire crane as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the motion control method for a tire crane as described in any one of claims 1-7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, the motion control method for the tire crane trolley as described in any one of claims 1-7 is implemented.

Images