Trailer-mounted container control method and device, computer device and readable storage medium

Abstract

The application provides an external container truck landing box control method and device, computer equipment and a readable storage medium. The method comprises the following steps: determining the pose of the external container truck in a world coordinate system according to a pre-calibrated camera external parameter and an external container truck image; determining the pose of a spreader in the world coordinate system, and fine-tuning the pose of the spreader according to the difference between the poses of the external container truck and the spreader in the same coordinate system; after fine-tuning the spreader, determining the difference between the theoretical pixel coordinates and the actual pixel coordinates of each corner of the container, and judging whether the landing box can be successfully completed according to the difference; if yes, the spreader is controlled to land the box, otherwise, the camera external parameter is recalibrated, and the step of determining the pose of the external container truck in the world coordinate system according to the pre-calibrated camera external parameter and the external container truck image is returned. The application can avoid the possibility of incorrect landing in the unsuccessful case, effectively compensate for the defect of inaccurate rough positioning of the external container truck landing box, form a closed-loop external container truck landing box control, and improve the success rate of landing.

Description

Technical Field

[0001] This invention relates to the field of container loading and unloading, and more particularly to an external container holding control method, device, computer equipment, and readable storage medium. Background Technology

[0002] With the rapid increase in the number of containers, the volume and frequency of container handling in port yards are constantly increasing. Manual operation of containers in the yard is relatively inefficient, which has greatly stimulated and promoted the development and application of port automation systems. Currently, many large ports have implemented fully automated container operating systems in their yards. However, in the crucial step of automatic container loading by external trucks, the mainstream technical solution still adopts an open-loop control method. This method uses multiple lidar sensors installed around the yard to locate the real-time position of the external trucks and controls the corresponding cranes to move to that position to perform loading or unloading operations.

[0003] However, the control accuracy of the above-mentioned methods heavily relies on the accuracy of the external container truck's positioning. If the positioning error exceeds the design tolerance, the automated container loading task may fail, requiring manual reloading, which significantly reduces overall operational efficiency. Besides the positioning dependency issue, the LiDAR automated positioning solution itself also faces several technical challenges and cost issues. For example, the limited field of view of a single unit necessitates multiple units overlapping for coverage, resulting in high equipment costs and susceptibility to ambient lighting and weather conditions. Furthermore, the real-time processing of large amounts of point cloud data places high demands on computing resources. In addition, different container models require different template matching models for identification, posing challenges to the expansion and maintenance of automated operation capabilities. Therefore, improving the container loading success rate is a technical problem that needs to be solved. Summary of the Invention

[0004] The object of the present invention includes, for example, providing an external container landing control method, apparatus, computer device, and readable storage medium capable of improving container landing success rate. The present invention can be achieved as follows: In a first aspect, the present invention provides a method for controlling container landing by an external container truck, the method comprising: determining the pose of the external container truck in a world coordinate system based on pre-calibrated camera extrinsic parameters and an image of the external container truck; determining the pose of a spreader in the world coordinate system, and fine-tuning the pose of the spreader based on the difference between the poses of the external container truck and the spreader in the same coordinate system; after fine-tuning the spreader, determining the difference between the theoretical pixel coordinates and the actual pixel coordinates of each corner of the container, and predicting whether successful container landing is possible based on the difference; if so, controlling the spreader to land the container; otherwise, recalibrating the camera extrinsic parameters and returning to the step of determining the pose of the external container truck in the world coordinate system based on pre-calibrated camera extrinsic parameters and an image of the external container truck.

[0005] Secondly, the present invention provides an external container truck container landing control device, comprising: a positioning module, a control module, an inspection module, and a calibration module; the positioning module is used to determine the pose of the external container truck in the world coordinate system based on pre-calibrated camera extrinsic parameters and an image of the external container truck; the control module is used to determine the pose of the spreader in the world coordinate system, and to fine-tune the pose of the spreader based on the difference between the poses of the external container truck and the spreader in the same coordinate system; the inspection module is used to determine, after the spreader is fine-tuned, the difference between the theoretical pixel coordinates of each corner of the container before fine-tuning and the actual pixel coordinates after fine-tuning, and to predict whether successful container landing is possible based on the difference; the control module is also used to control the spreader to land the container if the result of the inspection module is yes; the calibration module is used to recalibrate the camera extrinsic parameters and return to the positioning module if the result of the inspection module is no.

[0006] Thirdly, the present invention provides a computer device including a processor and a memory, the memory storing a computer program executable by the processor, the processor executing the computer program to implement the external card box control method described in the first aspect.

[0007] Fourthly, the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the external card holder control method as described in the first aspect.

[0008] The present invention provides a method, apparatus, computer equipment, and readable storage medium for controlling container placement on external trucks. First, based on pre-calibrated camera extrinsic parameters and the image of the external truck, the precise position and orientation of the external truck in the world coordinate system are determined. This provides an accurate reference for subsequent calculations of spreader adjustments and verification. Then, based on the precise orientation of the external truck, the spreader is fine-tuned to align the container with the external truck. After fine-tuning, the difference between the theoretical and actual pixel coordinates of each corner of the container is determined. If the difference is too large, it indicates that the container has not aligned with the external truck and cannot be successfully placed, requiring further fine-tuning. Once it is predicted that successful placement is possible, the spreader can be controlled to complete the placement operation. Subsequent operations are selected based on the predicted success rate of placement, avoiding the possibility of incorrect placement in case of failure. This method effectively compensates for the current shortcomings of inaccurate coarse alignment during external truck container placement, forming a closed-loop external truck container placement control system and improving the success rate of placement. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a system architecture diagram of the external card holder control method provided in an embodiment of the present invention; Figure 2 This is a structural block diagram of the computer device 102 provided in an embodiment of the present invention; Figure 3 A schematic flowchart illustrating the external card holder control method provided in an embodiment of the present invention; Figure 4 A schematic flowchart of step S301 provided in an embodiment of the present invention; Figure 5 A schematic diagram of a training image provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the detection box corner and lock head provided in an embodiment of the present invention; Figure 7 This is a functional block diagram of the external card box control device provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0012] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0013] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0014] In the description of this invention, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0015] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0016] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0017] To address the low success rate of current automatic container loading methods for external container trucks, this invention provides a container loading control method. This method uses vision to perform servo fine-tuning of the spreader, and after the fine-tuning is completed, it predicts whether the container loading will be successful. This allows for closed-loop control of the loading process, which can greatly improve the loading success rate. Please see Figure 1 , Figure 1 The system architecture diagram of the external parameter control method for the external gearbox provided in this embodiment of the invention includes a camera 101 and a computer device 102. The camera 101 is deployed at the four corners of the work lane. A calibration plate with multiple feature points can be placed on the work lane. The calibration plate is placed in the center of the lane to calibrate the camera 101's external parameters.

[0018] Camera 101 may include cameras 1 to 4. The lenses of cameras 1 and 2 are positioned in the opposite direction to the direction of the truck's movement, while the lenses of cameras 3 and 4 are positioned in the direction of the truck's movement, ensuring that the locks on the flatbed of the outer truck and the corners of the container are visible in the camera's field of view. Cameras 1 to 4 can all acquire high-resolution images of the lane and the outer truck, and transmit the acquired high-resolution images to a computer device.

[0019] Computer device 102 is used to perform feature detection on high-resolution images from a camera using deep learning methods. This enables accurate identification of the locks on the outer truck pallet and the corners of the container, which is simpler and more accurate than LiDAR. Computer device 102 is also used to control the fine-tuning of the spreader using visual servo control methods, allowing the container to align with the outer truck, achieving fine-tuned container placement and improving the success rate of container placement.

[0020] Please see Figure 2 , Figure 2This is a structural block diagram of a computer device 102 provided in an embodiment of the present invention. The computer device 102 is used to execute the external card holder control method provided in this embodiment of the present invention. The computer device 102 includes: a memory 1021, a processor 1022, a communication interface 1023, and a bus 1024. The memory 1021, processor 1022, and communication interface 1023 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0021] Optionally, the bus 1024 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0022] In this embodiment, the processor 1022 may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this embodiment. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this embodiment can be directly implemented by the hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules may reside in the memory 1021. The processor 1022 reads the program instructions from the memory 1021 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0023] In this embodiment, the memory 1021 can be non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or it can be volatile memory, such as RAM. The memory can also be any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. The memory in this embodiment can also be a circuit or any other device capable of implementing storage functions, used to store instructions and / or data.

[0024] The memory 1021 can be used to store software programs and modules, such as the instructions / modules of the external card holder control device 300 provided in this embodiment of the invention. These can be stored in the memory 1021 in the form of software or firmware, or embedded in the operating system (OS) of the computer device 1020. The processor 1022 executes various functional applications and data processing by executing the software programs and modules stored in the memory 1021. The communication interface 1023 can be used to communicate with other node devices for signaling or data.

[0025] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0026] Understandable. Figure 2 The structure shown is for illustrative purposes only; the computer device 102 may also include more than [other components]. Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown. Figure 2 The components shown can be implemented using hardware, software, or a combination thereof.

[0027] The computer device 102 can be any electronic product that can interact with the user, such as a personal computer, tablet computer, personal digital assistant (PDA), etc.

[0028] The computer device 102 may also include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0029] The network in which the computer device 102 is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0030] Based on the above embodiments, this application also provides a storage medium storing a computer program. When the computer program is executed by a computer, it causes the computer to execute the external card box control method provided in the above embodiments.

[0031] The following will be based on Figure 2 The computer device 102 in this invention is the executing entity. This paper describes the external card holder control method provided in this embodiment of the invention. Please refer to [link to relevant documentation]. Figure 3 , Figure 3A schematic flowchart of the external locking box control method provided in the embodiments of the present invention may include the following steps: S301. Determine the pose of the external container in the world coordinate system based on the pre-calibrated camera extrinsic parameters and external container image; In this embodiment of the invention, the pose of the outer container card includes the coordinates and rotation angle of the outer container card.

[0032] S302. Determine the pose of the spreader in the world coordinate system, and fine-tune the pose of the spreader based on the difference between the poses of the external container and the spreader in the same coordinate system. In this embodiment of the invention, the coordinates and rotation angle of the lifting device can be finely adjusted based on the difference between the coordinates and rotation angles of the external container truck and the lifting device in the world coordinate system.

[0033] S303. After fine-tuning the spreader, determine the difference between the theoretical and actual pixel coordinates of each corner of the container, and predict whether the container can be successfully placed based on the difference. If yes, then execute S305; otherwise, execute S304. S304, Recalibrate camera extrinsic parameters, return to S301.

[0034] S305, Control the lifting device to lower the box; In the technical solutions of steps S301 to S305 above, the precise position and attitude of the external container truck in the world coordinate system are first determined based on the pre-calibrated camera extrinsic parameters and the image of the external container truck. This provides an accurate reference for subsequent calculation of the spreader adjustment amount and verification. Then, the pose of the spreader is fine-tuned based on the difference between the poses of the external container truck and the spreader in the same coordinate system, so that the container can be aligned with the external container truck. After the spreader is fine-tuned, the difference between the theoretical pixel coordinates and the actual pixel coordinates of each corner of the container is determined. If the difference is too large, it indicates that the container has not been aligned with the external container truck and cannot be successfully placed, requiring readjustment. When it is predicted that successful placement can be achieved, the spreader can be controlled to complete the placement operation. The subsequent operation is selected based on the prediction of whether successful placement can be achieved, avoiding the possibility of incorrect placement in the event of failure. This method can effectively make up for the current defect of inaccurate coarse alignment when placing containers on external container trucks, forming a closed-loop control of container placement and improving the success rate of placement.

[0035] The following will provide a detailed introduction and explanation of each of the above steps.

[0036] In step S301, the external container card needs to be accurately located based on the pre-calibrated camera extrinsic parameters and the external container card image. Therefore, before performing step S301, it is necessary to first... Figure 1 Each camera deployed in the system needs to have its camera parameters calibrated, and the calibrated cameras need to be used to acquire images from the external graphics card. These two processes will be described below.

[0037] First, for each camera, internal parameters such as focal length, stationary point position, and distortion parameters can be determined using parameters provided by the camera manufacturer or through camera calibration tools (such as a camera calibration board). Then, after the internal parameter calibration is completed, the camera's extrinsic parameters are calibrated. This invention provides the following optional implementation method for camera extrinsic parameter calibration: Step a1: Use cameras deployed at the four corners of the work lane to acquire images of the work lane; wherein, a calibration plate is placed at the center of the work lane; the calibration plate includes multiple feature points; like Figure 1 As shown, a large calibration plate with 9 or 16 feature points can be selected and placed in the center of the lane. Then, the coordinates of the center of the lane in the world coordinate system and the position of each feature point on the calibration plate in the world coordinate system can be determined.

[0038] In an optional implementation, the calibration plate can be placed sequentially at different locations in the lane to ensure that the calibration plate at each location is visible in the field of view of each camera, and the coordinates of each location in the world coordinate system are recorded.

[0039] Step a2: Perform feature point detection in the work lane image and determine the pixel coordinates of each feature point; In this embodiment of the invention, after the calibration board is positioned in the lane, each camera can simultaneously acquire images of the calibration board. Then, the acquired images are corrected using the camera's intrinsic parameters. Finally, the corrected external card image is input into the pre-trained feature detection model in this embodiment of the invention for detection, which can determine the pixel coordinates of each feature point in the image.

[0040] Step a3: Based on the coordinates of the feature points in the world coordinate system and their pixel coordinates, calibrate the camera extrinsic parameters; Step a2 yields the coordinates of the feature points in the world coordinate system, and step a3 yields the pixel positions of the feature points in the images captured by each camera. Then, algorithms in OpenCV or the least squares method can be used to solve for the extrinsic parameters (rotation and translation matrices) of each camera relative to the world coordinate system, i.e., the camera extrinsic parameters.

[0041] Understandably, after camera deployment, the camera's extrinsic parameters can be calibrated using the methods described above. The calibrated extrinsic parameters, along with the previously calibrated intrinsic parameters, can be directly stored for easy maintenance. When automatic container loading is required, the following steps should be performed: Step a4: Once it is determined that the external container truck is stationary in the work lane, use the calibrated camera to acquire images of the external container truck.

[0042] It is understandable that once the external container truck starts operating, it will enter the operating lane. Therefore, once the external container truck stops, image acquisition can be performed on it.

[0043] In an optional implementation, the lock head of the outer container truck can be identified in the acquired image of the outer container truck. Once it is determined that the pixel position of the lock head no longer changes, it can be confirmed that the outer container truck is in a stationary state. Then, the pre-calibrated camera intrinsic and extrinsic parameters can be directly called to execute step S301 to determine the precise pose of the outer container truck in the world coordinate system.

[0044] Regarding step S301, this embodiment of the invention provides an implementation method, please refer to [link to relevant documentation]. Figure 4 , Figure 4 A schematic flowchart of step S301 provided in the embodiments of the present invention may include the following steps: S301-1: Input the image of the outer card into the pre-trained feature detection model to detect the lock head and determine the pixel coordinates of each lock head of the outer card; In this embodiment of the invention, the images of the external container trucks are captured by cameras deployed around the operating lane after the external container trucks arrive at their operating positions. The feature detection model can be used to detect the locks on the external container truck pallets and the corners of the containers. The feature detection model is pre-trained using the following method: Step b1: Acquire multiple training images; In this embodiment of the invention, each training image includes the lock label of the external container truck and the corner label of the container; during implementation, images of the external container truck pallet and the container can be collected at the work site, and the locks of the external container truck and the corner labels of the container can be labeled to obtain training images. Figure 5 As shown, Figure 5 This is a schematic diagram of a training image provided in an embodiment of the present invention. In this embodiment, the corners of a container and the locks of an external truck are used as feature points to be identified for model training. Step b2: Construct the initial convolutional neural network; In this embodiment of the invention, a neural network architecture for object detection, such as a convolutional neural network, can be constructed by designing the number and connection methods of convolutional layers, pooling layers, fully connected layers, etc., and by setting the network depth and complexity.

[0045] Step b3: Train the convolutional neural network based on the training images until the positioning error of the lock head and the corner of the box is less than the preset error threshold, and obtain the feature detection model.

[0046] During training, the training images can be divided into training, validation, and test sets. Then, a loss function (such as MSE) is constructed and an optimization algorithm (such as Adam) is selected to train the initial convolutional neural network until the positioning error of the container corner and the lock of the outer truck is less than a preset error threshold, such as 3 pixels. This indicates that the training is complete and the feature detection model is obtained.

[0047] The trained feature detection model can be used to detect box corners and lock heads in the external card images captured by each camera, accurately determining the pixel coordinates of the box corners and lock heads under each camera. For details, please refer to [link to documentation / reference]. Figure 6 , Figure 6 This is a schematic diagram illustrating the detection of box corners and lock heads provided in an embodiment of the present invention. The process includes: inputting a collected image of the outer container card into a feature detection model; extracting a feature map from the image through convolution in the feature detection model; inputting the feature map into a region generation network; marking detection boxes for the lock head and box corner on the feature map; and outputting a feature map with detection boxes. The center of each detection box for the lock head and box corner can be used as their respective pixel coordinates. The pose of the outer container card can be determined using the detected pixel coordinates of the lock head.

[0048] S301-2: Determine the world coordinates of each lock head based on the pixel coordinates of each lock head and the camera's extrinsic parameters; In this embodiment of the invention, the world coordinates of each lock head in the world coordinate system can be calculated using the pixel coordinates of each lock head in the images captured by any two cameras, based on the binocular positioning principle and camera extrinsic parameters. This world coordinate is denoted as p. l，i (i=1,..,N), where N is the total number of locks, and each lock is composed of P l (x l ,y l ,z l It consists of three dimensions, x l The coordinate y represents the lock head along the shorter side of the lane. l The z-coordinate represents the position of the lock head along the longer side of the lane. l This represents the coordinates of the lock head along the anti-gravity line of the lane.

[0049] S301-3: Determine the pose of the outer container card based on the world coordinates of each lock and the geometric relationship of the lock on the outer container card.

[0050] After obtaining the world coordinates of each lock, the pose of the outer container can be calculated using simple rectangular geometry, including the center coordinates and rotation angle of the outer container, denoted as: P. c (x c ,y c ,z c , ), where x c ,y c ,z c These represent the offset in the world coordinate system, x and x respectively. c The coordinate y represents the position of the outer truck along the shorter side of the lane. c The z-coordinate represents the coordinate of the outer truck along the longer side of the lane. c The coordinates representing the outer container truck along the anti-gravity line of the lane. This represents the angle between the outer truck and the long side of the lane.

[0051] After accurately positioning the external truck, the spreader can be fine-tuned to align the external truck and the container, see step S302.

[0052] In step S302, the pose of the spreader in the world coordinate system is first determined. In this embodiment of the invention, the pose of the spreader at any position can be determined using the center position of the lane. Specifically, the center of the spreader can be aligned with the center of the lane, and the position reading in the position encoder at this time can be recorded. This position reading can be considered aligned with the world coordinate system and used as a nominal point. When the spreader is at any position, the nominal point is subtracted from the current position reading in the position encoder to obtain the position of the spreader in the world coordinate system. At this time, the spreader will stop at a certain position above the working lane, and the pose can be represented as P. t (x t ,y t ,z t , ), The initial value is 0.

[0053] Using the obtained poses of the outer container truck and the spreader, the pose of the spreader can be fine-tuned to align the outer container truck and the container. Therefore, this embodiment of the invention provides an optional servo fine-tuning method, including: Step c1: Control the spreader to move to the target height range directly above the outer container truck; In this embodiment of the invention, the spreader can be controlled to move to a height h0 directly above the outer container truck, where h0 is within the target height range. The target height range can be defined according to actual needs, for example, from 0.1 meters to 0.3 meters.

[0054] Step c2: Based on the rotation angle in the pose of the outer container truck, control the spreader to twist according to the rotation angle; Step c3: Calculate the horizontal distance difference between the outer container truck and the spreader based on their positions and orientations, and control the spreader to move according to the horizontal distance difference.

[0055] It is understandable that the pose of the outer container is determined as P through step S301. c (x c,y c ,z c , At this time, the position of the lifting device is P. t (x t ,y t ,z t ,0), at this point the lifting device can be rotated Then calculate the horizontal distance difference as The spreader is controlled to move according to the horizontal distance difference, thereby achieving fine-tuning control of the spreader.

[0056] After fine-tuning the spreader, theoretically, the lock holes of the container and the lock heads of the outer truck should correspond one-to-one, except that the height in the z-direction is increased by h0. Therefore, this embodiment of the invention predicts whether the container can be successfully loaded by using the difference between the theoretical and actual pixels of the same target object in the camera, i.e., the corner of the container. Therefore, step S303 can be executed as follows: Step d1: Based on the pose of the outer container truck and the size information of the container, calculate the coordinates of each corner of the container in the world coordinate system, and use the camera intrinsic parameters to convert the coordinates into theoretical pixel coordinates. In this embodiment of the invention, the pose of the outer assembly card is P. c (x c ,y c ,z c , Given the length, width, and height of the container, the world coordinates of each container corner in the world coordinate system can be calculated. Then, based on the intrinsic parameter matrix of each camera, the theoretical pixel coordinates of each container corner in each camera can be calculated. , i=1,..,4.

[0057] Step d2: Input the image of the outer container truck acquired after the fine-tuning of the spreader into the pre-trained feature detection model for box corner detection to determine the actual pixel coordinates.

[0058] After fine-tuning the spreader, images of the external container truck can continue to be captured by each camera. It's understood that the spreader is now directly above the external container truck, and the captured images naturally include the container. These newly captured images are then input into a feature detection model for detection, yielding the actual pixel coordinates of each corner of the container under each camera's view. , i=1,..,4.

[0059] Step d3: If the difference between the theoretical pixel coordinates and the actual pixel coordinates is within the preset error range, then the binning is successful; otherwise, the binning is unsuccessful.

[0060] If the difference between the theoretical pixel coordinates and the actual pixel coordinates of the container corners in each camera If all parameters are within the preset error range, the spreader adjustment is considered successful, and the container can be successfully placed. If any one of these parameters is not met, the spreader adjustment is considered successful. If the error is outside the preset error range, the spreader fine-tuning is considered a failure, and the container cannot be successfully placed. Theoretically, as long as the cameras deployed around the work lane are successfully calibrated, the fine-tuning will definitely succeed. However, the failure of the fine-tuning indicates that the current extrinsic parameter calibration is incorrect and needs to be recalibrated. Therefore, the camera extrinsic parameters can be recalibrated, and the above steps S301 to S305 can be executed again using the recalibrated camera extrinsic parameters until the container can be successfully placed.

[0061] As can be seen from the above embodiments, compared with the prior art, the external assembly card box control method provided by the embodiments of the present invention has the following advantages: First, compared to traditional LiDAR solutions, this invention deploys cameras around the perimeter of the work lane. These cameras provide high-resolution images, while LiDAR has relatively low resolution. High resolution improves the final positioning accuracy, and the cameras provide rich detail information such as color and contours. Thanks to the rapid development of deep learning, the recognition and detection of features like locks and corners is simpler and more accurate than with LiDAR. Furthermore, cameras are less affected by extreme weather conditions like rain and snow, eliminating the need for complex algorithms to handle noise caused by weather interference with LiDAR, thus speeding up processing. Moreover, the cameras are fixed at the four corners of the lane, making them less susceptible to the movement of the crane (which can cause sensor vibration, shaking, and detachment) compared to mounting them on a trolley frame. Therefore, once the extrinsic parameters are calibrated, they generally require no modification, and subsequent replacement and maintenance are much simpler. Cameras are significantly cheaper than LiDAR, and LiDAR has a narrow field of view, requiring more LiDAR units to cover the same scene. Using a camera solution effectively reduces overall deployment costs.

[0062] Secondly, this invention employs a visual servo control approach for fine-tuning the spreader, effectively overcoming the current shortcomings of inaccurate coarse alignment during container placement by external trucks, thus forming a closed-loop control system for container placement. Specifically, after the external truck stops in the work lane, its world coordinates are determined based on pre-calibrated camera extrinsic parameters and the acquired image of the external truck. Then, the spreader's translation and rotation are controlled according to these world coordinates to align the external truck with the container. Furthermore, after fine-tuning, this invention can verify successful container placement by comparing the theoretical and actual pixel coordinates of the container's corners in the camera. If successful, the placement operation is executed; otherwise, camera calibration is repeated, and fine-tuning is performed again, achieving a closed-loop placement scheme that improves the success rate of container placement.

[0063] Based on the same inventive concept, embodiments of the present invention also provide an external locking box control device, please refer to [link to relevant documentation]. Figure 7 , Figure 7This is a functional block diagram of an external container holding box control device 400 provided in an embodiment of the present invention. The external container holding box control device 400 may include: a positioning module 410, a control module 420, an inspection module 430, and a calibration module 440. The positioning module 410 is used to determine the pose of the external carrier in the world coordinate system based on the pre-calibrated camera extrinsic parameters and the external carrier image; The control module 420 is used to determine the pose of the spreader in the world coordinate system and to fine-tune the pose of the spreader based on the difference between the pose of the outer container and the spreader in the same coordinate system. The inspection module 430 is used to determine the difference between the theoretical pixel coordinates of each corner of the container before the fine-tuning and the actual pixel coordinates after the fine-tuning after the fine-tuning, and to predict whether the container can be successfully loaded based on the difference. The control module 420 is also used to control the spreader to lower the box if the result of the inspection module 430 is yes; the calibration module 440 is used to recalibrate the camera extrinsic parameters and return to the positioning module if the result of the inspection module 430 is no.

[0064] It is understandable that the positioning module 410, control module 420, inspection module 430, and calibration module 440 can perform collaboratively. Figure 3 Each step in the process is used to achieve the corresponding technical effect.

[0065] In an optional implementation, the verification module 430 is specifically used to: calculate the coordinates of each container corner in the world coordinate system based on the pose of the external container truck and the size information of the container, and convert the coordinates into theoretical pixel coordinates using camera intrinsic parameters; input the external container truck image acquired after the spreader is fine-tuned into a pre-trained feature detection model for container corner detection to determine the actual pixel coordinates; if the difference between the theoretical pixel coordinates and the actual pixel coordinates is within a preset error range, it is determined that the container landing can be successfully completed; otherwise, it is determined that the container landing cannot be successfully completed.

[0066] In an optional implementation, the positioning module 410 is specifically used to input the image of the outer card into a pre-trained feature detection model for lock detection, determine the pixel coordinates of each lock on the outer card; determine the world coordinates of each lock based on the pixel coordinates of each lock and the camera extrinsic parameters; and determine the pose of the outer card based on the world coordinates of each lock and the geometric relationship of the lock on the outer card.

[0067] In an optional embodiment, the external container truck loading control device 400 further includes an acquisition module, which is used to acquire images of the working lane using cameras deployed at the four corners of the working lane; wherein a calibration plate is placed at the center of the working lane; the calibration plate includes multiple feature points; the calibration module 440 is also used to perform feature point detection in the working lane image, determine the pixel coordinates of each feature point; and calibrate the camera extrinsic parameters based on the coordinates of the feature points in the world coordinate system and the pixel coordinates; the acquisition module is also used to acquire images of the external container truck using the calibrated camera when it is determined that the external container truck is stationary in the working lane.

[0068] In an optional implementation, the control module 420 is specifically used to: fine-tune the position of the spreader according to the position of the outer container truck, including: controlling the spreader to move to a target height range directly above the outer container truck; controlling the spreader to twist according to the rotation angle in the position of the outer container truck; calculating the horizontal distance difference between the outer container truck and the spreader according to the position of the outer container truck and the spreader, and controlling the spreader to move according to the horizontal distance difference.

[0069] In an optional implementation, the control module 420 is further specifically used to: read the current position reading of the spreader in the position encoder, and use the difference between the current position reading and the reference position data as the pose of the spreader in the world coordinate system; wherein, the reference position is the data recorded in the position encoder when the center of the spreader coincides with the center of the working lane.

[0070] In an optional implementation, the feature detection model is trained as follows: multiple training images are acquired; each training image includes a lock tag on the outer truck and a corner tag on the container; an initial convolutional neural network is constructed; the convolutional neural network is trained based on the training images until the positioning error of the lock and the corner is less than a preset error threshold, thus obtaining the feature detection model.

[0071] It should be noted that the module division in the above embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical entities, or have two or more units integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] Based on the above embodiments, this application also provides a computer program that, when run on a computer, causes the computer to execute the external card box control method provided in the above embodiments.

[0074] Based on the above embodiments, this application also provides a chip for reading a computer program stored in a memory and executing the external card box control method provided in the above embodiments.

[0075] This application also provides a computer program product, including instructions that, when run on a computer, cause the computer to execute the external card box control method provided in the above embodiments.

[0076] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by instructions. These instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for controlling an externally mounted jamming box, characterized in that, The method includes: The pose of the external container in the world coordinate system is determined based on the pre-calibrated camera extrinsic parameters and the external container image. Determine the pose of the spreader in the world coordinate system, and fine-tune the pose of the spreader based on the difference between the poses of the external container and the spreader in the same coordinate system; Based on the pose of the external container truck and the size information of the container, the coordinates of each container corner in the world coordinate system are calculated, and the coordinates are converted into theoretical pixel coordinates using camera intrinsic parameters; the external container truck image acquired after the spreader is fine-tuned is input into a pre-trained feature detection model for container corner detection to determine the actual pixel coordinates; if the difference between the theoretical pixel coordinates and the actual pixel coordinates is within a preset error range, it is determined that the container landing can be successfully completed; otherwise, it is determined that the container landing cannot be successfully completed. If the container is successfully landed, the lifting device is controlled to land the container; otherwise, the camera extrinsic parameters are recalibrated, and the process returns to the step of determining the pose of the external container in the world coordinate system based on the pre-calibrated camera extrinsic parameters and the external container image.

2. The external assembly jamming box control method according to claim 1, characterized in that, Determine the pose of the external transport vehicle in the world coordinate system based on pre-calibrated camera extrinsic parameters and external transport vehicle images, including: The image of the external card is input into a pre-trained feature detection model to detect locks and determine the pixel coordinates of each lock on the external card. The world coordinates of each lock head are determined based on the pixel coordinates of each lock head and the camera extrinsic parameters; The pose of the outer container is determined based on the world coordinates of each lock head and the geometric relationship of the lock head on the outer container.

3. The external jamming box control method according to claim 2, characterized in that, Before determining the pose of the external carrier in the world coordinate system based on pre-calibrated camera extrinsic parameters and external carrier images, the method further includes: Images of the work lane are captured using cameras deployed at the four corners of the work lane; a calibration plate is placed at the center of the work lane; the calibration plate includes multiple feature points; Feature point detection is performed on the image of the work lane to determine the pixel coordinates of each feature point; The camera extrinsic parameters are calibrated based on the coordinates of the feature points in the world coordinate system and the pixel coordinates. When it is determined that the external container truck is stationary in the work lane, the calibrated camera is used to capture images of the external container truck.

4. The external jamming box control method according to claim 1, characterized in that, The pose of the lifting device is fine-tuned based on the difference in pose between the external assembly card and the lifting device in the same coordinate system, including: Control the spreader to move to a target height range directly above the outer container truck; Based on the rotation angle in the pose of the outer container truck, control the spreader to twist according to the rotation angle; Based on the positions of the outer container truck and the lifting device, the horizontal distance difference between the outer container truck and the lifting device is calculated, and the lifting device is controlled to move according to the horizontal distance difference.

5. The external assembly jamming box control method according to claim 4, characterized in that, Determining the pose of the spreader in the world coordinate system includes: The current position reading of the spreader is read from the position encoder, and the difference between the current position reading and the reference position data is taken as the pose of the spreader in the world coordinate system; wherein, the reference position is the data recorded in the position encoder when the center of the spreader coincides with the center of the work lane.

6. The external jamming box control method according to claim 2, characterized in that, The feature detection model is trained in the following way: Acquire multiple training images; wherein each training image includes the lock tag of the outer container truck and the corner tag of the container. Construct the initial convolutional neural network; The convolutional neural network is trained based on the training images until the positioning error of the lock head and the box corner is less than a preset error threshold, thus obtaining the feature detection model.

7. An external locking box control device, characterized in that, include: Positioning module, control module, inspection module, and calibration module; The positioning module is used to determine the pose of the external container in the world coordinate system based on the pre-calibrated camera extrinsic parameters and the external container image; The control module is used to determine the pose of the spreader in the world coordinate system, and to fine-tune the pose of the spreader based on the difference between the pose of the outer container and the spreader in the same coordinate system. The inspection module is used to calculate the coordinates of each container corner in the world coordinate system based on the pose of the external container truck and the size information of the container, and convert the coordinates into theoretical pixel coordinates using camera intrinsic parameters; input the external container truck image acquired after the spreader is fine-tuned into a pre-trained feature detection model for container corner detection to determine the actual pixel coordinates; if the difference between the theoretical pixel coordinates and the actual pixel coordinates is within a preset error range, it is determined that the container landing can be successfully completed; otherwise, it is determined that the container landing cannot be successfully completed. The control module is further configured to control the lifting device to place the container if the result of the inspection module indicates that the container can be successfully placed; the calibration module is configured to recalibrate the camera extrinsic parameters and return to the positioning module if the result of the inspection module indicates that the container cannot be successfully placed.

8. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the external card holder control method according to any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the external card box control method as described in any one of claims 1 to 6.

Images