Double-box operation judgment method and system and field bridge

Abstract

The invention provides a double-box operation judgment method and system and a field bridge, and relates to the technical field of field bridge automation. The method comprises the following steps: acquiring double-box point cloud data of a to-be-operated area; the double-container point cloud data are processed, a first corner point and a second corner point of the first container and a third corner point and a fourth corner point of the second container are recognized, and the first corner point, the second corner point, the third corner point and the fourth corner point are located on the same side of the double containers; based on the first angular point, the second angular point, the third angular point and the fourth angular point, pose parameters are determined, and the pose parameters are used for indicating the relative pose relation between the first container and the second container; according to the pose parameters, whether the pose relation between the first container and the second container meets the double-container operation condition or not is judged, and a judgment result is generated. According to the method, the feasibility of double-container operation is pre-judged by accurately identifying the angular points of the containers and determining the pose parameters, so that the field bridge can plan the operation path and mode more efficiently, and the double-container operation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of yard crane automation technology, and in particular to a method, system and yard crane for determining dual-box operation. Background Technology

[0002] At container terminals, yard cranes (container gantry cranes) are key loading and unloading equipment, undertaking important tasks such as stacking and moving containers within the yard. Traditional yard cranes operate on a single-container mode, grabbing only one container at a time, resulting in limited operational efficiency. To improve loading and unloading efficiency, dual-container operation technology has emerged, allowing yard cranes to grab two containers simultaneously, thereby increasing operational efficiency.

[0003] Currently, automated yard crane operations involving two containers primarily rely on sensors on the spreader for real-time detection and feedback. For example, the trolley moves to the target position, lowers the two-container spreader to a certain height, and then the radar on the spreader scans the target container. Based on the scan results, it is determined whether the spreader can perform a two-container operation. However, this method cannot predict the status of containers in the yard in advance, resulting in low operational efficiency. Summary of the Invention

[0004] The dual-container operation judgment method, system, and yard crane provided in this application are used to solve the problem that existing methods cannot predict the status of containers in the yard in advance, resulting in low operation efficiency.

[0005] In a first aspect, embodiments of this application provide a method for determining dual-box operations, including:

[0006] Acquire dual-box point cloud data of the area to be worked on;

[0007] The point cloud data of the two containers is processed to identify the first and second corner points of the first container, as well as the third and fourth corner points of the second container, wherein the first, second, third, and fourth corner points are located on the same side of the two containers.

[0008] Based on the first corner point, the second corner point, the third corner point, and the fourth corner point, the pose parameters are determined. The pose parameters are used to indicate the relative pose relationship between the first container and the second container.

[0009] Based on the pose parameters, determine whether the pose relationship between the first container and the second container meets the conditions for dual-container operation, and generate the determination result.

[0010] In one possible implementation, acquiring dual-box point cloud data of the area to be worked on includes:

[0011] Initial point cloud data is collected by at least two sensor devices deployed on opposite sides below the vehicle platform in the field bridge, wherein each sensor device includes multiple lidars;

[0012] Preprocess the initial point cloud data to obtain the dual-box point cloud data of the area to be operated on.

[0013] In one possible implementation, the initial point cloud data is preprocessed to obtain dual-box point cloud data of the area to be processed, including:

[0014] The initial point cloud data is subjected to coordinate correction to obtain the corrected point cloud data;

[0015] Based on the preset dual-box operation task type, the dual-box point cloud data of the area to be operated is dynamically cropped from the correction point cloud data.

[0016] In one possible implementation, the pose parameters are determined based on the first corner point, the second corner point, the third corner point, and the fourth corner point, including:

[0017] Determine the first slope of the first container based on the first and second corner points;

[0018] Determine the second slope of the second container based on the third and fourth corner points;

[0019] The difference between the first slope and the second slope is taken as the yaw angle difference;

[0020] Determine the directional deviation of the trolley based on the second and third corner points;

[0021] The pose parameters include the yaw angle difference and the vehicle's directional deviation.

[0022] In one possible implementation, based on the pose parameters, it is determined whether the pose relationship between the first container and the second container satisfies the conditions for dual-container operation, and a determination result is generated, including:

[0023] The yaw angle difference is compared with a preset first threshold, and the trolley direction deviation is compared with a preset second threshold.

[0024] If the yaw angle difference is less than the first threshold and the trolley direction deviation is less than the second threshold, then a judgment result that satisfies the dual-box operation conditions is generated.

[0025] If the yaw angle difference is greater than or equal to the first threshold, and / or the trolley direction deviation is greater than or equal to the second threshold, a judgment result indicating that the dual-box operation conditions are not met is generated.

[0026] In one possible implementation, the first threshold and / or the second threshold are adaptive thresholds that are dynamically adjusted by a machine learning model; the machine learning model is trained based on historical double-container operation data, which includes pose parameters and container information corresponding to successful and / or failed operations.

[0027] In one possible implementation, the method further includes:

[0028] Acquire image data of the area to be processed;

[0029] Based on image data, the first visual features of the first container and the second visual features of the second container are determined.

[0030] Accordingly, the point cloud data of the two containers is processed to identify the first and second corner points of the first container, and the third and fourth corner points of the second container, including:

[0031] Based on the first visual features and the second visual features, the point cloud data of the two containers is fitted and segmented to obtain the first point cloud cluster of the first container and the second point cloud cluster of the second container.

[0032] Based on the preset corner detection algorithm, corner detection is performed on the first point cloud cluster and the second point cloud cluster respectively to obtain the first corner and the second corner of the first container, as well as the third and fourth corners of the second container.

[0033] In one possible implementation, the method further includes:

[0034] When the judgment result indicates that the positional relationship between the first container and the second container meets the conditions for dual-container operation, the identity matching verification of the first container and the second container is performed based on the first visual feature and the second visual feature to obtain the verification result.

[0035] If the verification result indicates that the identity matching verification failed, an error message indicating an abnormal judgment result will be generated.

[0036] Secondly, embodiments of this application provide a dual-box operation judgment system, including:

[0037] At least two sensor devices are deployed on opposite sides below the platform of the trolley in the yard bridge to collect point cloud data and image data of the area to be worked on;

[0038] The data processing module is communicatively connected to the sensor device and is used to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0039] Thirdly, embodiments of this application provide a yard crane, including the dual-box operation judgment system as described in the second aspect above.

[0040] Fourthly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0041] The memory stores the instructions that the computer executes;

[0042] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0043] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0044] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed, implements the first aspect and / or various possible implementations of the first aspect.

[0045] The dual-container operation judgment method, system, and yard crane provided in this application embodiment acquire dual-container point cloud data of the area to be operated; process the dual-container point cloud data to identify the first and second corner points of the first container, and the third and fourth corner points of the second container, wherein the first, second, third, and fourth corner points are located on the same side of the two containers; determine pose parameters based on the first, second, third, and fourth corner points, which are used to indicate the relative pose relationship between the first and second containers; and determine whether the pose relationship between the first and second containers meets the dual-container operation conditions based on the pose parameters, and generate a judgment result. By using point cloud data to accurately identify the corner points of the containers and determine the pose parameters, the success of the dual-container operation can be predicted. This allows the yard crane to understand the possibility of dual-container operation before approaching the containers, thereby planning the operation path and mode more efficiently, avoiding unnecessary movement and waiting time, and improving the efficiency of the yard crane's dual-container operation. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 A flowchart illustrating the dual-box operation judgment method provided in this application;

[0048] Figure 2 This application provides a schematic diagram of the identification results of the four corner points of the dual-box container. Figure 1 ;

[0049] Figure 3 This application provides a schematic diagram of the identification results of the four corner points of the dual-box container. Figure 2 ;

[0050] Figure 4 A schematic diagram illustrating the pose relationship between the two boxes provided in this application;

[0051] Figure 5 This application provides a schematic diagram of the deployment structure of the sensor device.

[0052] Figure 6 This is a schematic diagram of the dual-box operation judgment device provided in this application;

[0053] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0054] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0056] In existing technologies, during double-container operations by a yard crane, the trolley moves to the target location and lowers the double-container spreader to a certain height. As it approaches the containers, radar or other sensors on the spreader scan the target containers, collecting information about their position, angle, etc. The yard crane then determines whether double-container operation is possible based on the sensor detection results. However, because the sensors can only detect containers in real-time as the spreader approaches or reaches the target location, the yard crane cannot anticipate the specific situation of the containers in the yard, such as their position and alignment. This lack of predictive capability leads to situations where the yard crane discovers it cannot perform double-container operations only after approaching or reaching the work position, resulting in wasted time, energy, and resources and low operational efficiency. Furthermore, the lack of a predictive mechanism may cause the yard crane to frequently switch operating modes when double-container operations are not possible, further impacting operational efficiency and the level of intelligence of the yard crane.

[0057] To address the aforementioned issues, this application provides a method, system, and yard crane for determining dual-container operations. By acquiring point cloud data of the dual-container area to be operated, detailed three-dimensional information of the containers is captured, providing a reliable data foundation for subsequent accurate identification of container corners and determination of their positional relationships. Through processing the point cloud data, four corners of the two containers on the same side are identified to accurately capture the relative positional relationship between the two containers. Based on these corners, the relative positional parameters between the containers are determined, thereby predicting the feasibility of dual-container operations. This allows the yard crane to understand the possibility of dual-container operations before approaching the containers, enabling more efficient planning of operation paths and patterns, avoiding unnecessary movement and waiting time, and improving the efficiency of dual-container operations by the yard crane.

[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0059] The execution subject of the dual-container operation judgment method provided in this application embodiment can be a computing device such as a server or server cluster. The server can be a mobile phone, computer, tablet, or other device. This application embodiment does not impose any particular restrictions on the implementation method of the execution subject, as long as the execution subject can obtain the dual-container point cloud data of the area to be operated; process the dual-container point cloud data to identify the first and second corner points of the first container, and the third and fourth corner points of the second container, wherein the first, second, third, and fourth corner points are located on the same side of the two containers; based on the first, second, third, and fourth corner points, determine the pose parameters, which are used to indicate the relative pose relationship between the first and second containers; according to the pose parameters, determine whether the pose relationship between the first and second containers meets the dual-container operation conditions, and generate a judgment result.

[0060] In some embodiments, the executing entity of the dual-box operation judgment method can be a dual-box operation judgment system. This dual-box operation judgment system may include a data processing module and at least two sensor devices. The sensor devices are deployed on opposite sides below the trolley platform in the yard crane, capable of comprehensively and accurately collecting relevant raw data of the area to be operated. The type of data collected is related to the type of device in the sensor device; for example, if the sensor device includes LiDAR, the collected data may include point cloud data; when the sensor device includes multiple types of devices such as LiDAR and cameras, the collected data may include multi-dimensional data such as point cloud data and image data. The data processing module is communicatively connected to the sensor devices and is used to execute the dual-box operation judgment method.

[0061] For example, in a real-world port crane operation scenario, the dual-container operation judgment system starts operating. Sensor devices quickly collect point cloud data of the area to be operated and transmit the data to the data processing module. The data processing module first performs noise reduction and segmentation on the point cloud data, identifies the corner points of the first and second containers, then determines the pose parameters based on these corner points, and finally judges whether the dual-container operation conditions are met based on the parameters, generates the corresponding judgment result, and guides the crane to perform subsequent operations (such as dual-container operation, splitting into single containers, etc.).

[0062] Figure 1 This is a flowchart illustrating the dual-box operation judgment method provided in this application, as shown below. Figure 1 As shown, the method may include:

[0063] S101. Obtain the point cloud data of the dual-box area to be operated.

[0064] The area to be operated can refer to the specific area where the yard crane is prepared to carry out container loading and unloading operations. The dual-container point cloud data can refer to the data representing the three-dimensional contours and positions of two containers and their surrounding environment, which are presented in the form of a set of spatial coordinate points, obtained by sensor equipment (such as LiDAR, 3D laser scanner, etc.).

[0065] In some examples, mobile sensor devices can be used, mounted on equipment or devices that move with the operation as needed, such as yard crane trolleys or drones, and moved to the area to be worked on to scan as required. Fixed sensor devices can also be used, mounted at other fixed locations on the yard crane or in the yard, to scan the area to be worked on periodically or as needed. For example, in a container yard, before the yard crane prepares for a double-container operation (such as simultaneously grabbing two containers placed side-by-side), a LiDAR mounted under the trolley scans the double-container area to obtain point cloud data of the two containers.

[0066] The sensor devices can include multiple devices, such as those installed in different positions below the vehicle platform (e.g., upper left and lower right, or upper left, upper right, lower left and lower right). Each sensor device can also include multiple scanning devices such as LiDAR to avoid blind spots in the field of view of a single radar.

[0067] S102. Process the point cloud data of the two containers to identify the first and second corner points of the first container, and the third and fourth corner points of the second container, wherein the first, second, third and fourth corner points are located on the same side of the two containers.

[0068] In this step, point cloud data processing may include a series of operations such as coordinate correction, noise reduction, segmentation, and clustering, aiming to extract key feature points representing the edge contour of the container from the point cloud data, i.e., the required corner point information. In the point cloud data of the container, a corner point can refer to a specific point on the vertex or edge line of the container. The first container and the second container refer to two adjacent containers or containers placed in a certain positional relationship within the area to be operated. Corner points on the same side can refer to the corner points of two adjacent containers on the same side (usually along the length of the container) parallel to the running direction of the yard crane trolley. Figure 2 This application provides a schematic diagram of the identification results of the four corner points of the dual-box container. Figure 1 ,like Figure 2 As shown in the figure, the left side is the first container and the right side is the second container. The first corner point (a1) and the second corner point (a2) are two vertices on the same side of the first container. The third corner point (a3) ​​and the fourth corner point (a4) are two vertices on the same side of the second container as the first container (corresponding to the horizontal direction in the figure).

[0069] Optionally, the positions of the first corner point, second corner point, third corner point, and fourth corner point, in addition to... Figure 2 In addition to the diagonal vertices on the same side of the container shown, the points can also be other points on that side (i.e., the lines a1 to a2 and a3 to a4), such as... Figure 3 As shown, the first corner point is located Figure 2 As shown, the second corner point is located to the right of a1. Figure 2 To the left of a2 shown.

[0070] In one example, the point cloud can be cropped based on a pre-defined approximate area of ​​the double-container layout (e.g., the ROI region). Then, planar fitting (e.g., using the RANSAC algorithm) is performed on the point cloud to separate the container top surface. Next, edge detection algorithms can be used to extract the top surface contour, clearly representing the shape and extent of the container top surface. This provides crucial geometric clues for corner identification, transforming complex point cloud data into a simpler geometric representation like contour lines, facilitating subsequent corner identification and processing. After obtaining the container top surface contour, corners on the same side are determined based on the shape and positional relationship of the contour. Furthermore, for two adjacent containers, their top surface contours will have a certain relative positional relationship on adjacent sides, which can be determined by calculating the coordinates and direction information of points on the contour. For example, based on the container placement direction and the direction of the contour, it can be determined which corners are located on adjacent sides facing the same direction.

[0071] In another example, a 3D object detection neural network (such as VoteNet) can be trained using a point cloud dataset containing point cloud data of two containers with accurately labeled container corner positions. This dataset should cover point cloud data of two containers in various real-world scenarios, including different placement methods, distances, angles, and the presence of noise interference, to ensure that the trained network has good generalization ability. During training, the model learns the mapping relationship from the input point cloud data to the labeled corner positions, and continuously adjusts its parameters (such as the weights and biases of the convolutional kernels) to minimize the error between the predicted corner positions and the labeled true corner positions.

[0072] During the prediction phase, new point cloud data of the two containers is input into the trained model. The network extracts and analyzes features from the input point cloud and then outputs the predicted corner positions of the containers. These predicted corner positions are represented in coordinate form, with each corner having its corresponding three-dimensional coordinate value. Corner points belonging to the same side can be selected based on features such as the relative positional relationship, distance, and direction between them. Furthermore, since containers are usually cuboid in shape, corner points on the same side will have a certain arrangement pattern in space. Based on the predicted corner coordinates, the vector relationship between the corner points can be calculated. If the vector direction between two corner points matches the expected placement direction of the container, and the distance between them falls within the range of the side length of the container on the same side, then these two corner points can be determined to be corner points on the same side.

[0073] Alternatively, clustering algorithms (such as K-means clustering) can be used to cluster the predicted corner points or corner points on the contour line. Based on the coordinate information of the corner points, in a double-container scenario, corner points on the same side are clustered into one class. Furthermore, by setting an appropriate number of clusters (for example, for corner points on the same side of two containers, the number of clusters is set to 4, corresponding to two corner points on the same side of the two containers respectively), the required first, second, third, and fourth corner points can be further determined.

[0074] S103. Based on the first corner point, the second corner point, the third corner point, and the fourth corner point, determine the pose parameters, which are used to indicate the relative pose relationship between the first container and the second container.

[0075] In this step, the pose parameters can be one or more parameters used to describe the relative position and attitude relationship of the two containers in the horizontal plane. It directly determines whether the double container spreader can successfully grab or place the two containers at the same time.

[0076] In some embodiments, the yaw angle difference and trolley directional deviation can be calculated using the first, second, third, and fourth corner points to determine the pose parameters. The yaw angle difference reflects the difference in the rotation angle of the containers, while the trolley directional deviation reflects the alignment of the containers in the horizontal direction. For example, the side vector of the first container can be determined based on the coordinate information of the first and second corner points, and the side vector of the second container on the same side as the first container can be determined based on the coordinate information of the third and fourth corner points; then, the yaw angle difference is obtained based on the difference in the directional angles of the two side vectors. The corner points where the two containers are closest in the horizontal direction are determined, such as... Figure 2 The second and third corner points are closer in the x-direction. The directional deviation of the car is obtained based on the coordinate difference of the closest corner points in the vertical direction (such as the y-direction).

[0077] In other embodiments, the pose parameters can also be determined by center distance and angle. For example, the center points C1 and C2 of the two corner points on the same side of the two containers are calculated, and then the distance between C1 and C2 (center distance deviation) and the angle between the vector connecting C1 and C2 and the axis of the reference coordinate system are calculated.

[0078] In some embodiments, the corner point of the first container can be used as a reference to calculate a relative transformation matrix (such as a two-dimensional rigid body transformation matrix, which includes rotation and translation) that can transform the corner point to be optimally aligned with the corner point of the second container. The rotation angle and translation vector in the matrix can be directly used as pose parameters.

[0079] S104. Based on the pose parameters, determine whether the pose relationship between the first container and the second container meets the conditions for dual-container operation, and generate the judgment result.

[0080] The dual-container operation conditions refer to the specific requirements that the two containers need to meet in terms of position, angle, etc., to ensure that the yard crane can safely and effectively grab the two containers at the same time. These conditions can be further set as judgment conditions related to position and posture parameters. When the condition is met, it means that the two containers meet the requirements in terms of position, angle, etc., and can ensure that the yard crane can safely and effectively grab them at the same time. If the condition is not met, subsequent processes such as splitting into single-container operations, alarms, or repositioning can be triggered based on the judgment result.

[0081] For example, the preset conditions for dual-box operation set thresholds for each pose parameter. If the calculated pose parameter is less than these thresholds, the dual-box operation is deemed feasible; otherwise, it is deemed infeasible. For instance, if the yaw angle difference θ < 0.03 and the trolley direction deviation d < 50mm, the conditions are met, and a judgment result of "dual-box operation is possible" is generated; otherwise, a judgment result of "dual-box operation is not recommended" is generated, and a return message is generated to trigger subsequent processes.

[0082] Optionally, when the dual-box operation conditions are at a preset threshold, an adaptive threshold model can be used to dynamically output a recommended threshold based on the current actual operation conditions (such as box type, wind speed, and other environmental information), thereby improving adaptability to different operating environments.

[0083] Figure 4 The schematic diagram of the dual-box pose relationship provided in this application is as follows: Figure 4 As shown, in a normal double-container operation, the two containers are aligned. Under normal conditions, the positional relationship between the two containers satisfies the double-container operation requirements, meaning that the positional parameters are small, such as the offset in the trolley direction and the difference in yaw angle between the two containers. If the positional relationship of the two containers does not meet the double-container operation requirements, it indicates that the two containers may be misaligned, offset, or a combination of misalignment and offset. In this case, the success rate of double-container operations may be affected, and the grabbing cannot be performed. Therefore, in this situation, there is no need to attempt to align the spreader; instead, information is returned directly, and the task is split into a single-container task to avoid unnecessary operation interruptions and improve operational efficiency.

[0084] The dual-container operation judgment method provided in this application acquires and processes dual-container point cloud data to accurately identify corner points on specific sides of the containers and determines the relative pose relationship between the containers based on these corner points, thereby accurately judging the feasibility of dual-container operations. This not only improves the accuracy and efficiency of dual-container operation prediction but also avoids operation interruptions and resource waste caused by misjudgments. Furthermore, because rapid prediction can be made before operations, yard cranes can plan operation paths and modes more efficiently, thereby avoiding unnecessary waiting and adjustment time and improving the overall efficiency of dual-container operations.

[0085] Based on the above embodiments, the method for obtaining dual-box point cloud data of the area to be operated as described in S101 may include: collecting initial point cloud data by using at least two sensor devices deployed on opposite sides below the trolley platform of the yard bridge, wherein each sensor device includes multiple lidars; and preprocessing the initial point cloud data to obtain dual-box point cloud data of the area to be operated.

[0086] In this embodiment, in the yard crane structure, the trolley platform can be a mechanism for supporting the movement of the trolley. The trolley can move along a specific direction on the platform to realize operations such as handling containers at different locations. It has a certain load-bearing capacity and motion accuracy, and is one of the key components of the yard crane to realize container loading and unloading operations.

[0087] LiDAR (Light Detection and Ranging) can acquire three-dimensional information about the surrounding environment by emitting laser beams and receiving reflected signals, thereby generating point cloud data. The initial point cloud data is the raw data directly collected by the LiDAR in the sensor device. This data contains the three-dimensional coordinate information of all object surface points within the sensor's field of view, and may also contain some noise data, invalid data, and point cloud information of other objects unrelated to the area to be worked on (such as the surrounding environment, other equipment, etc.).

[0088] Preprocessing refers to a series of operations performed on the initial point cloud data. The purpose is to remove noise and invalid data, and to correct and optimize the data to obtain more accurate point cloud data of the dual-container area that better meets the requirements of subsequent processing. For example, in radar data with adjacent or overlapping views, common features (such as container edges and corners) are automatically identified; the relative pose transformation relationship between different radar point clouds is calculated using feature matching algorithms, thereby achieving data alignment and fusion. Another example is that before and after coordinate fusion, filtering algorithms are applied to remove invalid points caused by dust, rain, small debris, or sensor noise, retaining the main container structural point cloud.

[0089] The purpose of deploying sensors below the vehicle platform is to obtain a vertically downward view, directly scanning the top of the container below. This allows the point cloud data to primarily reflect the geometric features of the container's top surface, providing the most direct perspective for corner identification and pose calculation. "Relative sides" can refer to the front and rear sides, or the left and right sides, along the vehicle's direction of travel (usually the container's length). This symmetrical or staggered arrangement aims to expand the scanning coverage and reduce blind spots for individual sensors.

[0090] For example, Figure 5 This is a schematic diagram of the deployment structure of the sensor device provided in this application, such as... Figure 5 As shown, two sensor devices, A1 and A2, are installed below the trolley platform of the crane. The two sensor devices are located at the upper left corner (corresponding to position A1) and the lower right corner (corresponding to position A2) under the trolley, with their mirror surfaces facing downwards perpendicular to the ground. These sensor devices contain multiple lidar units. This positioning allows for long-range identification and moves with the trolley.

[0091] As another example, the number of sensors can be increased or the installation position and angle can be changed when deploying sensor devices. For example, installing one sensor device at each of the four corners of the vehicle platform will result in more uniform and comprehensive coverage, but this will increase costs, power consumption, and data fusion complexity. Another example is installing sensor devices on the side of the vehicle at an angle downwards, which can simultaneously show the side and top of the container, but this results in greater point cloud distortion, more complex coordinate system transformation, and greater susceptibility to interference from other moving objects on site. Therefore, this embodiment preferably deploys two sensor devices, located at the upper left and lower right corners below the vehicle, with their mirrors facing downwards perpendicular to the ground.

[0092] Since the original point cloud data contains a lot of noise and irrelevant information, and the data volume is large, direct use will affect the accuracy and processing efficiency of subsequent dual-box operation judgment. In this embodiment, by reasonably deploying sensor devices under the trolley platform in the yard bridge to collect initial point cloud data and performing preprocessing, more accurate and concise dual-box point cloud data of the area to be operated can be obtained, providing a reliable data foundation for subsequent accurate judgment of the feasibility of dual-box operation.

[0093] Based on the above embodiments, the method for preprocessing the initial point cloud data to obtain the dual-box point cloud data of the area to be operated may include: performing coordinate correction on the initial point cloud data to obtain corrected point cloud data; and dynamically cropping the dual-box point cloud data of the area to be operated from the corrected point cloud data according to the preset dual-box operation task type.

[0094] In this process, due to potential errors during sensor installation or the inherent characteristics of the sensors themselves, the coordinates of points in the initial point cloud data may deviate from their actual positions. Coordinate correction, through specific algorithms and reference information, corrects these deviations, transforming the initial point cloud data from different sensors and located in their respective independent measurement coordinate systems into a standardized and consistent operational reference coordinate system that facilitates subsequent processing.

[0095] Corrected point cloud data refers to point cloud data after coordinate system one is completed. At this point, the three-dimensional coordinates (X, Y, Z) of each point in the point cloud have physical meaning in the same global coordinate system. For example, the origin of the coordinate system is the projection of the geometric center point (or drive center point) of the trolley platform onto the horizontal plane; the X-axis is parallel to the running direction of the gantry crane (i.e., the entire gantry) and can represent the length direction of the container; the Y-axis is parallel to the running direction of the trolley and its platform and is horizontally orthogonal to the X-axis, which can represent the width direction of the container, and is also the spacing and lateral alignment direction of two side-by-side containers; the Z-axis is perpendicular to the horizontal plane, vertically upward or downward, representing the height, used to determine the height of the container top surface, identify corner points (such as top corner points), and ensure the clearance for the safe descent of the spreader.

[0096] In one example, after the sensors are installed, a high-precision calibration process can be used to accurately measure the installation position and orientation (e.g., rotation matrix R and translation vector T) of each lidar relative to a preset operational reference coordinate system (e.g., the center coordinate system of the vehicle platform). During preprocessing, the initial point cloud data of each lidar is transformed to a common coordinate system using its corresponding extrinsic parameter matrix. The extrinsic parameter matrix transformation can satisfy the following:

[0097] ;

[0098] After performing this operation on all initial point cloud data, you will obtain the corrected point cloud data in the job reference coordinate system.

[0099] In this embodiment, the dual-box operation task type can refer to predefined different dual-box operation scenario classifications. Each type is associated with a set of key geometric and layout parameters. Dynamic clipping refers to flexibly determining a three-dimensional Region of Interest (ROI) based on the specific circumstances of the current task (i.e., task type) and extracting point cloud data falling within this region from the corrected point cloud data. The position and size of the ROI will change according to the task to ensure that the acquired point cloud data best meets the current operation requirements, thereby improving the efficiency of subsequent processing.

[0100] In some examples, the system database pre-stores templates for various double-container operation task types, such as: 20-foot double containers packed side-by-side, 40-foot double containers spaced 1 meter apart, and mixed loading of 20-foot and 40-foot containers. Each template defines the expected length, width, spacing, and relative positional relationship of the double containers. When the system receives a specific double-container operation task (such as unloading two 20-foot containers), it calls the corresponding template parameters. Based on the real-time positioning of the spreader or trolley, it calculates the expected spatial range (one or two 3D bounding boxes) of the two containers, which is then used as the ROI for cropping.

[0101] It is evident that the initial point cloud data may contain coordinate deviations, affecting data accuracy. Furthermore, different dual-container operation tasks have varying requirements for point cloud data, making it difficult for fixed processing methods to meet diverse operational needs. Therefore, this embodiment improves the accuracy of point cloud data through coordinate correction, while dynamic cropping flexibly acquires relevant point cloud data according to different dual-container operation task types, making data processing more targeted and efficient. This reduces the risk of subsequent misidentification caused by other objects in the scene (such as corner points of adjacent containers) and improves the accuracy of corner point extraction and pose calculation.

[0102] Based on the above embodiments, the method for determining pose parameters based on the first corner point, the second corner point, the third corner point, and the fourth corner point described in S103 may include: determining a first slope of the first container based on the first corner point and the second corner point; determining a second slope of the second container based on the third corner point and the fourth corner point; using the difference between the first slope and the second slope as the yaw angle difference; and determining the trolley direction deviation based on the second corner point and the third corner point; wherein the pose parameters include the yaw angle difference and the trolley direction deviation.

[0103] In this embodiment, the yaw angle can refer to the orientation angle of the container in the horizontal plane (i.e., the XY plane, usually with the ground as a reference), such as the angle α between the sides containing the two corner points of the container and the positive direction of the X-axis. Figure 2 As shown, assuming the coordinates of the first corner point a1 are (a1.x, a1.y), the second corner point a2 are (a2.x, a2.y), the third corner point a3 are (a3.x, a3.y), and the fourth corner point a4 are (a4.x, a4.y), then the first slope k1 can be (a2.y – a1.y) / (a2.x – a1.x). Similarly, the second slope k2 can be (a4.y – a3.y) / (a4.x – a3.x), where k1 = tan(α1) and k2 = tan(α2). The yaw angle difference can be θ = (a2.y – a1.y) / (a2.x – a1.x) – (a4.y – a3.y) / (a4.x – a3.x).

[0104] The yaw angle difference can be a positive or negative value, indicating whether the second container is yawed clockwise or counterclockwise relative to the first, providing directional information for possible subsequent fine-tuning operations. Alternatively, this value can be the absolute difference between the first and second slopes, facilitating comparison with a preset threshold to determine if the conditions for dual-container operation are met.

[0105] The trolley directional deviation refers to the lateral alignment error of two containers perpendicular to their long sides and parallel to the trolley's direction of movement. It determines whether the two locking heads of the double-container spreader can simultaneously fall into the corner holes of the two containers in the lateral (Y-direction). Optionally, the second and third corner points are the closest corner points of the two containers in the horizontal direction. For example, such as... Figure 2 As shown, the second corner point a2 and the third corner point a3 are relatively close in the horizontal direction. The coordinates of the second corner point a2 are (a2.x, a2.y), and the coordinates of the third corner point a3 are (a3.x, a3.y). The directional deviation of the trolley is d = (a3.y - a2.y). For example, if the y-coordinate of a2 is 1000mm and the y-coordinate of a3 is 1005mm, then the directional deviation of the trolley is d = 5mm.

[0106] By clearly defining the pose parameters, including the yaw angle difference and the trolley direction deviation, the relative pose relationship of the containers can be accurately characterized, namely the parallelism and lateral alignment of the two containers. These two parameters are the minimum necessary and sufficient set of indicators for judging the feasibility of two-container operations. This avoids interference caused by the use of redundant or irrelevant parameters, improves the judgment efficiency, and thus reduces the overall time of two-container operations and improves the operational efficiency.

[0107] Based on the above embodiments, the method described in S104 for determining whether the pose relationship between the first container and the second container meets the dual-container operation conditions based on the pose parameters, and generating a determination result, may include: comparing the yaw angle difference with a preset first threshold, and comparing the trolley direction deviation with a preset second threshold; if the yaw angle difference is less than the first threshold and the trolley direction deviation is less than the second threshold, then a determination result that meets the dual-container operation conditions is generated; if the yaw angle difference is greater than or equal to the first threshold, and / or the trolley direction deviation is greater than or equal to the second threshold, then a determination result that does not meet the dual-container operation conditions is generated.

[0108] In this embodiment, the first threshold can be a pre-set critical value used to measure whether the yaw angle difference is within an acceptable range. When the yaw angle difference is less than the threshold, it is considered that the difference in the rotation angle of the two containers in the horizontal plane is small, and the impact on the two-container operation is within the allowable range; otherwise, it is considered that the difference is large and may affect the normal operation.

[0109] The second threshold can be a critical value used to determine whether the trolley's directional deviation is reasonable. If the trolley's directional deviation is less than this threshold, it indicates that the lateral alignment direction of the two boxes is close to the trolley's driving direction, which can meet the requirements of the trolley's driving direction for the two-box operation; if it is greater than or equal to this threshold, it indicates that the lateral alignment deviation of the two boxes is too large, which may affect the accuracy of the two-box operation.

[0110] The judgment result is based on the comparison of the yaw angle difference and the trolley direction deviation with the corresponding threshold, and the conclusion is drawn on whether the positional relationship between the first container and the second container meets the conditions for dual-container operation. It can be divided into two cases: meeting and not meeting.

[0111] Furthermore, the calculated yaw angle difference is compared with a preset first threshold, and the trolley directional deviation is compared with a preset second threshold in the same way. If the yaw angle difference is less than the first threshold and the trolley directional deviation is less than the second threshold, a judgment result indicating that the dual-box operation conditions are met is generated. If the yaw angle difference is greater than or equal to the first threshold, or the trolley directional deviation is greater than or equal to the second threshold, or both are met, a judgment result indicating that the dual-box operation conditions are not met is generated. In addition, when generating a judgment result indicating that the dual-box operation conditions are not met, indication information indicating splitting into single-box operation tasks or abnormal warnings can also be generated simultaneously.

[0112] In some examples, thresholds can be set through experiments and practical operational experience. For instance, multiple double-container operation tests can be conducted under different operating environments and conditions, recording the yaw angle difference and trolley directional deviation range that ensure smooth operation. Then, based on this data, a first threshold and a second threshold can be determined. For example, based on historical safe operation experience and the relationship between container inclination and yaw angle, such as k1=tan(α1) and k2=tan(α2), the first threshold can be set to 0.03, and the second threshold to 50mm.

[0113] In other examples, multiple thresholds can be preset based on different types of dual-container operations (such as 20-foot containers, 40-foot containers, and mixed containers mentioned above). Different types of containers may have different corner hole sizes and spreader lock spacing, thus resulting in different safety tolerances. Furthermore, the thresholds can be fine-tuned based on the operational stage and environmental conditions.

[0114] By setting reasonable thresholds and comparing the pose parameters with the thresholds to determine whether the conditions for dual-box operation are met, a clear and operable basis is provided for the decision-making of dual-box operation, which improves the reliability and accuracy of dual-box operation, avoids unnecessary movement and waiting time, and thus improves the efficiency of dual-box operation of yard cranes.

[0115] Based on the above embodiments, the first threshold and / or the second threshold are adaptive thresholds that are dynamically adjusted by a machine learning model; the machine learning model is trained based on historical double-container operation data, which includes pose parameters and container information corresponding to successful and / or failed operations.

[0116] In this embodiment, the machine learning model learns from historical double-container operation data to understand the relationship between different pose parameters and the success or failure of double-container operations, thereby gaining the ability to predict and adjust thresholds. Historical double-container operation data refers to records of relevant information from past double-container operations. Pose parameters may include yaw angle differences, trolley direction deviations, etc., while container information may include other prior or environmental information related to the container itself or the operational scenario. This information is crucial for determining the difficulty and safety range of double-container operations, as the permissible pose deviations may vary significantly depending on the type of container. This data forms the basis for training the machine learning model, reflecting the actual situation under different operational scenarios.

[0117] An adaptive threshold is a threshold that can flexibly change based on the analysis results of a machine learning model to better adapt to different operating environments and conditions. This means that in each judgment or each operating cycle, the first threshold or the second threshold, or both the first and second thresholds, can be dynamically adjusted according to different operating scenarios and container conditions based on the machine learning model's learning and analysis of historical dual-container operation data.

[0118] Furthermore, a large amount of historical dual-container operation data can be collected in advance, recording in detail the pose parameters (such as yaw angle difference, trolley direction deviation, etc.), container information, and the final result (success or failure) of each operation. This data is then categorized and labeled according to factors such as container type and operation scenario for subsequent model training. The machine learning model is then pre-trained using this organized historical data. The model learns the intrinsic relationships between various pose parameters and operation success or failure under different container information, establishing a corresponding relational model. During this process, the model summarizes appropriate threshold ranges for different types of scenarios based on a large number of historical cases. In actual dual-container operation scenarios, relevant information about the containers involved in the current operation is input into the pre-trained machine learning model. Based on the previously learned patterns, the model directly outputs adjustment suggestions for the first and / or second thresholds for the current container type and operation scenario. The system dynamically adjusts the thresholds according to these suggestions.

[0119] In some examples, suitable machine learning algorithms, such as decision trees, support vector machines, or neural networks, can be used to build the model. The compiled historical data is divided into training and validation sets according to a certain ratio. The model is trained using the training set, and by continuously adjusting the model's parameters, it can accurately learn the relationship between pose parameters, container information, and operational results. Then, the trained model is evaluated using the validation set, and its performance is further optimized based on the evaluation results.

[0120] Traditional fixed thresholds cannot fully account for the diversity of containers and the differences in various operating scenarios, resulting in inaccurate threshold settings in dual-container operations, which may affect the safety and efficiency of the operation. This embodiment dynamically adjusts the first and / or second thresholds by utilizing a machine learning model pre-trained based on historical dual-container operation data. This allows the thresholds to better match the actual needs of different container types and operating scenarios, improving the accuracy and adaptability of dual-container operation judgments, and enhancing the safety and efficiency of the operation.

[0121] Furthermore, in complex real-world scenarios, when two containers are close together, have similar surface reflectivity, or have snow / water accumulation, the lidar point cloud may not form a clear physical separation, leading to the point clouds of the two containers being mistakenly identified as a single entity, or the inability to correctly assign corner points. Alternatively, when there is occlusion, sparse point clouds, or the presence of structurally similar background objects, relying solely on the point cloud data of both containers for corner point identification may result in low accuracy and efficiency. Therefore, based on the above embodiments, the method may further include: acquiring image data of the area to be operated on; determining the first visual features of the first container and the second visual features of the second container based on the image data; correspondingly, processing the point cloud data of both containers to identify the first and second corner points of the first container, and the third and fourth corner points of the second container, including: performing planar fitting and segmentation on the point cloud data of both containers based on the first and second visual features to obtain the first point cloud cluster of the first container and the second point cloud cluster of the second container; and performing corner detection on the first and second point cloud clusters based on a preset corner detection algorithm to obtain the first and second corner points of the first container, and the third and fourth corner points of the second container.

[0122] In this embodiment, the image data can be visual information about the area to be operated on by the double container, which is obtained by image acquisition devices such as cameras. This information is presented in the form of images, reflecting the environment of the work site, the appearance of the containers, and other conditions.

[0123] Furthermore, the two sensor devices deployed beneath the trolley platform of the yard crane can also include high-resolution cameras for synchronously acquiring image data of the area to be worked on. Simultaneously, the preprocessing of the initial point cloud data can also include spatiotemporal registration of the image data and the corrected point cloud data for data fusion. Alternatively, multiple high-definition cameras can be installed at appropriate locations on the yard crane to capture images of the area to be worked on from different angles.

[0124] Visual features refer to the characteristic information extracted from image data that can identify, distinguish, or locate containers. For example, the color, shape, and serial number of a container can all serve as visual features. For instance, OCR (Optical Character Recognition) technology can be used to identify and read the serial number on the surface of a container.

[0125] In point cloud data processing, planar fitting refers to the process of identifying and extracting (usually planar) point clouds of the top surface of containers from the entire scene's point cloud; planar segmentation, on the other hand, uses visual features to separate point clouds belonging to different containers. A point cloud cluster refers to a spatially aggregated set of point clouds belonging to the same physical container, obtained after segmentation. In some examples, for feature points extracted from an image, based on their affiliation in the image's feature space (whether they belong to the first or second container), these known-affiliation 3D points are used as target points. Then, region growing or graph-based clustering algorithms are used to group points in the entire point cloud that are spatially and normally similar to these target points into the same cluster.

[0126] Furthermore, a preset corner detection algorithm (such as the Harris corner detection algorithm, the Shi-Tomasi corner detection algorithm, etc.) can be used to detect corners in the first point cloud cluster and the second point cloud cluster respectively, thereby obtaining the first and second corners of the first container, as well as the third and fourth corners of the second container. For example, for the first point cloud cluster, the Harris corner detection algorithm is used to calculate the corner response function value of each point. When the value is greater than a preset threshold, the point is considered a corner. In this way, the first and second corners of the first container are detected.

[0127] By acquiring image data of the area to be worked on and extracting visual features, the visual features are used to perform planar fitting and segmentation on the point cloud data of the two containers. The complex point cloud data of the two containers is separated into point cloud clusters of individual containers, and then corner detection is performed separately. This improves the accuracy and efficiency of corner detection and provides a more reliable data foundation for accurately determining the pose parameters of the containers in the future.

[0128] Based on the above embodiments, the method may further include: when the judgment result indicates that the positional relationship between the first container and the second container meets the conditions for dual-container operation, performing identity matching verification on the first container and the second container based on the first visual feature and the second visual feature to obtain the verification result; if the verification result indicates that the identity matching verification fails, generating a prompt message indicating that the judgment result is abnormal.

[0129] In this embodiment, identity matching verification can refer to comparing the actual visual feature information of the first container and the second container with the pre-stored container identity information (which may include specific visual feature identifiers, etc.) to verify whether the two containers are the correct container combination for the planned double container operation.

[0130] For example, a visual feature database for shipping containers can be established, storing the unique visual features of each container (such as specific color distributions, special marking patterns, etc.) and associating them with corresponding container numbers and other information. During identity matching and verification, the first and second visual features of the first and second containers are compared with the information in the database.

[0131] The verification result can be divided into two cases: pass and fail. A pass indicates that the first and second containers are the correct combination for the operation; a fail means that the two are not the planned combination, and there may be a problem with the operation objects.

[0132] The notification message is generated to alert operators when the identity verification fails, indicating an anomaly in the current judgment result. This message can be presented through sound, pop-up windows, indicator lights, etc. For example, the control system of the yard crane can be connected to the display and sound equipment in the control room. When the verification result indicates that the identity verification has failed, the control system sends a signal to the display device, displaying a notification window saying "Judgment result abnormal, container identity does not match"; simultaneously, it sends a signal to the sound equipment, emitting an alarm sound.

[0133] By adding an identity matching verification step and generating a prompt message when the verification fails, it is possible to effectively avoid operating on the wrong container, improve the accuracy and safety of double container operations, and reduce losses caused by operational errors.

[0134] Based on the above embodiments, it should be noted that the application scenario of the dual-container operation judgment method provided in this application can be a dual-container operation scenario in automated ports or logistics centers. In an automated yard crane system, the trolley platform is responsible for laterally moving the spreader to the target container position, and the spreader completes the stacking or handling of containers through the gripping mechanism. Dual-container operation requires the simultaneous gripping of two adjacent containers, and its feasibility depends on the stacking state of the containers. In the prior art, the sensors at the end of the spreader only detect when approaching the container, resulting in delayed prediction and inability to cover long-distance scenarios. The method of this application, by deploying long-distance sensing sensor devices (such as LiDAR, cameras, etc.) under the trolley platform, combined with multi-sensor data fusion and adaptive threshold algorithms, can complete the prediction of dual-container operation during the spreader movement stage, adapting to complex scenarios such as high-density stacking and obstructed environments.

[0135] Furthermore, this method uses LiDAR data from the trolley platform to perform ROI region clipping. Simultaneously, it identifies two corner points of the two containers. Based on these corner points, it calculates the container pose information and uses this information to determine whether the double-container operation can be executed normally, either when the spreader is at a high position or before the trolley reaches its position. If it cannot be executed normally, it returns information and splits the operation into a single-container task.

[0136] In some embodiments, the method may include: (1) after acquiring the lidar data under the vehicle platform, first perform coordinate correction on the lidar data. (2) extract the point cloud of the ROI region by task type, and identify the corner points of the point cloud in the region. (3) calculate the pose of container 1 and container 2: that is, the yaw angle difference θ = (a2.y – a1.y) / (a2.x – a1.x) - (a4.y – a3.y) / (a4.x – a3.x). (4) calculate the vehicle direction deviation d = (a3.y – a2.y) of container 1 and container 2. (5) determine whether the yaw angle difference between the two containers and the vehicle direction difference are less than a certain threshold. If so, the container can be successfully grabbed. Otherwise, it is predicted that the two containers cannot be operated.

[0137] The dual-box operation judgment method provided in this application uses a laser radar installed on the trolley for sensing, which solves the problem of predicting in advance whether the dual-box operation task can be successfully executed, and improves the intelligence and efficiency of automated yard cranes.

[0138] This application also provides a dual-box operation judgment system, including: at least two sensor devices deployed on opposite sides below the trolley platform of the yard bridge, for collecting point cloud data and image data of the area to be operated; and a data processing module, which is communicatively connected to the sensor devices and is used to execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0139] This application also provides a yard crane, including the dual-container operation judgment system described in the above embodiments. A yard crane can refer to an important piece of equipment used for loading and unloading containers in ports, docks, and other similar locations. The yard crane in this application embodiment is equipped with a dual-container operation judgment system, which, compared to existing technologies, possesses a more intelligent and accurate dual-container operation judgment capability.

[0140] For example, the dual-box operation judgment system can be integrated into the electrical control system of the yard crane. Yard crane operators can view the status and results of the dual-box operation judgment system through a human-machine interface. When the dual-box operation judgment system determines that the operation conditions are met, it automatically or manually sends a signal to the yard crane's control system allowing operation, and the yard crane performs the dual-box operation according to the normal procedure; when the conditions are not met, the system issues an alarm and stops the operation or prompts the operator to make adjustments.

[0141] Figure 6 This is a schematic diagram of the dual-box operation judgment device provided in this application, as shown below. Figure 6 As shown, the dual-box operation judgment device 60 provided in this embodiment includes:

[0142] Module 601 is used to acquire dual-box point cloud data of the area to be worked on;

[0143] The identification module 602 is used to process the point cloud data of the two containers and identify the first corner point and the second corner point of the first container, as well as the third corner point and the fourth corner point of the second container, wherein the first corner point, the second corner point, the third corner point and the fourth corner point are located on the same side of the two containers.

[0144] The determination module 603 is used to determine the pose parameters based on the first corner point, the second corner point, the third corner point, and the fourth corner point. The pose parameters are used to indicate the relative pose relationship between the first container and the second container.

[0145] The judgment module 604 is used to determine whether the positional relationship between the first container and the second container meets the conditions for dual-container operation based on the positional parameters, and to generate a judgment result.

[0146] In one possible implementation, the acquisition module 601 can also be used to: collect initial point cloud data by means of at least two sensor devices deployed on opposite sides below the trolley platform of the yard bridge, wherein each sensor device includes multiple lidars; and preprocess the initial point cloud data to obtain dual-box point cloud data of the area to be operated.

[0147] In one possible implementation, the acquisition module 601 can also be used to: perform coordinate correction on the initial point cloud data to obtain corrected point cloud data; and dynamically cut out the double-box point cloud data of the area to be operated from the corrected point cloud data according to the preset double-box operation task type.

[0148] In one possible implementation, the determining module 603 can also be used to: determine the first slope of the first container based on the first corner point and the second corner point; determine the second slope of the second container based on the third corner point and the fourth corner point; use the difference between the first slope and the second slope as the yaw angle difference; and determine the trolley direction deviation based on the second corner point and the third corner point; wherein the pose parameters include the yaw angle difference and the trolley direction deviation.

[0149] In one possible implementation, the judgment module 604 can also be used to: compare the yaw angle difference with a preset first threshold and compare the trolley direction deviation with a preset second threshold; if the yaw angle difference is less than the first threshold and the trolley direction deviation is less than the second threshold, then a judgment result that meets the dual-box operation conditions is generated; if the yaw angle difference is greater than or equal to the first threshold and / or the trolley direction deviation is greater than or equal to the second threshold, then a judgment result that does not meet the dual-box operation conditions is generated.

[0150] In one possible implementation, the acquisition module 601 can also be used to: acquire image data of the area to be operated; and based on the image data, determine the first visual features of the first container and the second visual features of the second container.

[0151] In one possible implementation, the recognition module 602 can also be used to: perform planar fitting and segmentation on the point cloud data of the two containers based on the first visual feature and the second visual feature to obtain the first point cloud cluster of the first container and the second point cloud cluster of the second container; and perform corner detection on the first point cloud cluster and the second point cloud cluster based on the preset corner detection algorithm to obtain the first corner point and the second corner point of the first container, as well as the third and fourth corner points of the second container.

[0152] In one possible implementation, the judgment module 604 can also be used to: when the judgment result indicates that the positional relationship between the first container and the second container meets the conditions for dual-container operation, perform identity matching verification on the first container and the second container based on the first visual feature and the second visual feature, and obtain the verification result; if the verification result indicates that the identity matching verification fails, generate a prompt message indicating that the judgment result is abnormal.

[0153] The dual-box operation judgment device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0154] Figure 7 A schematic diagram of the structure of the electronic device provided in this application. Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0155] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0156] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0157] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0158] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0159] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0160] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0161] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0162] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0163] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0164] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0167] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0169] It should be understood that the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover but not exclude inclusion. For example, a product or device that includes a series of components is not necessarily limited to those components that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0170] As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0171] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A dual tank operation determination method characterized by comprising: include: Acquire dual-box point cloud data of the area to be worked on; The point cloud data of the two containers is processed to identify the first and second corner points of the first container, as well as the third and fourth corner points of the second container, wherein the first, second, third, and fourth corner points are located on the same side of the two containers. Based on the first corner point, the second corner point, the third corner point, and the fourth corner point, pose parameters are determined, and the pose parameters are used to indicate the relative pose relationship between the first container and the second container. Based on the pose parameters, determine whether the pose relationship between the first container and the second container satisfies the conditions for dual-container operation, and generate a determination result.

2. The method of claim 1, wherein, The acquisition of dual-box point cloud data of the area to be worked on includes: Initial point cloud data is collected by at least two sensor devices deployed on opposite sides below the vehicle platform in the field bridge, wherein each sensor device includes multiple lidars; The initial point cloud data is preprocessed to obtain the dual-box point cloud data of the area to be worked on.

3. The method of claim 2, wherein, The preprocessing of the initial point cloud data to obtain the dual-box point cloud data of the area to be processed includes: The initial point cloud data is subjected to coordinate correction to obtain corrected point cloud data; Based on the preset dual-box operation task type, the dual-box point cloud data of the area to be operated is dynamically cropped from the corrected point cloud data.

4. The method of claim 1, wherein, The step of determining pose parameters based on the first corner point, the second corner point, the third corner point, and the fourth corner point includes: Based on the first corner point and the second corner point, determine the first slope of the first container; The second slope of the second container is determined based on the third and fourth corner points; The difference between the first slope and the second slope is taken as the yaw angle difference; The directional deviation of the vehicle is determined based on the second corner point and the third corner point; The pose parameters include the yaw angle difference and the vehicle direction deviation.

5. The method of claim 4, wherein, The step of determining whether the pose relationship between the first container and the second container satisfies the dual-container operation conditions based on the pose parameters, and generating a determination result, includes: The yaw angle difference is compared with a preset first threshold, and the trolley direction deviation is compared with a preset second threshold. If the yaw angle difference is less than the first threshold and the trolley direction deviation is less than the second threshold, then a judgment result that satisfies the dual-box operation conditions is generated. If the yaw angle difference is greater than or equal to the first threshold, and / or the trolley direction deviation is greater than or equal to the second threshold, then a judgment result indicating that the dual-box operation conditions are not met is generated.

6. The method of claim 5, wherein, The first threshold and / or the second threshold are adaptive thresholds that are dynamically adjusted by a machine learning model; the machine learning model is trained based on historical double-container operation data, which includes pose parameters and container information corresponding to successful and / or failed operations.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Acquire image data of the area to be worked on; Based on the image data, a first visual feature of the first container and a second visual feature of the second container are determined. Accordingly, the processing of the dual-container point cloud data to identify the first and second corner points of the first container, and the third and fourth corner points of the second container includes: Based on the first visual feature and the second visual feature, the dual-container point cloud data is subjected to planar fitting and segmentation to obtain the first point cloud cluster of the first container and the second point cloud cluster of the second container. Based on a preset corner detection algorithm, corner detection is performed on the first point cloud cluster and the second point cloud cluster respectively to obtain the first corner and the second corner of the first container, as well as the third and fourth corners of the second container.

8. The method according to claim 7, characterized in that, The method further includes: When the judgment result indicates that the positional relationship between the first container and the second container meets the conditions for dual-container operation, the identity matching verification of the first container and the second container is performed based on the first visual feature and the second visual feature to obtain the verification result; If the verification result indicates that the identity matching verification failed, a prompt message indicating that the judgment result is abnormal will be generated.

9. A dual-box operation judgment system, characterized in that, include: At least two sensor devices are deployed on opposite sides below the platform of the trolley in the yard bridge to collect point cloud data and image data of the area to be worked on; A data processing module, communicatively connected to the sensor device, is used to execute the method as described in any one of claims 1-8.

10. A field bridge, characterized in that, Includes the dual-box operation judgment system as described in claim 9.

Images