A method and device for constructing a map of a container yard

Abstract

The application provides a container yard map construction method and device, and relates to the technical field of map construction. The method comprises the following steps: constructing an air-ground collaborative long-term SLAM system comprising a container truck and a UAV; collecting point cloud data, sensing data and image data of the container yard by using the air-ground collaborative long-term SLAM system, and constructing an initial map of the container yard based on the point cloud data and the image data; determining the planar features of the container yard according to the size information of the containers in the container yard and the point cloud data; determining the target relative pose between the current frame and the nearest adjacent plane in the initial map according to the sensing data and the planar features of the container yard, and constructing a target map of the container yard according to the target relative pose. In this way, the map of the container yard can be accurately constructed, and the work efficiency of the container yard management can be improved.

Description

Technical Field

[0001] This invention relates to the field of map building technology, and more specifically, to a method and apparatus for map building of container yards. Background Technology

[0002] With the rapid development of robotics, the application of intelligent unmanned systems, represented by unmanned vehicles and drones, in port container yards has gradually become a research hotspot. However, the unique characteristics of the port environment bring many challenges to the application of unmanned systems. Due to the special nature of ports, the requirements for safety are extremely high. Although current port containers are managed intelligently, and containers entering the port are numbered by the upper-level management system (upper-level control system) and uniformly coordinated and allocated to workers or unmanned loading trucks in various zones for placement, the system cannot detect the actual placement of containers. For example, when the system assigns workers to operate on a certain container, the worker's operation on other unrelated containers during the operation cannot be detected by the management system, which can easily cause safety hazards in the port container yard.

[0003] Currently, container yards mostly rely on inspection personnel, surveillance cameras, and single inspection robots to monitor the placement of containers and identify potential hazards. However, inspection personnel not only require more workers for management but also suffer from low efficiency, inability to guarantee high response speeds, and incompatibility with the requirements of modern smart ports. While using single inspection robots to scan the yard is simple, the complex port environment makes it difficult to effectively map and quickly inspect container yards in obstructed or large areas. Furthermore, introducing additional inspection vehicles interferes with the existing system's operation.

[0004] Air-to-ground collaborative mapping has always been a key issue in the field of intelligent robotics. A Chinese invention patent (CN117191005A) proposes a method, apparatus, device, and storage medium for heterogeneous air-to-ground collaborative mapping. This method extracts keyframes by setting a threshold and uses GNSS coordinates to locate the keyframe positions of air and ground platforms, applying closed-loop constraints to the heterogeneous air-to-ground data, which can reduce computation time and improve matching efficiency. However, this method does not effectively utilize environmental features and cannot achieve good results in the special environment of port yards. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a method and apparatus for constructing a map of a container yard, so as to accurately construct a map of the container yard and thereby improve the efficiency of container yard management.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, embodiments of this application provide a method for constructing a map of a container yard, comprising: constructing an air-ground cooperative long-term SLAM system including a container truck and a drone; the drone is equipped with a first radar, an image acquisition device, and a first inertial sensor on its bottom, and the container truck is equipped with a second radar and a second inertial sensor on its top; using the air-ground cooperative long-term SLAM system to collect point cloud data, sensor data, and image data of the container yard, and constructing an initial map of the container yard based on the point cloud data and the image data; determining the planar features of the container yard according to the size information of the containers in the container yard and the point cloud data; determining the target relative pose between the current frame and the nearest adjacent plane in the initial map according to the sensor data and the planar features of the container yard, and constructing a target map of the container yard according to the target relative pose.

[0008] In one embodiment, the use of the air-ground collaborative long-term SLAM system to collect point cloud data, sensor data, and image data of the container yard includes: the UAV in the air-ground collaborative long-term SLAM system collecting point cloud data above the container yard via the first radar and collecting image data of the container yard via the image acquisition device; the truck in the air-ground collaborative long-term SLAM system collecting point cloud data of the vertical plane of the containers in the container yard via the second radar; the point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the containers; the first inertial sensor and the second inertial sensor in the air-ground collaborative long-term SLAM system collecting sensor data; the sensor data includes angular rate and acceleration.

[0009] In one embodiment, determining the planar features of the container yard based on the size information of the containers in the container yard and the point cloud data includes: deleting ground point clouds from the point cloud data; using the size information of different types of containers and the direction of the plane normal vector as constraints, fitting the intensity information of the point clouds in the point cloud data with a preset algorithm to obtain fitted point cloud data; segmenting the point cloud plane of the highest layer container and the vertical container point cloud plane of each container position from the fitted point cloud data to obtain a target point cloud plane; and determining the corner coordinates of each container position in the container yard based on the target point cloud plane to obtain the planar features of the container yard.

[0010] In one implementation, determining the target relative pose between the current frame and the nearest neighboring plane in the initial map based on the sensing data and the planar features of the container yard includes: determining the nearest neighboring point cloud plane to the point cloud plane in the current frame from the initial map; determining the initial relative pose between the point cloud plane in the current frame and the neighboring point cloud plane in the initial map based on the sensing data; and matching the point cloud plane in the current frame with the neighboring point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between them.

[0011] In one implementation, matching the point cloud plane in the current frame with adjacent point cloud planes in the initial map to optimize the initial relative pose and obtain the target relative pose between them includes: determining the corner coordinates and normal vector direction of the point cloud plane in the initial map corresponding to the point cloud plane in the current frame; determining a translation amount based on the corner coordinates, a preset number of point cloud planes from the perspective of UAVs, and a preset number of point cloud planes from the perspective of trucks; determining a rotation amount based on the normal vector direction, the normal vector direction of the preset number of point cloud planes from the perspective of UAVs, and the normal vector direction of the preset number of point cloud planes from the perspective of trucks; and determining the target relative pose based on the translation amount and the rotation amount.

[0012] In one implementation, after obtaining the target relative pose between the two, the method further includes: determining edge point clouds from the point cloud plane, performing point cloud registration between the edge point clouds in the current frame and the edge point clouds in the initial map, and obtaining the optimized target relative pose.

[0013] In one embodiment, after constructing the target map of the container yard based on the target relative pose, the method further includes: generating a depth image based on the point cloud data of consecutive frames from the same viewpoint; comparing the depth image with the pixel depth at the same location in the initial map, determining and deleting dynamic point clouds through point cloud occlusion to obtain a target point cloud; and updating the map of the container yard based on the target point cloud and the historical map.

[0014] In one implementation, updating the map of the container yard based on the target point cloud and the historical map includes: determining the current map based on the target point cloud; comparing the pixel depth of the same location in the current map and the historical map, retaining the dynamic point cloud of the current map relative to the historical map and the static point cloud of the two, to obtain the updated map of the container yard.

[0015] In one embodiment, the method further includes: using the air-ground collaborative long-term SLAM system to collect location descriptors of the container yard, and constructing the current topology of the container yard through a chessboard structure; the location descriptors include container feature data and location information; determining a loop closure loss value based on the difference between the current topology and historical topology of the container yard and a preset loop closure loss function; using the UAV to detect the position of the container trucks, and determining the position of the container trucks as the initial open space positions at both ends of the map of the container yard for stitching together; performing loop closure detection on the container yard based on the loop closure loss value, and optimizing the map of the container yard based on the loop closure loss function, the initial open space positions, the preset chessboard topology, and the preset outer ring distribution topology.

[0016] Secondly, embodiments of this application also provide a map building device for a container yard, comprising: a hardware system building module configured to build an air-ground cooperative long-term SLAM system including a container truck and an unmanned aerial vehicle (UAV); the UAV is equipped with a first radar, an image acquisition device, and a first inertial sensor on its bottom, and the container truck is equipped with a second radar and a second inertial sensor on its top; a data acquisition module configured to use the air-ground cooperative long-term SLAM system to acquire point cloud data, sensor data, and image data of the container yard, and to build an initial map of the container yard based on the point cloud data and the image data; a feature extraction module configured to determine the planar features of the container yard based on the size information of the containers in the container yard and the point cloud data; and a map building module configured to determine the target relative pose between the current frame and the nearest adjacent plane in the initial map based on the sensor data and the planar features of the container yard, and to build a target map of the container yard based on the target relative pose.

[0017] In one embodiment, the data acquisition module is configured as follows: the UAV in the air-ground cooperative long-term SLAM system acquires point cloud data above the container yard via the first radar and acquires image data of the container yard via the image acquisition device; the truck in the air-ground cooperative long-term SLAM system acquires point cloud data of the vertical plane of the containers in the container yard via the second radar; the point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the containers; the first inertial sensor and the second inertial sensor in the air-ground cooperative long-term SLAM system acquire sensing data; the sensing data includes angular rate and acceleration.

[0018] In one embodiment, the feature extraction module is configured to: delete ground point clouds from the point cloud data; use a preset algorithm to fit the intensity information of point clouds in the point cloud data, constrained by the size information of different types of containers and the direction of the plane normal vector, to obtain fitted point cloud data; segment the point cloud plane of the highest layer container and the vertical container point cloud plane of each container position from the fitted point cloud data to obtain a target point cloud plane; and determine the corner coordinates of each container position in the container yard based on the target point cloud plane to obtain the planar features of the container yard.

[0019] In one implementation, the map building module is configured to: determine the nearest neighboring point cloud plane to the point cloud plane in the current frame from the initial map; determine the initial relative pose between the point cloud plane in the current frame and the neighboring point cloud plane in the initial map based on the sensing data; and match the point cloud plane in the current frame with the neighboring point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between them.

[0020] In one implementation, the map building module is configured to: determine the corner coordinates and normal direction of the point cloud plane corresponding to the point cloud plane in the current frame in the initial map; determine a translation amount based on the corner coordinates, a preset number of point cloud planes from the perspective of UAVs, and a preset number of point cloud planes from the perspective of trucks; determine a rotation amount based on the normal direction, the preset number of point cloud planes from the perspective of UAVs, and the preset number of point cloud planes from the perspective of trucks; and determine the relative pose of the target based on the translation amount and the rotation amount.

[0021] In one embodiment, the map building device further includes a pose optimization module, which is configured to: determine edge point clouds from the point cloud plane, perform point cloud registration between the edge point clouds in the current frame and the edge point clouds in the initial map, and obtain the optimized target relative pose.

[0022] In one embodiment, the map building apparatus further includes a map updating module, which is configured to: generate a depth image based on the point cloud data of consecutive frames from the same viewpoint; compare the depth image with the pixel depth at the same location in the initial map, determine and delete dynamic point clouds through point cloud occlusion, and obtain a target point cloud; and update the map of the container yard based on the target point cloud and the historical map.

[0023] In one implementation, the map update module is configured to: determine the current map based on the target point cloud; compare the pixel depth of the same location in the current map with that in the historical map, retain the dynamic point cloud of the current map relative to the historical map and the static point cloud of the two, and obtain an updated map of the container yard.

[0024] In one embodiment, the map building device further includes a map optimization module, which is configured to: collect location descriptors of the container yard using the air-ground collaborative long-term SLAM system, and construct the current topology of the container yard through a chessboard structure; the location descriptors include container feature data and location information; determine a loop closure loss value based on the difference between the current topology and historical topology of the container yard and a preset loop closure loss function; use the UAV to detect the position of the container trucks, and determine the position of the container trucks as the initial open space positions at both ends of the map of the container yard for stitching together; perform loop closure detection on the container yard based on the loop closure loss value, and optimize the map of the container yard based on the loop closure loss function, the initial open space positions, the preset chessboard topology, and the preset outer ring distribution topology.

[0025] Thirdly, embodiments of this application also provide a visualization platform for a container yard, including: a processing module configured to match container numbers, container positions, and location information; and a visualization module configured to visualize the container yard based on the matching relationship between the container numbers, container positions, and actual location information using location descriptors.

[0026] Fourthly, embodiments of this application also provide a computer device, including: a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the computer device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of any of the above methods.

[0027] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of any of the above methods.

[0028] The beneficial effects of this application are as follows: by modifying the existing hardware system, an air-ground collaborative long-term SLAM system is obtained, which is constructed by UAVs and container trucks. Based on the point cloud data, sensor data and image data collected by the air-ground collaborative long-term SLAM system, an initial map of the container yard is constructed. Then, planar features are extracted, relative pose is determined based on the planar features, and an accurate map (target map) of the container yard is generated based on the relative pose. The constructed map of the container yard is more accurate, thereby improving the work efficiency of container yard management. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0031] Figure 2 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0032] Figure 3 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0033] Figure 4 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0034] Figure 5 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0035] Figure 6 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0036] Figure 7 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0037] Figure 8 This is a schematic diagram illustrating the pre-set chessboard-like topology in a map construction method for a container yard provided in an embodiment of this application.

[0038] Figure 9 This is a schematic diagram illustrating the preset outer ring-shaped topological relationship in a map construction method for a container yard provided in an embodiment of this application.

[0039] Figure 10 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0040] Figure 11 A flowchart illustrating a method for constructing a map of a container yard, provided in an embodiment of this application;

[0041] Figure 12 A schematic diagram of a map-building device for a container yard provided in an embodiment of this application;

[0042] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0044] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0045] In the description of this application, it should be noted that if the terms "upper", "lower", etc. appear to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in, it is only for the convenience of describing this application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0046] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0048] Figure 1 A flowchart illustrating a method for constructing a map of a container yard, as provided in this application embodiment; Figure 1 As shown, the method includes:

[0049] Step 110: Construct a long-term air-ground collaborative SLAM system that includes trucks and drones.

[0050] The drone is equipped with a first radar, image acquisition equipment, and a first inertial sensor on its bottom, while the container truck is equipped with a second radar and a second inertial sensor on its top.

[0051] In practice, there can be one or more drones and container trucks; no limit is set here.

[0052] The first radar carried by the drone can be a solid-state lidar; the second radar carried by the truck can be a mechanical lidar.

[0053] The image acquisition device can be any device capable of acquiring images, such as a visible light camera or a CCD camera; there are no restrictions here.

[0054] An inertial sensor, or IMU (Inertial Measurement Unit), is a device that measures an object's three-axis attitude angles (or angular rates) and acceleration. An IMU is typically a fully functional device comprised of two or more inertial measurement MEMS chips and an ASIC chip packaged together. Depending on the built-in sensors (three-axis magnetometer, three-axis accelerometer, and three-axis gyroscope), IMUs are classified as six-axis or nine-axis, capable of meeting the high-precision measurement needs of various application scenarios.

[0055] The air-ground collaborative long-term SLAM system in the container yard map construction method provided in this application embodiment is obtained by modifying an existing hardware system. Specifically, the following modifications were made:

[0056] (1) Configure UAVs (swarms) and trucks (swarms): Configure one or more UAVs in the container yard to patrol the container yard in different areas. The UAVs are equipped with a first radar, image acquisition equipment and a first IMU on their bottoms; the trucks are equipped with a second radar and a second IMU on their tops; the UAVs (swarms) and trucks (swarms) together form the above-mentioned air-ground collaborative long-term SLAM system.

[0057] (2) Unified Multi-Sensor Coordinate System: Point cloud data and image data include six coordinate systems: the world coordinate system, the truck body coordinate system, the UAV fuselage coordinate system, the solid-state lidar coordinate system, the mechanical lidar coordinate system, the radar coordinate system, and the image acquisition device coordinate system. In the container yard map construction method provided in this application embodiment, the world coordinate system is used as the global coordinate system. Specifically, the reference coordinate system of the UAV departure area can be used as the world coordinate system, and the origin can be the center point of the UAV departure area.

[0058] (3) Collaborative Air-Ground Mapping Using UAVs and Trucks: UAVs have a wide field of view, enabling them to map large areas of the yard and collect features from the top of containers. However, while the UAV's view encompasses most scene features, higher-layered containers may obstruct views of containers near lane lines, and ground vehicles may encounter containers being lifted into the air. Therefore, to improve the quality of scanned data, a ground scanning system (i.e., trucks (groups)) is introduced. Simultaneously, to avoid the impact of introducing additional ground robots on the existing work environment, existing trucks are modified to perform close-range, detailed mapping of each sub-area during operations, supplementing the UAV's blind spot view and achieving higher mapping efficiency. In actual operation, when the UAV is inspecting above the yard, the upper-level management system activates the trucks within the UAV's area. The UAV and trucks can then collect data and create maps, with the trucks' maps supplementing the UAV's maps.

[0059] Step 120: Use an air-ground collaborative long-term SLAM system to collect point cloud data, sensor data and image data of the container yard, and build an initial map of the container yard based on the point cloud data and image data.

[0060] The sensor data includes the aforementioned three-axis attitude angles (or angular rates) and accelerations.

[0061] In practice, when a drone patrols above the yard, the upper-level management system or control system activates the trucks in the drone's area, allowing the drone and trucks to collect data; specifically, for example... Figure 2 As shown, step 120 above may further include steps 210 to 230:

[0062] Step 210: In the air-ground collaborative long-term SLAM system, the UAV collects point cloud data above the container yard through the first radar and collects image data of the container yard through the image acquisition equipment.

[0063] Point cloud data refers to a set of vectors in a three-dimensional coordinate system. Scanned data is recorded in the form of points, each containing three-dimensional coordinates, and some may contain color information (RGB) or reflectance intensity information.

[0064] For example, a single drone can be used to scan the entire yard, with its flight path set according to the row and column arrangement of containers. The yard can be divided into different sub-areas based on the loading and unloading lane lines. The drone's flight path can be above the central symmetry line of each sub-area, primarily scanning feature points above the containers. In practice, mapping efficiency can reach 90m. 2 / s.

[0065] The image acquisition device can be any device capable of acquiring images, such as a visible light camera or a CCD camera; there are no restrictions here.

[0066] Step 220: In the air-ground collaborative long-term SLAM system, the trucks collect point cloud data of the vertical plane of the containers in the container yard through the second radar.

[0067] The point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the containers.

[0068] Container trucks can use the point cloud data of the vertical plane of the containers they collect to create maps of the containers on both sides of the aisles in the yard, supplementing the initial map built by the drone.

[0069] It should be noted that both drones and container trucks have mapping capabilities. After collecting image data, sensor data, and / or point cloud data, an initial map of the container yard can be constructed based on the image data, sensor data, and / or point cloud data. In practice, the mapping function can be started simultaneously with the data acquisition function of the drone and container truck.

[0070] Step 230: The first and second inertial sensors in the air-ground collaborative long-term SLAM system acquire sensing data.

[0071] The sensor data includes angular rate and acceleration.

[0072] The IMU is the core device of inertial positioning technology. After error compensation and inertial navigation calculation, it finally outputs navigation information such as the change in coordinates and velocity of the carrier relative to its initial position.

[0073] Step 130: Determine the planar features of the container yard based on the size information of the containers in the container yard and the point cloud data.

[0074] The container's dimensions include its length-to-width-to-height ratio. In practice, this dimension information can be pre-recorded in a database.

[0075] This step mainly utilizes the intensity information of point clouds in the point cloud data. The reason for introducing the intensity information of point clouds is that during the scanning process, factors such as different angles, distances, and materials will affect the intensity of point clouds. Introducing intensity information can more efficiently segment the container plane.

[0076] This step can be achieved through the following process: First, delete the ground point cloud in the point cloud data; second, using the container size information, point cloud intensity information, and plane normal vector as constraints, apply a preset algorithm (such as the RANSAC algorithm) to segment the plane in the current frame, and select the point cloud plane of the highest layer container and the side container point cloud plane of each container position as the effective plane from the perspective of the UAV and container truck; finally, determine the corner coordinates of each container position in the yard through these effective planes, thus obtaining the planar features of the container yard.

[0077] Step 140: Based on the sensor data and the planar features of the container yard, determine the target relative pose between the current frame and the nearest adjacent plane in the initial map, and construct the target map of the container yard based on the target relative pose.

[0078] This step can be achieved through the following process: First, find the nearest neighboring face in the current frame and the initial map; second, using the corner points and plane normal vector constraints of the plane, match the extracted planar features in the current frame with the planar features of the nearest neighboring face in the initial map to obtain the relative pose between the two planes; here, the relative pose can be understood as a plane-to-plane constraint; finally, construct the target map of the container yard based on the target relative pose.

[0079] The target map of the container yard is more accurate than the initial map of the container yard.

[0080] The container yard map construction method provided in this application first constructs an air-ground cooperative long-term SLAM system including trucks and UAVs; second, it uses the air-ground cooperative long-term SLAM system to collect point cloud data, sensor data, and image data of the container yard, and constructs an initial map of the container yard based on the point cloud data and image data; third, it determines the planar features of the container yard based on the size information of the containers in the container yard and the point cloud data; finally, it determines the target relative pose between the nearest adjacent plane in the current frame and the initial map based on the sensor data and the planar features of the container yard, and constructs a target map of the container yard based on the target relative pose. Thus, by modifying the existing hardware system, an air-ground cooperative long-term SLAM system constructed by UAVs and trucks is obtained. An initial map of the container yard is constructed based on the point cloud data, sensor data, and image data collected by the air-ground cooperative long-term SLAM system. Planar features are then extracted, relative poses are determined based on the planar features, and a precise map (target map) of the container yard is generated based on the relative pose. The constructed container yard map is more accurate, thereby improving the efficiency of container yard management.

[0081] The following example further illustrates the process of determining the planar features of a container yard based on the size information of the containers and point cloud data in the above embodiments. Figure 3 This is a flowchart illustrating the method for constructing a map of a container yard provided in an embodiment of this application. Figure 3 As shown, step 130 above may further include steps 310 to 340:

[0082] Step 310: Delete the ground point cloud from the point cloud data.

[0083] In particular, due to the high flatness of the port ground, the ground is the most important complete plane in the point cloud data. The RANSAC algorithm can be used to segment the main plane (i.e., the ground point cloud) in the point cloud data.

[0084] Step 320: Using the size information of different types of containers and the direction of the plane normal vector as constraints, a preset algorithm is used to fit the intensity information of the point cloud in the point cloud data to obtain the fitted point cloud data.

[0085] The container's dimensions can be its prior dimensions, such as its length, width, height, and aspect ratio; there are no restrictions on these dimensions.

[0086] The preset algorithm can be a parameter estimation algorithm applied in the field of computer vision, such as RANSAC (RandomSample Consensus) algorithm, clustering algorithm, least squares method, etc., without limitation here.

[0087] For example, the process of fitting using the RANSAC algorithm is as follows: First, a small subset of samples from the data is randomly selected, and a model is fitted based on these samples; second, the distance from other data points to the model is calculated, and data points with a distance less than a certain threshold are classified as inliers, while data points with a distance greater than the threshold are classified as outliers; this process is repeated multiple times, and the model with the most inliers is selected as the final estimation result.

[0088] Step 330: Segment the point cloud plane of the highest layer container and the vertical container point cloud plane of each container position from the fitted point cloud data to obtain the target point cloud plane.

[0089] This step involves segmenting the point cloud plane {F1,......,F1} from the fitted point cloud data in the current frame. i}, and then filter out the point cloud plane {F} of the highest-level container in each container position from the point cloud plane in the current frame. u1 ,......,F um} and vertical container point cloud plane {F s1 ,......,F sw} is the effective plane (target plane) from the perspective of both the drone and the truck.

[0090] During the scanning process by drones and container trucks, the reflectivity of planes facing different directions varies. Therefore, at the junctions of different planes of a container, the reflectivity of the point cloud will exhibit a significant gradient change. This gradient change information can be used to better distinguish the edges of different planes of the container, thus improving the efficiency of plane segmentation.

[0091] Step 340: Based on the target point cloud plane, determine the corner coordinates of each container location in the container yard to obtain the planar features of the container yard.

[0092] In this step, the corner coordinates of each container location can be directly determined from the target point cloud plane. Specifically, the coordinates of the center point cloud of the point cloud where the corner is located can be determined as the corner coordinates, and then the corner coordinates of each container location can be directly determined as the planar features of the container yard.

[0093] The container yard map construction method provided in this application firstly deletes ground point clouds from the point cloud data; secondly, using the size information of different types of containers and the direction of the plane normal vector as constraints, a preset algorithm is used to fit the intensity information of the point cloud to obtain fitted point cloud data; thirdly, the point cloud plane of the highest layer container and the vertical container point cloud plane of each container position are segmented from the fitted point cloud data to obtain the target point cloud plane; finally, based on the target point cloud plane, the corner coordinates of each container position in the container yard are determined to obtain the planar features of the container yard. Thus, by introducing the intensity information of the point cloud, the point cloud plane can be better segmented, thereby accurately obtaining the planar features of the container yard.

[0094] The following example further illustrates the process of determining the relative pose of the target between the current frame and the nearest adjacent plane in the initial map based on sensor data and the planar features of the container yard, as described in the above embodiments. Figure 4 This is a flowchart illustrating the method for constructing a map of a container yard provided in an embodiment of this application. Figure 4 As shown, step 140 above may further include steps 410 to 430:

[0095] Step 410: Determine the nearest neighboring point cloud plane from the initial map that is the point cloud plane in the current frame.

[0096] This step can be achieved through the following process: Step (1) Based on the right-hand coordinate system, with the travel direction of the UAV and the container truck as the x-axis, from the perspective of the UAV, randomly select a preset number of point cloud planes outside the row with the smallest x-coordinate in the xy plane within the field of view. Similarly, from the perspective of the container truck, randomly select a preset number of point cloud planes outside the row with the smallest x-coordinate in the xz plane; Step (2) Calculate the center point coordinates {A} of the preset number of point cloud planes. i (x i ,y i ,z i |1≤i≤n}、Corner coordinates (usually 4 corner points){B ij (x ij ,y ij ,z ij |1≤i≤n,1≤j≤4}, normal vector direction Step (3): Using the center point of each cloud plane as a reference, find the nearest neighboring point C in the initial map. i (x i ,y i ,z iStep (4) determines the point cloud plane corresponding to the internal point in the initial map. Here, step (1) ensures the integrity of the point cloud plane in the before-and-after comparison.

[0097] Step 420: Determine the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map based on the sensor data.

[0098] This step can directly determine the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map using the motion model of the inertial sensor; here, the initial relative pose can also be understood as the preliminary relative pose.

[0099] Step 430: Match the point cloud plane in the current frame with the adjacent point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between them.

[0100] The relative pose of the target can be obtained from the corner coordinates and normal vector direction of the point cloud plane. Specifically, Figure 5 This is a flowchart illustrating the method for constructing a map of a container yard provided in an embodiment of this application. Figure 5 As shown, step 430 above may further include steps 510 to 540:

[0101] Step 510: Determine the corner coordinates and normal vector direction of the point cloud plane corresponding to the point cloud plane in the current frame in the initial map.

[0102] For example, calculate the corner coordinates {D} of the point cloud plane corresponding to the interior point in the initial map. ij (x ij ,y ij ,z ij |1≤i≤n,1≤j≤4}, normal vector direction

[0103] Step 520: Determine the translation amount based on the corner coordinates, the corner coordinates of the point cloud plane from the perspective of a preset number of UAVs, and the corner coordinates of the point cloud plane from the perspective of a preset number of trucks.

[0104] Among them, the preset number of point cloud planes under the view of UAVs and the preset number of point cloud planes under the view of trucks are the point cloud planes under the view of UAVs and the point cloud planes under the view of trucks selected in step (1) of step 410 above.

[0105] For example, the translation amount is determined based on the corner coordinates calculated in step (2) and step 510 in step 410.

[0106] For example, the translation amount can be calculated using the mean value method, and the calculation formula is shown in formula (1) below:

[0107]

[0108] Step 530: Determine the rotation amount based on the normal vector direction, the normal vector direction of the point cloud plane under the view of a preset number of UAVs and the point cloud plane under the view of a preset number of trucks.

[0109] For example, the rotation amount is determined based on the normal vector direction calculated in step (2) and step 510 in step 410.

[0110] For example, the rotation amount can be calculated using the mean value method, and the calculation formula is shown in formula (2) below:

[0111]

[0112] Step 540: Determine the relative pose of the target based on the translation and rotation amounts.

[0113] The translation and rotation amounts constitute the target relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map.

[0114] The container yard map construction method provided in this application firstly determines the nearest neighboring point cloud plane in the current frame from the initial map; secondly, it determines the initial relative pose between the point cloud plane in the current frame and the neighboring point cloud plane in the initial map based on sensor data; and thirdly, it matches the point cloud plane in the current frame with the neighboring point cloud plane in the initial map to obtain the target relative pose between them. Thus, based on the planar features of the container yard, the initial relative pose between the point cloud plane in the current frame and the neighboring point cloud plane in the initial map is determined. Then, through planar matching (point cloud registration), a more accurate target relative pose can be obtained, and a more accurate container yard map can be generated subsequently based on this target relative pose.

[0115] The container yard map construction method provided in this application embodiment can further optimize the target relative pose to obtain a more accurate target relative pose. The container yard map construction method provided in this application embodiment may further include the following steps:

[0116] The edge point cloud is determined from the point cloud plane, and the edge point cloud in the current frame is registered with the edge point cloud in the initial map to obtain the optimized target relative pose.

[0117] Point cloud registration can be PCA registration, ICP registration, etc., and there is no limitation here.

[0118] In this step, the edge point cloud (frame point cloud) is determined from the point cloud plane, that is, the planar point cloud {F1,......,F...} within the point cloud. i} Deleting points reduces the computational load for point cloud registration. Furthermore, point cloud registration provides a more accurate relative pose without compromising real-time performance. In other words, the target relative pose in this step is more accurate than the target relative pose in step 430.

[0119] The container yard map construction method provided in this application embodiment can also update the container yard map in real time. Figure 6 This is a flowchart illustrating the method for constructing a map of a container yard provided in an embodiment of this application. Figure 6 As shown, the container yard map construction method provided in this application embodiment may further include the following steps 610 to 630:

[0120] Step 610: Generate a depth image based on the point cloud data of consecutive frames from the same viewpoint.

[0121] This step generates a depth image based on the projection of point cloud data from consecutive frames of the same viewpoint. Specifically, this step can be implemented through the following process: First, the point cloud is divided into n groups of point clouds, forming batch data {S1, ..., S...}. n}, each group of point clouds S n The data includes a point cloud of the storage yard from one radar scan cycle; secondly, for each batch of data, a point cloud frame S is extracted from it through multiple random traversals. i As a reference frame, the other frames {S1,......,S i-1 ,S i+1 ,......,S n} Used to build a local map of the storage yard My local (i.e., the initial map); finally, the local map My... local To the reference frame S i The point cloud coordinate system is transformed, with the depth difference between the highest and lowest points at each location being the pixel depth, and the projection is a high-resolution r of a fixed window size. h Depth image.

[0122] In practice, before this step, the container yard point cloud can be segmented based on the identification information in the image data; here, the identification information can be the lane markings in the loading and unloading area. Specifically, this step can use an object detection algorithm to detect the coordinates of the lane markings in the loading and unloading area, then map the lane marking coordinates to the point cloud data, and then use the lane marking coordinates on both sides of the yard as the vertices of the rectangular area of ​​the container yard to segment the yard point cloud Sy local From the entire point cloud S localThe points are then divided into sections. Furthermore, a height threshold can be set to delete point clouds exceeding the threshold, thus avoiding the impact of container yard loading and unloading machines on the algorithms involved in map updates.

[0123] Step 620: Compare the pixel depth of the depth image with the same location in the initial map, determine the dynamic point cloud through point cloud occlusion and delete it to obtain the target point cloud.

[0124] This step can be implemented through the following process: First, compare the depth image Iy generated from the reference frame with the depth image Iy. i With local map depth image Iy local The pixel depth; secondly, the local map depth value dy local Greater than the map depth value dy of the reference frame i At that time, determine the local map My corresponding to the storage yard. local There are dynamic occlusions in the data; furthermore, the depth difference point cloud {P} d1 ,......,P dm} is deleted as a dynamic point cloud, resulting in the target point cloud (i.e., a static local map) Mys local .

[0125] In practice, due to the possibility of mistakenly deleting static point clouds during the process of deleting dynamic point clouds from high-resolution depth maps, SLAM pose errors and point cloud distortions can affect the depth map. Therefore, an algorithm for recovering mistakenly deleted point clouds based on multi-resolution depth maps can be designed. Specifically, this can be achieved by gradually reducing the resolution r of the depth image to recover static point clouds {P} that were mistakenly identified as dynamic point clouds. ds1 ,......,P dsw}Re-label as static point cloud {P s1 ,......,P sw This is done to reduce the error rate of the algorithm.

[0126] Step 630: Update the container yard map based on the target point cloud and historical map.

[0127] This step can be achieved through the following process: First, process the static local map Mys... local By stitching the images together, a static global map Mys is obtained. glocal Secondly, compared with the static global map Mys glocal Pixel depth at the same location as the historical site map (historical global map); again, retain the static global map Mys. glocal The dynamic point cloud relative to the historical map and the static point cloud of the two are used to update the map of the container yard.

[0128] The container yard map construction method provided in this application first generates a depth image based on point cloud data from consecutive frames of the same viewpoint; second, it compares the pixel depth of the depth image with that of the same location in the initial map, identifies and deletes dynamic point clouds through point cloud occlusion, and obtains the target point cloud; third, it updates the container yard map based on the target point cloud and the historical map. In this way, dynamic point clouds can be deleted in a timely manner, avoiding false detections, and thus obtaining a more accurate target point cloud and a more accurate updated map.

[0129] The container yard map construction method provided in this application embodiment can also improve the efficiency of container yard map construction. Figure 7 This is a flowchart illustrating the method for constructing a map of a container yard provided in an embodiment of this application. Figure 7 As shown, the container yard map construction method provided in this application embodiment may further include the following steps 710 to 740:

[0130] Step 710: Use an air-ground collaborative long-term SLAM system to collect location descriptors of the container yard and construct the current topology of the container yard through a chessboard structure.

[0131] The location descriptor includes the container's feature data and location information. The container's feature data can be its aspect ratio, the reflectivity of its paint, its stacking height, etc.; the container's location information can be the center coordinates, vertex coordinates, etc. of its corresponding location.

[0132] The location descriptor can be a three-dimensional matrix that records the characteristic data and location information of the aforementioned containers.

[0133] The location descriptor of a container yard is determined by the location descriptor of each container location.

[0134] This step can be achieved through the following process: UAV component: The UAV scans outwards from the container position directly below the current location, recording the stacking height and aspect ratio of the containers at each position, thus obtaining a position descriptor (topology) from the UAV's perspective. Truck component: Introducing the reflectivity of the container paint as a constraint, the truck uses the containers on the left and right sides of the current location as two centers, and records the stacking height, aspect ratio, and reflectivity of the container paint at corresponding positions along the perimeter of the container yard, thus obtaining a position descriptor (topology) from the truck's perspective. The topology from the UAV's perspective and the topology from the truck's perspective together constitute the current topology of the container yard.

[0135] Step 720: Determine the loop loss value based on the difference between the current topology and the historical topology of the container yard and the preset loop loss function.

[0136] This step can be achieved through the following process: assign the container aspect ratio a:b:c, the container paint reflectivity k (which can be given a larger weight), and the container stacking height h (which can be given a smaller weight) to the topology relationship, and calculate the loop loss value according to the preset loop loss function calculation formula.

[0137] Step 730: Use drones to detect the location of the container trucks and determine the location of the container trucks as the initial location of the open space at both ends of the map for splicing container yards.

[0138] This step can be achieved through the following process: the position of the container truck is detected by the image acquisition device at the bottom of the drone, and the position of the container truck is determined as the initial position for map registration at both the air and ground ends.

[0139] Step 740: Perform loop closure detection on the container yard based on the loop closure loss value, and optimize the map of the container yard based on the loop closure loss function, the initial location of the open space, the preset chessboard topology, and the preset outer ring distribution topology.

[0140] Among them, the preset chessboard-like topological relationship can be as follows: Figure 8 As shown; the preset outer ring distribution topology can be as follows: Figure 9 As shown, the preset chessboard-like topology and the preset outer ring-shaped distribution topology are used to improve the current topology of the container yard.

[0141] The loop closure detection process can be performed according to the process of collecting location descriptors in the air-ground collaborative long-term SLAM system in step 710 above. After each collection of location descriptors, the current topology relationship is compared with the historical topology relationship to obtain the loop closure loss value. Then, based on the loop closure loss value, it is determined whether to perform the next loop closure detection until the final loop closure loss value meets the preset convergence condition.

[0142] For example, the loop closure loss function can be Among them, the current container space has a length-width-height ratio of 1:l1:l2, and the container space with the same length-width-height ratio is 1:l1′:l2′; the current container paint reflectivity is k, and the container paint reflectivity with the same reflectivity is k′.

[0143] The initial location of the open space is used to complete the map of the container yard.

[0144] The container yard map construction method provided in this application firstly uses an air-ground collaborative long-term SLAM system to collect location descriptors of the container yard and constructs the current topology of the container yard through a chessboard structure. Secondly, based on the difference between the current and historical topology of the container yard and a preset loop closure loss function, a loop closure loss value is determined. Thirdly, a drone is used to detect the positions of container trucks, and the positions of the container trucks are determined as the initial positions of the empty spaces at both ends of the map used to stitch together the container yard. Finally, loop closure detection is performed on the container yard based on the loop closure loss value, and the map of the container yard is optimized based on the loop closure loss function, the initial positions of the empty spaces, the preset chessboard topology, and the preset outer ring distribution topology. In this way, the map construction efficiency can be improved through loop closure detection and the topology of each container location, and the map can be improved through the preset chessboard topology, the preset outer ring distribution topology, and the initial positions of the empty spaces.

[0145] After introducing the container yard map construction method according to exemplary embodiments of this disclosure, the following will refer to... Figure 10 A visualization platform 1000 for a container yard according to an exemplary embodiment of the present disclosure will be described.

[0146] refer to Figure 10 The container yard visualization platform 1000 includes: a processing module 1010 configured to match container numbers, container positions, and actual location information; and a visualization module 1020 configured to visualize the container yard based on the matching relationship between container numbers, container positions, and actual location information using location descriptors.

[0147] In practice, a container yard visualization platform can specifically include the following functions:

[0148] (1) Container risk assessment based on convolution method: Extract the height layer matrix from the location descriptor and perform zero-padding on the periphery. This is achieved by setting the convolution kernel. b,c∈{k∈R|0<k<1}, perform convolution operations on the height layer matrix after padding with zeros, and take the absolute value of the calculation result. The influence weight of the number of stacking layers in the front, back, left and right positions of the container on the degree of danger can be set by the parameter (bc).

[0149] (2) Container movement feedback system based on existing management system: After the upper management system (upper control system) assigns the location of newly arrived containers, the SLAM system can scan the yard and detect whether the containers are correctly installed at the corresponding locations; at the same time, it can bind information such as container number, origin, and cargo type to the actual location to improve the entire yard management system and form a data closed loop.

[0150] (3) Simulation visualization system based on location descriptors: Based on location descriptors, simulation algorithms are used to generate cuboid frames at the corresponding locations of containers to replace point clouds for visualization, thus constructing a semantic map of container distribution in the yard; at the same time, it is displayed in real time on the upper-level monitoring interface.

[0151] The visualization platform for container yards can not only optimize information transmission efficiency, but also provide a more intuitive display; in addition, it can integrate information such as container number, origin, and cargo type, and display them completely in the visualization interface.

[0152] This application provides another implementation of a method for constructing a map of a container yard, such as... Figure 11 As shown, it includes the following steps:

[0153] Step (1) Construct an integrated multi-sensor air-ground collaborative system to collect point cloud data and image data of the yard: The UAV uses a solid-state radar that scans vertically to the ground to quickly collect the overall point cloud data above the yard, and at the same time collects real-time image data of the yard through a visible light camera that scans vertically to the ground; when the UAV patrols above the yard, the upper management system activates the mapping function of the trucks in the area where the UAV is located, and cooperates with the yard UAV to collect the vertical plane information of the containers in the yard through the mechanical radar installed on the roof of the truck;

[0154] Step (2) Design an air-ground collaborative map update algorithm based on inter-frame depth map mapping complement difference to solve the problem of yard change detection in large port scenarios: Use the visible light camera on the UAV to detect the lane lines in the yard loading and unloading area and the container trucks in the area, and divide the yard area; Project the scanned point cloud data before and after consecutive frames at the same viewpoint to generate a depth map and compare the pixel depth at the same position. By judging the point cloud occlusion, dynamic point clouds can be distinguished and deleted, and the global map can be updated by using the current static map to cover the corresponding position of the historical map.

[0155] Step (3) Design a container segmentation algorithm that integrates prior size information to quickly detect container features in the current frame: Using the aspect ratio and normal vector direction of different container models as constraints, the RANSAC algorithm is used to segment the plane above the highest layer container in the current frame point cloud based on point cloud intensity information. The coordinates of the four corner points of each container in the yard can be obtained through the top layer plane.

[0156] Step (4) In the special environment of the container yard, since the plane occupies most of the environmental features, the accurate pose relationship can be quickly obtained through surface feature matching; design a SLAM front-end odometry based on container surface features to optimize the initial pose estimation: the UAV and the truck randomly select a pre-set number of n planes in the highest plane and the side plane of the yard, respectively, and extract the four corner points of each plane; by registering the current frame plane with the four corner points of the nearest adjacent plane in the corresponding local map, a relatively accurate relative pose can be constrained; finally, by performing ICP registration on the skeleton map with the container plane points and ground points removed, the accurate relative pose can be obtained.

[0157] Step (5) Design an air-ground collaborative loop closure detection method based on the topological relationship of adjacent container positions to improve map building efficiency: Use the global topological relationship of the yard as the global coordinate descriptor for each container position, and record the center coordinates, eight vertex coordinates, length-width-height ratio, container paint reflectivity and stacking height of each container position through a three-dimensional matrix; Convert the topological structure map corresponding to the UAV and the truck respectively from the top view and the side view, calculate the loss value and detect loop closure through the difference in topological relationship; This step not only has extremely high computational efficiency and can be updated during the map building process, but also can play a loop closure constraint on the map from the perspectives of UAV and unmanned vehicle. At the same time, the UAV is used to detect the relative position of the truck in the yard, which can provide the initial position of the truck map when stitching the map at both ends of the air and ground;

[0158] Step (6) Design a visualization management platform based on the results of multi-machine collaborative SLAM to integrate with the existing intelligent digital management system: use the height layer descriptor to calculate the hazard of each location and match the container location with the real coordinates and container information in the digital management system, visualize the container in the yard in the form of a three-dimensional frame and mark it on the map to construct a semantic map of the yard cargo distribution; integrate the semantic map with the information and visualize it on the upper platform.

[0159] After introducing the container yard map construction method according to exemplary embodiments of this disclosure, the following will refer to... Figure 12 The container yard map building apparatus 1200 of the present disclosure will be described in an exemplary embodiment.

[0160] refer to Figure 12The container yard map building device 1200 includes: a hardware system building module 1210, configured to build an air-ground collaborative long-term SLAM system including container trucks and UAVs; the UAVs are equipped with a first radar, image acquisition equipment and a first inertial sensor on their bottom, and the container trucks are equipped with a second radar and a second inertial sensor on their top; a data acquisition module 1220, configured to use the air-ground collaborative long-term SLAM system to acquire point cloud data, sensor data and image data of the container yard, and build an initial map of the container yard based on the point cloud data and image data; a feature extraction module 1230, configured to determine the planar features of the container yard based on the size information of the containers in the container yard and the point cloud data; and a map building module 1240, configured to determine the target relative pose between the nearest adjacent plane in the current frame and the initial map based on the sensor data and the planar features of the container yard, and build a target map of the container yard based on the target relative pose.

[0161] In one implementation, the data acquisition module 1220 is configured such that: a UAV in the air-ground collaborative long-term SLAM system acquires point cloud data above the container yard via a first radar and acquires image data of the container yard via an image acquisition device; a truck in the air-ground collaborative long-term SLAM system acquires point cloud data of the vertical plane of the containers in the container yard via a second radar; the point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the containers; a first inertial sensor and a second inertial sensor in the air-ground collaborative long-term SLAM system acquire sensing data; the sensing data includes angular rate and acceleration.

[0162] In one implementation, the feature extraction module 1230 is configured to: delete ground point clouds from the point cloud data; use a preset algorithm to fit the intensity information of point clouds in the point cloud data with the size information of different types of containers and the direction of the plane normal vector as constraints to obtain fitted point cloud data; segment the point cloud plane of the highest layer container and the vertical container point cloud plane of each container position from the fitted point cloud data to obtain the target point cloud plane; and determine the corner coordinates of each container position in the container yard based on the target point cloud plane to obtain the planar features of the container yard.

[0163] In one implementation, the map building module 1240 is configured to: determine the nearest neighboring point cloud plane to the point cloud plane in the current frame from the initial map; determine the initial relative pose between the point cloud plane in the current frame and the neighboring point cloud plane in the initial map based on sensing data; and match the point cloud plane in the current frame with the neighboring point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between them.

[0164] In one implementation, the map building module 1240 is configured to: determine the corner coordinates and normal direction of the point cloud plane corresponding to the point cloud plane in the current frame in the initial map; determine a translation amount based on the corner coordinates, a preset number of point cloud planes from the perspective of UAVs, and a preset number of point cloud planes from the perspective of trucks; determine a rotation amount based on the normal direction, the preset number of point cloud planes from the perspective of UAVs, and the preset number of point cloud planes from the perspective of trucks; and determine the relative pose of the target based on the translation amount and the rotation amount.

[0165] In one embodiment, the map building apparatus 1200 further includes a pose optimization module, which is configured to: determine edge point clouds from the point cloud plane, perform point cloud registration between the edge point clouds in the current frame and the edge point clouds in the initial map, and obtain the optimized target relative pose.

[0166] In one embodiment, the map building apparatus 1200 further includes a map updating module, which is configured to: generate a depth image based on point cloud data of consecutive frames from the same viewpoint; compare the depth image with the pixel depth at the same location in the initial map, determine and delete dynamic point clouds through point cloud occlusion, and obtain a target point cloud; and update the map of the container yard based on the target point cloud and the historical map.

[0167] In one implementation, the map update module is configured to: determine the current map based on the target point cloud; compare the pixel depth of the same location in the current map with that in the historical map, retain the dynamic point cloud of the current map relative to the historical map and the static point cloud of the two, and obtain an updated map of the container yard.

[0168] In one embodiment, the map building device 1200 further includes a map optimization module, which is configured to: collect location descriptors of the container yard using an air-ground collaborative long-term SLAM system, and construct the current topology of the container yard through a chessboard structure; the location descriptors include the feature data and location information of the containers; determine the loop closure loss value based on the difference between the current topology and the historical topology of the container yard and a preset loop closure loss function; use a drone to detect the position of the container trucks and determine the position of the container trucks as the initial position of the empty space at both ends of the map for stitching the container yard; perform loop closure detection on the container yard based on the loop closure loss value, and optimize the map of the container yard based on the loop closure loss function, the initial position of the empty space, the preset chessboard topology, and the preset outer ring distribution topology.

[0169] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0170] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0171] Figure 13 This is a schematic diagram of a computer device provided in an embodiment of this application. The device can be integrated into a terminal device or a chip of a terminal device. The terminal can be a computing device with data processing capabilities.

[0172] The device includes: a processor 1301, a storage medium 1302, and a bus 1303.

[0173] Storage medium 1302 stores program instructions executable by processor 1301. When computer device 1300 is running, processor 1301 communicates with storage medium 1302 via bus 1303, and processor 1301 executes the program instructions to perform the above-described method embodiment. The specific implementation and technical effects are similar and will not be described in detail here.

[0174] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, is used to perform the above-described method embodiments.

[0175] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0178] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for constructing a map of a container yard, characterized in that, include: Construct a long-term air-ground collaborative SLAM system including container trucks and drones; the drones are equipped with a first radar, image acquisition equipment and a first inertial sensor on their bottom, and the container trucks are equipped with a second radar and a second inertial sensor on their top. The air-ground collaborative long-term SLAM system is used to collect point cloud data, sensor data and image data of the container yard, and an initial map of the container yard is constructed based on the point cloud data and the image data. Based on the size information of the containers in the container yard and the point cloud data, the planar features of the container yard are determined, including... Delete the ground point cloud from the point cloud data; Using the size information and the direction of the plane normal vector of different types of containers as constraints, a preset algorithm is used to fit the intensity information of the point cloud in the point cloud data to obtain the fitted point cloud data. The target point cloud plane is obtained by segmenting the point cloud plane of the highest layer container and the vertical container point cloud plane of each container position from the fitted point cloud data. Based on the target point cloud plane, the corner coordinates of each container location within the container yard are determined, thus obtaining the planar features of the container yard; Based on the sensor data and the planar features of the container yard, the target relative pose between the current frame and the nearest adjacent plane in the initial map is determined, and a target map of the container yard is constructed based on the target relative pose, including: Determine the nearest neighboring point cloud plane to the point cloud plane in the current frame from the initial map; The initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map is determined based on the sensor data. Matching the point cloud planes in the current frame with adjacent point cloud planes in the initial map to optimize the initial relative pose and obtain the target relative pose between them, including: Determine the corner coordinates and normal vector direction of the point cloud plane in the initial map that corresponds to the point cloud plane in the current frame; The translation amount is determined based on the corner coordinates, the corner coordinates of the point cloud plane from the perspective of a preset number of UAVs, and the corner coordinates of the point cloud plane from the perspective of a preset number of trucks. The rotation amount is determined based on the normal vector direction, the normal vector direction of the point cloud plane under the view of a preset number of UAVs, and the normal vector direction of the point cloud plane under the view of a preset number of trucks. The relative pose of the target is determined based on the translation and rotation amounts.

2. The method according to claim 1, characterized in that, The point cloud data, sensor data, and image data of the container yard collected using the air-ground collaborative long-term SLAM system include: The UAV in the air-ground collaborative long-term SLAM system collects point cloud data above the container yard through the first radar and collects image data of the container yard through the image acquisition device; In the air-ground collaborative long-term SLAM system, the trucks collect point cloud data of the vertical plane of the containers in the container yard through the first radar; the point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the containers. The first and second inertial sensors in the air-ground collaborative long-term SLAM system collect sensing data; the sensing data includes angular rate and acceleration.

3. The method according to claim 1, characterized in that, After obtaining the target relative pose between the two, the method further includes: The edge point cloud is determined from the point cloud plane, and the edge point cloud in the current frame is registered with the edge point cloud in the initial map to obtain the optimized target relative pose.

4. The method according to claim 1, characterized in that, After constructing the target map of the container yard based on the target relative pose, the method further includes: A depth image is generated based on the point cloud data of consecutive frames from the same viewpoint; By comparing the pixel depth of the depth image with that of the initial map at the same location, dynamic point clouds are determined and deleted through point cloud occlusion to obtain the target point cloud; The map of the container yard is updated based on the target point cloud and historical map.

5. The method according to claim 4, characterized in that, The step of updating the map of the container yard based on the target point cloud and the historical map includes: The current map is determined based on the target point cloud; the pixel depth of the same location in the current map and the historical map are compared, and the dynamic point cloud of the current map relative to the historical map and the static point cloud of the two are retained to obtain an updated map of the container yard.

6. The method according to claim 1, characterized in that, After constructing the target map of the container yard based on the target relative pose, the method further includes: The air-ground collaborative long-term SLAM system is used to collect the location descriptors of the container yard, and the current topology of the container yard is constructed through a chessboard structure; the location descriptors include the characteristic data and location information of the containers; The loopback loss value is determined based on the difference between the current and historical topology of the container yard and a preset loopback loss function. The drone is used to detect the position of the container truck, and the position of the container truck is determined as the initial position of the open space at both ends of the map for splicing the container yard. The container yard is subjected to loop closure detection based on the loop closure loss value, and the map of the container yard is optimized based on the loop closure loss function, the initial location of the open space, the preset chessboard topology, and the preset outer ring distribution topology.

7. A map-building apparatus for a container yard, based on the method described in any one of claims 1 to 6, characterized in that, include: The hardware system construction module is configured to build an air-ground collaborative long-term SLAM system including a truck and an unmanned aerial vehicle (UAV); the UAV is equipped with a first radar, an image acquisition device and a first inertial sensor on its bottom, and the truck is equipped with a second radar and a second inertial sensor on its top. The data acquisition module is configured to use the air-ground collaborative long-term SLAM system to acquire point cloud data, sensor data and image data of the container yard, and to construct an initial map of the container yard based on the point cloud data and the image data. The feature extraction module is configured to determine the planar features of the container yard based on the size information of the containers in the container yard and the point cloud data. The map building module is configured to determine the target relative pose between the current frame and the nearest adjacent plane in the initial map based on the sensor data and the planar features of the container yard, and to build a target map of the container yard based on the target relative pose.

Images