Map construction method and device for container yard

Abstract

The invention provides a map construction method and device for a container yard, and relates to the technical field of map construction. The method comprises the following steps: constructing an air-ground cooperative long-term SLAM system comprising a container truck and an unmanned aerial vehicle; acquiring point cloud data, sensing data and image data of the container yard by adopting the air-ground collaborative long-term SLAM system, and constructing an initial map of the container yard based on the point cloud data and the image data; determining plane features of the container yard according to the size information of the containers in the container yard and the point cloud data; and determining a target relative pose between the current frame and the nearest adjacent plane in the initial map according to the sensing data and the plane characteristics of the container yard, and constructing a target map of the container yard according to the target relative pose. Therefore, the map of the container yard can be accurately constructed, and the working efficiency of container yard management is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of map construction, and in particular to a method and device for constructing a map of a container yard. Background Art

[0002] With the vigorous development of robots, the application of intelligent unmanned systems represented by unmanned vehicles and drones in port container yards has gradually become a research hotspot. However, the particularity of the port environment has brought many challenges to the application of unmanned systems. Due to the particularity of the port, the safety requirements are extremely high. Although the current containers in the port are managed intelligently, the containers entering the port will be numbered by the upper management system (upper control system) and uniformly coordinated and allocated to workers in various divisions or unmanned loading trucks for placement, but the system cannot detect the actual placement of the containers; for example: when the system dispatches workers to operate a certain container, the workers' operations on other unrelated containers during the operation cannot be detected by the management system, which can easily cause safety hazards in the port container yard.

[0003] At present, container yards mostly use patrol staff, surveillance cameras, and single patrol robots to detect the placement of containers in the yard and potential dangerous scenes. However, patrol staff inspections not only require the introduction of more workers for management, but also have low detection efficiency and cannot guarantee a high response speed, which is inconsistent with the requirements of current smart ports. Although the system is simple when using a single patrol robot to scan the yard, the port operating environment is complex. In scenarios where the port yard is blocked or the area is large, it is impossible to effectively perform detailed mapping and rapid detection of the container yard. In addition, the introduction of additional patrol vehicles will interfere with the operation of the existing system.

[0004] The problem of air-ground collaborative mapping has always been a key issue in the field of intelligent robots. A Chinese invention patent proposes an air-ground heterogeneous collaborative mapping method, device, equipment and storage medium (CN117191005A). This method extracts key frames by setting thresholds and uses GNSS coordinates to find the key frame positions of air-ground platforms, and performs closed-loop constraints on air-ground heterogeneous data, which can reduce calculation time and improve matching efficiency. However, this method does not make good use of environmental features and cannot achieve good results in the special environment of port yards. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for constructing a container yard map in view of the above-mentioned deficiencies in the prior art, so as to accurately construct a map of the container yard, thereby improving the work efficiency of container yard management.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for constructing a map of a container yard, comprising: constructing an air-ground collaborative long-term SLAM system including a container truck and an unmanned aerial vehicle; the unmanned aerial vehicle is equipped with a first radar, an image acquisition device and a first inertial sensor at the bottom, and the container truck is equipped with a second radar and a second inertial sensor on the top; the air-ground collaborative long-term SLAM system is used to collect point cloud data, sensor data and image data of the container yard, and an initial map of the container yard is constructed based on the point cloud data and the image data; the plane features of the container yard are determined according to the size information of the containers in the container yard and the point cloud data; the target relative pose between the current frame and the nearest adjacent plane in the initial map is determined according to the sensor data and the plane features of the container yard, and a target map of the container yard is constructed according to the target relative pose.

[0008] In one embodiment, the use of the air-ground collaborative long-term SLAM system to collect point cloud data, sensor data and image data of the container yard includes: the drone in the air-ground collaborative long-term SLAM system collects point cloud data above the container yard through the first radar, and collects image data of the container yard through the image acquisition device; the container truck in the air-ground collaborative long-term SLAM system collects point cloud data of the vertical plane of the containers in the container yard through the second radar; the point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the containers; the first inertial sensor and the second inertial sensor in the air-ground collaborative long-term SLAM system collect sensor data; the sensor data includes angular velocity and acceleration.

[0009] In one embodiment, the planar features of the container yard are determined based on the size information of the containers in the container yard and the point cloud data, including: deleting the ground point cloud in the point cloud data; fitting the intensity information of the point cloud in the point cloud data using a preset algorithm with the size information of different types of containers and the direction of the plane normal vector as constraints to obtain the fitted point cloud data; segmenting the top container point cloud plane and the vertical container point cloud plane of each container position from the fitted point cloud data to obtain the target point cloud plane; determining the corner point coordinates of each container position in the container yard based on the target point cloud plane to obtain the planar features of the container yard.

[0010] In one embodiment, determining the target relative pose between the current frame and the nearest adjacent plane in the initial map based on the sensor data and the plane features of the container yard includes: determining the adjacent point cloud plane closest to the point cloud plane in the current frame from the initial map; determining the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map based on the sensor data; matching the point cloud plane in the current frame with the adjacent point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between the two.

[0011] In one embodiment, the point cloud plane in the current frame is matched with the adjacent point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between the two, including: determining the corner point coordinates and normal vector direction of the point cloud plane in the initial map corresponding to the point cloud plane in the current frame; determining the translation amount according to the corner point coordinates, a preset number of point cloud planes under the perspective of drones, and a preset number of point cloud planes under the perspective of trucks; determining the rotation amount according to the normal vector direction, a preset number of point cloud planes under the perspective of drones, and a preset number of point cloud planes under the perspective of trucks; determining the target relative pose according to the translation amount and the rotation amount.

[0012] In one embodiment, after obtaining the relative pose of the target between the two, the method further includes: determining an edge point cloud from the point cloud plane, performing point cloud registration on the edge point cloud in the current frame and the edge point cloud in the initial map to obtain the optimized relative pose of the target.

[0013] In one embodiment, after constructing the target map of the container yard according to the target relative posture, the method further includes: generating a depth image based on the point cloud data of continuous frames of the same viewpoint; comparing the depth image with the pixel depth at the same position in the initial map, determining the dynamic point cloud through point cloud occlusion and deleting it to obtain the target point cloud; and updating the map of the container yard according to the target point cloud and the historical map.

[0014] In one embodiment, updating the map of the container yard based on the target point cloud and the historical map includes: determining a current map based on the target point cloud; comparing the pixel depth of the current map and the same position in the historical map, retaining the dynamic point cloud of the current map relative to the historical map and the static point cloud in the historical map, and obtaining an updated map of the container yard.

[0015] In one embodiment, the method also includes: using the air-ground collaborative long-term SLAM system to collect the location descriptor of the container yard, and constructing the current topological relationship of the container yard through a checkerboard structure; the location descriptor includes characteristic data and location information of the container; determining the loop loss value based on the difference between the current topological relationship and the historical topological relationship of the container yard and a preset loop loss function; using the drone to detect the position of the container truck, and determining the position of the container truck as the initial position of the open space at both ends of the map for splicing the container yard; performing loop detection on the container yard according to the loop loss value, and optimizing the map of the container yard according to the loop loss function, the initial position of the open space, the preset checkerboard topological relationship, and the preset peripheral ring distribution topological relationship.

[0016] In the second aspect, an embodiment of the present application also provides a map construction device for a container yard, comprising: a hardware system construction module, configured to construct an air-ground collaborative long-term SLAM system including a container truck and a drone; the drone is equipped with a first radar, an image acquisition device and a first inertial sensor at the bottom, and the container truck is equipped with a second radar and a second inertial sensor on the top; a data acquisition module, configured to use the air-ground collaborative long-term SLAM system to collect point cloud data, sensor data and image data of the container yard, and construct an initial map of the container yard based on the point cloud data and the image data; a feature extraction module, configured to determine the plane features of the container yard according to the size information of the containers in the container yard and the point cloud data; a map construction module, configured to determine the target relative pose between the current frame and the nearest adjacent plane in the initial map according to the sensor data and the plane features of the container yard, and construct a target map of the container yard according to the target relative pose.

[0017] In one embodiment, the data acquisition module is configured as follows: the drone in the air-ground collaborative long-term SLAM system collects point cloud data above the container yard through the first radar, and collects image data of the container yard through the image acquisition device; the container truck in the air-ground collaborative long-term SLAM system collects point cloud data of the vertical plane of the containers in the container yard through the second radar; the point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the containers; the first inertial sensor and the second inertial sensor in the air-ground collaborative long-term SLAM system collect sensor data; the sensor data includes angular velocity and acceleration.

[0018] In one embodiment, the feature extraction module is configured to: delete the ground point cloud in the point cloud data; fit the intensity information of the point cloud in the point cloud data using a preset algorithm with the size information of different types of containers and the direction of the plane normal vector as constraints to obtain the fitted point cloud data; segment the top container point cloud plane and the vertical container point cloud plane of each container position from the fitted point cloud data to obtain the target point cloud plane; determine the corner point coordinates of each container position in the container yard based on the target point cloud plane to obtain the plane features of the container yard.

[0019] In one embodiment, the map construction module is configured to: determine from the initial map the adjacent point cloud plane closest to the point cloud plane in the current frame; determine the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map based on the sensor data; match the point cloud plane in the current frame with the adjacent point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between the two.

[0020] In one embodiment, the map construction module is configured to: determine the corner point coordinates and normal vector direction of the point cloud plane in the initial map corresponding to the point cloud plane in the current frame; determine the translation amount according to the corner point coordinates, the corner point coordinates of a preset number of point cloud planes under the perspective of drones, and the corner point coordinates of a preset number of point cloud planes under the perspective of trucks; determine the rotation amount according to the normal vector direction, the normal vector directions of the point cloud planes under the perspective of drones, and the normal vector directions of the point cloud planes under the perspective of trucks; determine the relative position of the target according to the translation amount and the rotation amount.

[0021] In one embodiment, the map construction device also includes a posture optimization module, which is configured to: determine the edge point cloud from the point cloud plane, perform point cloud registration on the edge point cloud in the current frame and the edge point cloud in the initial map, and obtain the optimized relative posture of the target.

[0022] In one embodiment, the map construction device also includes a map update module, which is configured to: generate a depth image based on the point cloud data of continuous frames of the same viewpoint; compare the depth image with the pixel depth of the same position in the initial map, determine the dynamic point cloud through point cloud occlusion and delete it, and obtain the target point cloud; update the map of the container yard based on the target point cloud and the historical map.

[0023] In one embodiment, the map update module is configured to: determine the current map based on the target point cloud; compare the pixel depth of the current map with the pixel depth of the same position in the historical map, retain the dynamic point cloud of the current map relative to the historical map and the static point cloud in the historical map, and obtain an updated map of the container yard.

[0024] In one embodiment, the map construction device also includes a map optimization module, which is configured to: use the air-ground collaborative long-term SLAM system to collect the location descriptor of the container yard, and construct the current topological relationship of the container yard through a chessboard structure; the location descriptor includes characteristic data and location information of the container; determine the loop loss value based on the difference between the current topological relationship and the historical topological relationship of the container yard and a preset loop loss function; use the drone to detect the position of the container truck, and determine the position of the container truck as the initial position of the open space at both ends of the map for splicing the container yard; perform loop detection on the container yard according to the loop loss value, and optimize the map of the container yard according to the loop loss function, the initial position of the open space, the preset chessboard topological relationship, and the preset peripheral ring distribution topological relationship.

[0025] On the third aspect, an embodiment of the present application also provides a visualization platform for a container yard, including: a processing module configured to match container numbers, container locations and location information; a visualization module configured to visualize the container yard through location descriptors according to the matching relationship between the container numbers, container locations and actual location information.

[0026] In a fourth aspect, an embodiment of the present application further provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the computer device is running, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to perform the steps of any one of the above methods.

[0027] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the above methods are executed.

[0028] The beneficial effects of the present application are as follows: by modifying the existing hardware system, an air-ground collaborative long-term SLAM system constructed by drones and container trucks is obtained, and an initial map of the container yard is constructed based on the point cloud data, sensor data and image data collected by the air-ground collaborative long-term SLAM system, and then plane features are extracted, and the relative posture is determined based on the plane features. According to the relative posture, a precise map (target map) of the container yard is generated, and the constructed map of the container yard is more accurate, thereby improving the work efficiency of container yard management. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of a preset chessboard topological relationship in a method for constructing a container yard map provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of a preset peripheral ring distribution topological relationship in a method for constructing a container yard map provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of a flow chart of a method for constructing a container yard map provided in an embodiment of the present application;

[0041] Fig.12 A schematic diagram of the structure of a container yard map construction device provided in an embodiment of the present application;

[0042] Fig.13 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0044] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0045] In the description of the present application, it should be noted that if the terms "upper", "lower", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0046] In addition, the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0048] Figure 1 A flowchart of a method for constructing a container yard map provided in an embodiment of the present application; Figure 1 As shown, the method includes:

[0049] Step 110: construct an air-ground collaborative long-term SLAM system including a container truck and a drone.

[0050] Among them, the bottom of the drone is equipped with a first radar, an image acquisition device and a first inertial sensor, and the top of the container truck is equipped with a second radar and a second inertial sensor.

[0051] In actual operation, there can be one or more drones and container trucks, and there is no limitation here.

[0052] The first radar carried by the drone can be a solid-state laser radar; the second radar carried by the container truck can be a mechanical laser radar.

[0053] The image acquisition device may be any device capable of acquiring images, such as a visible light camera, a CCD camera, etc., which is not limited here.

[0054] Inertial sensor, or IMU (Inertial Measurement Unit), is a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. IMU is usually a fully functional device that consists of two or more inertial measurement MEMS chips and ASIC chips sealed together. Depending on the built-in sensors (three-axis magnetic sensor, three-axis accelerometer and three-axis gyroscope), it is divided into six-axis and nine-axis IMUs, which can meet the needs of high-precision measurement in different application scenarios.

[0055] The air-ground collaborative long-term SLAM system in the container yard map construction method provided in the embodiment of the present application is obtained by modifying the existing hardware system. Specifically, the following modifications are made:

[0056] (1) Deployment of drones (groups) and container trucks (groups): One or more drones are deployed in the container yard to patrol the container yard in different areas. The bottom of the drone is equipped with a first radar, an image acquisition device and a first IMU; the top of the container truck is equipped with a second radar and a second IMU. The drones (groups) and the container trucks (groups) constitute the above-mentioned air-ground collaborative long-term SLAM system.

[0057] (2) Unified multi-sensor coordinate system: Point cloud data and image data include six coordinate systems, namely, the world coordinate system, the body coordinate system of the container truck, the body coordinate system of the drone, the solid-state laser radar coordinate system, the mechanical laser radar coordinate system radar coordinate system, and the image acquisition device coordinate system. In the map construction method of the container yard provided in the embodiment of the present application, the world coordinate system is used as the global coordinate system. Specifically, the reference coordinate system of the drone departure area can be used as the world coordinate system, and the origin can be the center point of the drone departure area;

[0058] (3) UAVs and container trucks work together to achieve air-ground collaborative mapping: UAVs have a wide field of view and can map a large area of ​​the scene above the yard. At the same time, they can collect the top features of the container. However, although the UAV's perspective includes most of the scene features, the containers in the middle of the higher layers may block the containers near the lane line. During the operation of the ground vehicle, it may also encounter the situation of lifting containers in the air. Therefore, in order to improve the quality of the scanned data, a ground scanning system (i.e., container trucks (groups)) is introduced. At the same time, in order to avoid the introduction of additional ground robots to affect the existing working scene, the existing operating container trucks are modified to build a close-up and detailed map of each sub-area during the operation process, supplement the UAV's blind spot perspective, and achieve higher mapping efficiency. In actual operation, when the UAV is inspecting above the yard, the upper management system starts the container trucks in the area where the UAV is located. The UAV and the container trucks can collect data and build maps. Among them, the map built by the container truck can supplement the map built by the UAV.

[0059] Step 120: Use the air-ground collaborative long-term SLAM system to collect point cloud data, sensor data and image data of the container yard, and build an initial map of the container yard based on the point cloud data and image data.

[0060] The sensing data includes the above three-axis attitude angle (or angular rate) and acceleration.

[0061] In actual operation, when the drone patrols above the yard, (the upper management system or upper control system) starts the truck in the area where the drone is located, and then the drone and the truck can collect data; specifically, Figure 2 As shown, the above step 120 may further include the following steps 210 to 230:

[0062] Step 210: The UAV in the air-ground collaborative long-term SLAM system collects point cloud data above the container yard through the first radar, and collects image data of the container yard through the image acquisition device.

[0063] Point cloud data refers to a set of vectors in a three-dimensional coordinate system. Scanning data is recorded in the form of points, each of which contains three-dimensional coordinates, and some may contain color information (RGB) or reflection intensity information (Intensity).

[0064] For example, one drone is responsible for scanning the entire yard, and the flight route is set according to the row and column arrangement of the containers in the yard. The yard can be divided into different sub-areas according to the lane lines of the loading and unloading area. The flight route of the drone can be above the central symmetry line of each sub-area, and the main scanning target is the feature points above the containers in the yard. In actual operation, the mapping efficiency can reach 90m 2 / s.

[0065] The image acquisition device may be any device capable of acquiring images, such as a visible light camera, a CCD camera, etc., which is not limited here.

[0066] Step 220: The container truck in the air-ground collaborative long-term SLAM system collects point cloud data of the vertical plane of the containers in the container yard through the second radar.

[0067] The point cloud data of the container yard includes point cloud data above the container yard and point cloud data on the vertical plane of the container.

[0068] The container truck can build a map of the containers on both sides of the passage in the yard based on the point cloud data of the vertical plane of the containers it collects, so as to supplement the initial map built by the drone.

[0069] It should be noted that both drones and container trucks have a mapping function. After collecting image data, sensor data and / or point cloud data, an initial map of the container yard can be constructed based on the image data, sensor data and / or point cloud data. In actual operation, the mapping function can be started at the same time as the data collection function of the drone and container truck is started.

[0070] Step 230: The first inertial sensor and the second inertial sensor in the air-ground collaborative long-term SLAM system collect sensor data.

[0071] Among them, the sensing data includes angular rate and acceleration.

[0072] IMU is the core device of inertial positioning technology. After error compensation and inertial navigation solution, it finally outputs navigation information such as the coordinate change and speed of the carrier relative to the initial position.

[0073] Step 130: Determine the plane features of the container yard based on the size information of the containers in the container yard and the point cloud data.

[0074] The size information of the container includes the length, width and height ratio of the container. In actual operation, the size information of the container can be recorded in a database in advance.

[0075] This step mainly uses the strength information of the point cloud in the point cloud data. The reason for introducing the strength information of the point cloud is that during the scanning process, different angles, distances, materials and other factors will affect the strength of the point cloud. Introducing strength information can more efficiently segment the container plane.

[0076] This step can be achieved through the following process: first, the ground point cloud in the point cloud data is deleted; second, the size information of the container, the intensity information of the point cloud, and the plane normal vector are used as constraints, and a preset algorithm (such as the RANSAC algorithm) is applied to segment the plane in the current frame, and the top container point cloud plane and the side container point cloud plane of each container position are screened out as valid planes from the perspective of the drone and the container truck; finally, the corner point coordinates of each container position in the yard are determined through these valid planes, and the plane features of the container yard are obtained.

[0077] Step 140: Determine the target relative pose between the current frame and the nearest adjacent plane in the initial map based on the sensor data and the plane features of the container yard, and construct a target map of the container yard based on the target relative pose.

[0078] Among them, this step can be achieved through the following process: first, find the nearest adjacent faces in the current frame and the initial map; second, use the corner points and plane normal vector constraints of the plane to match the extracted plane features in the current frame with the plane features of the nearest adjacent faces in the initial map to obtain the relative pose between the two planes; here, the relative pose can be understood as a plane-to-plane constraint; finally, construct a target map of the container yard based on the target relative pose.

[0079] The target map of the container yard is more accurate than the initial map of the container yard.

[0080] The map construction method of the container yard provided in the embodiment of the present application, first, constructs an air-ground collaborative long-term SLAM system including a container truck and an unmanned aerial vehicle; secondly, adopts the air-ground collaborative long-term SLAM system to collect point cloud data, sensor data and image data of the container yard, and constructs the initial map of the container yard based on the point cloud data and image data; thirdly, according to the size information of the container in the container yard and the point cloud data, the plane features of the container yard are determined; finally, according to the sensor data and the plane features of the container yard, the target relative pose between the current frame and the nearest adjacent plane in the initial map is determined, and the target map of the container yard is constructed according to the target relative pose. In this way, by transforming the existing hardware system, an air-ground collaborative long-term SLAM system constructed by the unmanned aerial vehicle and the container truck is obtained, and the initial map of the container yard is constructed based on the point cloud data, sensor data and image data collected by the air-ground collaborative long-term SLAM system, and then the plane features are extracted, the relative pose is determined based on the plane features, and the accurate map (target map) of the container yard is generated according to the relative pose, and the constructed map of the container yard is more accurate, thereby improving the work efficiency of the container yard management.

[0081] The following is a further explanation of the process of determining the plane features of the container yard according to the size information of the containers in the container yard and the point cloud data in the above embodiment with reference to examples. Figure 3 The following is a flow chart of a method for constructing a container yard map provided in an embodiment of the present application. Figure 3 As shown, the above step 130 may further include the following steps 310 to 340:

[0082] Step 310: Delete the ground point cloud in the point cloud data.

[0083] Among them, due to the high flatness of the port ground, the ground is the most important complete plane in the point cloud data, and the RANSAC algorithm can be used to segment the main plane in the point cloud data (that is, the ground point cloud).

[0084] Step 320: Using the size information of different types of containers and the direction of the plane normal vector as constraints, a preset algorithm is used to fit the intensity information of the point cloud in the point cloud data to obtain fitted point cloud data.

[0085] The size information of the container may be a priori size information of the container, such as the length, width, height, length, width, height ratio, etc. of the container, which is not limited here.

[0086] The preset algorithm may be a parameter estimation algorithm applied in the field of computer vision, such as a RANSAC (Random Sample Consensus) algorithm, a clustering algorithm, a least squares method, etc., which is not limited here.

[0087] Exemplarily, the process of fitting using the RANSAC algorithm is as follows: first, randomly select a small number of samples in the data and fit the model based on these samples; second, calculate the distance from other data points to the model, and classify data points with a distance less than a certain threshold as inliers, and data points with a distance greater than the threshold as outliers; repeat this process multiple times, and select the model with the most inliers as the final estimation result.

[0088] Step 330: Segment the top container point cloud plane and the vertical container point cloud plane of each container location from the fitted point cloud data to obtain the target point cloud plane.

[0089] Among them, this step can segment the point cloud plane {F1,......,F i}, and then select the highest level container point cloud plane of each container from the point cloud plane in the current frame {F u1 ,......,F um} and vertical container point cloud plane {F s1 ,......,F sw}As the effective plane (target plane) from the perspective of the drone and the truck.

[0090] During the scanning process of drones and trucks, the reflectivity of planes in different directions and orientations is different. Therefore, at the connection points of different planes of the container, the reflection intensity of the point cloud will have a large gradient change. Therefore, using this gradient change information can better distinguish the edges of different planes of the container. In this way, the efficiency of plane segmentation can be improved.

[0091] Step 340: Determine the corner point coordinates of each container location in the container yard according to the target point cloud plane, and obtain the plane features of the container yard.

[0092] Among them, this step can directly determine the corner point coordinates of each container location from the target point cloud plane; specifically, the coordinates of the center point cloud of the point cloud where the corner point is located can be determined as the corner point coordinates, and then, the corner point coordinates of each container location can be directly determined as the plane features of the container yard.

[0093] The map construction method of the container yard provided in the embodiment of the present application first deletes the ground point cloud in the point cloud data; secondly, the size information of different types of containers and the direction of the plane normal vector are constrained, and the intensity information of the point cloud is fitted using a preset algorithm to obtain the fitted point cloud data; thirdly, the highest level container point cloud plane and the vertical container point cloud plane of each container position are segmented from the fitted point cloud data to obtain the target point cloud plane; finally, according to the target point cloud plane, the coordinates of the corner points of each container position in the container yard are determined to obtain the plane features of the container yard. In this way, after introducing the intensity information of the point cloud, the point cloud plane can be better segmented, and then, the plane features of the container yard can be accurately obtained.

[0094] The following is an example explanation of the process of determining the relative pose of the target between the current frame and the nearest adjacent plane in the initial map according to the sensor data and the plane features of the container yard in the above embodiment. Figure 4 The following is a flow chart of a method for constructing a container yard map provided in an embodiment of the present application. Figure 4 As shown, the above step 140 may further include the following steps 410 to 430:

[0095] Step 410: Determine from the initial map the adjacent point cloud plane closest to the point cloud plane in the current frame.

[0096] Among them, this step can be achieved by the following process: step (1), based on the right-hand coordinate system, with the traveling direction of the drone and the truck as the x-axis, randomly select a preset number of point cloud planes outside the row with the smallest x coordinate in the xy plane within the field of view from the perspective of the drone; similarly, randomly select a preset number of point cloud planes outside the row with the smallest x coordinate in the xz plane from the perspective of the truck; step (2), calculate the coordinates of the center points of the preset number of point cloud planes {A i (x i ,y i ,z i )|1≤i≤n}, corner point coordinates (usually 4 corner points) {B ij (x ij ,y ij ,z ij )|1≤i≤n,1≤j≤4}, normal vector direction Step (3): Taking the center point of each point cloud plane as the reference, find the nearest adjacent point C corresponding to it in the initial map. i (x i ,y i ,z i), the adjacent point is regarded as the inner point of the corresponding surface in the initial map; step (4), according to the inner point, the corresponding point cloud plane in the initial map is determined. Here, step (1) can ensure the integrity of the point cloud plane in the before and after comparison.

[0097] Step 420: Determine the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map based on the sensor data.

[0098] In this step, the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map can be directly determined through the motion model of the inertial sensor; here, the initial relative pose can also be understood as a preliminary relative pose.

[0099] Step 430: Match the point cloud plane in the current frame with the adjacent point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between the two.

[0100] The relative position of the target can be obtained through the corner coordinates and normal vector direction of the point cloud plane. Specifically, Figure 5 The following is a flow chart of a method for constructing a container yard map provided in an embodiment of the present application. Figure 5 As shown, the above step 430 may further include the following steps 510 to 540:

[0101] Step 510: Determine the corner point coordinates and normal vector direction of the point cloud plane in the initial map corresponding to the point cloud plane in the current frame.

[0102] Exemplarily, the coordinates of the corner points of the point cloud plane corresponding to the inner points in the initial map are calculated. ij (x ij ,y ij ,z ij )|1≤i≤n,1≤j≤4}, normal vector direction

[0103] Step 520: Determine the translation amount according to the coordinates of the corner points, the coordinates of the corner points of the point cloud planes from the perspectives of a preset number of drones, and the coordinates of the corner points of the point cloud planes from the perspectives of a preset number of container trucks.

[0104] Among them, the preset number of point cloud planes from the perspective of the drone and the preset number of point cloud planes from the perspective of the truck are the point cloud planes from the perspective of the drone and the point cloud planes from the perspective of the truck selected in step (1) in the above step 410.

[0105] Exemplarily, the translation amount is determined based on the corner point coordinates calculated in step (2) in step 410 and step 510 .

[0106] For example, the translation amount can be calculated using a mean value calculation method, and the calculation formula is shown in the following formula (1):

[0107]

[0108] Step 530: Determine the rotation amount according to the normal vector direction, the normal vector directions of a preset number of point cloud planes from the perspective of the drone, and the normal vector directions of a preset number of point cloud planes from the perspective of the truck.

[0109] Exemplarily, the amount of rotation is determined based on the direction of the normal vector calculated in step (2) in step 410 and step 510 .

[0110] Exemplarily, the rotation amount can be calculated using a mean value calculation method, and the calculation formula is shown in the following formula (2):

[0111]

[0112] Step 540: Determine the relative position and orientation of the target according to the translation and rotation amounts.

[0113] Among them, the translation and rotation constitute the target relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map.

[0114] The method for constructing a map of a container yard provided in an embodiment of the present application first determines the adjacent point cloud plane closest to the point cloud plane in the current frame from the initial map; secondly, determines the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map based on the sensor data; and thirdly, matches the point cloud plane in the current frame with the adjacent point cloud plane in the initial map to obtain the target relative pose between the two. In this way, according to the plane features of the container yard, the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map is determined, and then a more accurate target relative pose can be obtained through plane matching (point cloud registration), and a more accurate container yard map can be generated based on the target relative pose.

[0115] The container yard map construction method provided in the embodiment of the present application can further optimize the relative position of the target to obtain a more accurate relative position of the target. The container yard map construction method provided in the embodiment of the application can also include the following steps:

[0116] The edge point cloud is determined from the point cloud plane, and the edge point cloud in the current frame is aligned with the edge point cloud in the initial map to obtain the optimized relative position and pose of the target.

[0117] The point cloud registration may be PCA registration, ICP registration, etc., which is not limited here.

[0118] In this step, the edge point cloud (frame point cloud) is determined from the point cloud plane, that is, the plane point cloud {F1,......,F i} is deleted. In this way, by reducing the point cloud, the amount of calculation for point cloud registration can be reduced. In addition, by performing point cloud registration, a more accurate relative pose can be obtained without affecting real-time performance. In other words, the target relative pose in this step is more accurate than the target relative pose in step 430.

[0119] The container yard map construction method provided in the embodiment of the present application can also update the container yard map in real time. Figure 6 The following is a flow chart of a method for constructing a container yard map provided in an embodiment of the present application. Figure 6 As shown, the container yard map construction method provided in the embodiment of the present application may also include the following steps 610 to 630:

[0120] Step 610: Generate a depth image based on point cloud data of consecutive frames from the same viewpoint.

[0121] This step can generate a depth image based on the projection of point cloud data of consecutive frames of the same viewpoint; specifically, this step can be implemented by the following process: first, the point cloud is divided to obtain n groups of point clouds, forming batch data {S1,...,S n}, each point cloud S n It contains the point cloud of the yard during one cycle of radar scanning; secondly, a frame of point cloud S is extracted from each batch of data through multiple random traversals. i As control frames, other frames {S1,......,S i-1 ,S i+1 ,......,S n} Used to construct the local map of the yard My local (i.e. the initial map); finally, the local map My local To the reference frame S i The point cloud coordinate system is converted, and the depth difference between the highest point and the lowest point at each position is the pixel depth, and the projection is a high-resolution r with a fixed window size. h The depth image of .

[0122] In actual operation, before this step, the container yard point cloud can be divided according to the identification information in the image data; here, the identification information can be the lane identification of the loading and unloading area. Specifically, this step can use the target detection algorithm to detect the coordinates of the lane identification of the loading and unloading area, and then correspond the lane identification coordinates to the point cloud data, and then use the lane identification coordinates on both sides of the yard as the vertices of the rectangular area of ​​the container yard, and map the yard point cloud Sy local From all point clouds S localIn addition, a height threshold can be set to delete the point cloud that exceeds the height threshold, so that the impact of the loaders and unloaders in the container yard on the algorithms involved in the map update process can be avoided.

[0123] Step 620: Compare the depth image with the pixel depth at the same position in the initial map, determine the dynamic point cloud through point cloud occlusion and delete it, and obtain the target point cloud.

[0124] This step can be specifically implemented by the following process: First, compare the depth image Iy generated by the reference frame i With the local map depth image Iy local The pixel depth; secondly, when the local map depth value dy local Greater than the reference frame map depth value dy i When determining the local map My corresponding to the yard local There are dynamic occluders in the d1 ,......,P dm} is deleted as a dynamic point cloud to obtain the target point cloud (i.e. static local map) Mys local .

[0125] In actual operation, static point clouds may be mistakenly deleted in the process of deleting dynamic point clouds through high-resolution depth maps. Therefore, SLAM pose errors and point cloud distortion will affect the depth map. Therefore, an algorithm for recovering mistakenly deleted point clouds based on multi-resolution depth maps can be designed. Specifically, the static point clouds {P ds1 ,......,P dsw}Re-label as static point cloud{P s1 ,......,P sw} to reduce the error rate of the algorithm.

[0126] Step 630: Update the map of the container yard according to the target point cloud and the historical map.

[0127] This step can be achieved through the following process: First, the static local map Mys local Splice and get the static global map Mys glocal ; Secondly, compare the static global map Mys glocal The pixel depth at the same location as the historical map of the yard (historical global map); again, keep the static global map Mys glocal The dynamic point cloud relative to the historical map and the static point cloud of the two can realize the map update of the container yard.

[0128] The container yard map construction method provided in the embodiment of the present application first generates a depth image based on the point cloud data of continuous frames of the same viewpoint; secondly, the depth image is compared with the pixel depth of the same position in the initial map, and the dynamic point cloud is determined and deleted through point cloud occlusion to obtain the target point cloud; thirdly, the map of the container yard is updated based on the target point cloud and the historical map. In this way, the dynamic point cloud can be deleted in time to avoid false detection, and then a more accurate target point cloud can be obtained, and the updated map is more accurate.

[0129] The container yard map construction method provided in the embodiment of the present application can also improve the efficiency of container yard map construction. Figure 7 The following is a flow chart of a method for constructing a container yard map provided in an embodiment of the present application. Figure 7 As shown, the container yard map construction method provided in the embodiment of the present application may also include the following steps 710 to 740:

[0130] Step 710: Use the air-ground collaborative long-term SLAM system to collect location descriptors of the container yard, and construct the current topological relationship of the container yard through a chessboard structure.

[0131] The location descriptor includes the characteristic data and location information of the container. The characteristic data of the container may be the length-width-height ratio of the container, the reflectivity of the container paint layer, the stacking height of the container, etc. The location information of the container may be the center coordinates and vertex coordinates of the corresponding position, etc.

[0132] The location descriptor may be a three-dimensional matrix, which may record the characteristic data and location information of the container.

[0133] The location descriptor of the container yard is determined by the location descriptor of each container slot.

[0134] Among them, this step can be achieved through the following process: UAV part: The UAV gradually scans around with the container position directly below the current position as the center, records the stacking height of the container at each position and the length, width and height ratio of the container, and obtains the position descriptor (topological relationship) from the perspective of the UAV. Truck vehicle part: Introducing the reflectivity of the container paint layer as a constraint, the truck vehicle takes the containers on the left and right sides of the current position as the two centers, and gradually records the stacking height, length, width and height ratio of the container, and reflectivity of the container paint layer at the corresponding positions along the periphery of the container yard to the left and right, and obtains the position descriptor (topological relationship) from the perspective of the truck vehicle; the topological relationship from the perspective of the above-mentioned UAV and the topological relationship from the perspective of the truck vehicle constitute the current topological relationship of the above-mentioned container yard.

[0135] Step 720: Determine a loop loss value according to the difference between the current topological relationship and the historical topological relationship of the container yard and a preset loop loss function.

[0136] This step can be achieved through the following process: assigning the container length-width-height ratio a:b:c, the container paint layer reflectivity k (which can be assigned a larger weight), and the container stacking height h (which can be assigned a smaller weight) in the topological relationship, and calculating the loop loss value according to the preset loop loss function calculation formula.

[0137] Step 730: Use a drone to detect the position of the container truck, and determine the position of the container truck as the initial position of the open space at both ends of the map for splicing the container yard.

[0138] This step can be specifically implemented through the following process: the position of the container truck is detected by the image acquisition device at the bottom of the drone, and the position of the container truck is determined as the initial position for map registration at both ends of the open space.

[0139] Step 740: perform loop detection on the container yard according to the loop loss value, and optimize the map of the container yard according to the loop loss function, the initial position of the open space, the preset chessboard topological relationship, and the preset peripheral ring distribution topological relationship.

[0140] Among them, the preset chessboard topological relationship can be as follows Figure 8 As shown; the preset peripheral ring distribution topology relationship can be as follows Fig. 9 The preset chessboard topological relationship and the preset peripheral ring distribution topological relationship are used to improve the current topological relationship of the container yard.

[0141] The loop detection process can be performed according to the process of collecting position descriptors by the air-to-ground collaborative long-term SLAM system in the above step 710. After each position descriptor is collected, the current topological relationship is compared with the historical topological relationship to obtain a loop loss value; then, based on the loop loss value, it is determined whether to perform the next loop detection until the final loop loss value meets the preset convergence condition.

[0142] For example, the loop closure loss function can be Among them, the length, width and height ratio of the current container position is 1:l1:l2, and the length, width and height ratio of the container position compared with the current container position is 1:l1′:l2′; the reflectivity of the current container paint layer is k, and the reflectivity of the container paint layer compared with the current container paint layer is k′.

[0143] The initial location of the empty lot is used to complete the map of the container yard.

[0144] The map construction method of the container yard provided in the embodiment of the present application is as follows: first, the location descriptor of the container yard is collected by using the air-ground collaborative long-term SLAM system, and the current topological relationship of the container yard is constructed through a chessboard structure; secondly, the loop loss value is determined according to the difference between the current topological relationship and the historical topological relationship of the container yard and the preset loop loss function; thirdly, the position of the container truck is detected by using a drone, and the position of the container truck is determined as the initial position of the open space at both ends of the map for splicing the container yard; finally, the container yard is looped according to the loop loss value, and the map of the container yard is optimized according to the loop loss function, the initial position of the open space, the preset chessboard topological relationship, and the preset peripheral ring distribution topological relationship. In this way, the map construction efficiency can be improved through loop detection and the topological relationship of each box position, and the map can be improved through the preset chessboard topological relationship, the preset peripheral ring distribution topological relationship and the initial position of the open space.

[0145] After introducing the map construction method of the container yard according to the exemplary embodiment of the present disclosure, next, refer to Fig.10 A visualization platform 1000 for a container yard according to an exemplary embodiment of the present disclosure will be described.

[0146] refer to Fig.10 The visualization platform 1000 of the container yard includes: a processing module 1010, which is configured to match the container number, the container position and the actual location information; and a visualization module 1020, which is configured to visualize the container yard according to the matching relationship between the container number, the container position and the actual location information through the location descriptor.

[0147] In actual operation, the visualization platform of the container yard can specifically include the following functions:

[0148] (1) Container hazard assessment based on convolution method: Extract the height layer matrix in the location descriptor and perform zero padding on the periphery. b,c∈{k∈R|0<k<1}, respectively perform convolution operation on the height layer matrix after filling with 0, and take the absolute value of the calculation result. Among them, the parameter (bc) can be used to set the influence weight of the number of stacking layers of the container in front, behind, left and right positions on the degree of danger.

[0149] (2) Container movement feedback system based on the existing management system: After the upper-level management system (upper-level control system) assigns the location of the newly arrived containers, the SLAM system can scan the yard and detect whether the containers are correctly installed at the corresponding locations; at the same time, the container number, source, cargo type and other information can be bound to the actual location to improve the entire yard management system and form a data closed loop.

[0150] (3) Simulation and visualization system based on location descriptors: Based on location descriptors, a simulation algorithm is used to generate a rectangular frame appearance at the corresponding position of the container to replace the point cloud for visualization, and a semantic map of the container distribution in the yard is constructed; at the same time, it is displayed in real time on the upper-level monitoring interface.

[0151] The visualization platform of the container yard can not only optimize the efficiency of information transmission, but also provide a more intuitive display screen; in addition, it can also integrate information such as container number, source, and cargo type for a complete display in the visualization interface.

[0152] Another implementation of a method for constructing a container yard map provided in an embodiment of the present application is as follows: Fig.11 As shown, the following steps are included:

[0153] Step (1) constructing an air-ground collaborative system integrating multiple sensors to collect point cloud data and image data of the yard: the UAV uses a solid-state radar for vertical ground scanning to quickly collect the overall point cloud data above the yard, and at the same time uses a visible light camera for vertical ground scanning to collect real-time image data of the yard; when the UAV patrols above the yard, the upper management system activates the mapping function of the container truck in the area where the UAV is located, and cooperates with the yard UAV to collect vertical plane information of the containers in the yard through the mechanical radar installed on the roof;

[0154] Step (2) Design an air-ground collaborative map update algorithm based on the complementary difference of inter-frame depth map mapping to solve the problem of yard change detection in large port scenes: use the visible light camera carried by the drone to detect the lane lines of the yard loading and unloading area and the container trucks in the area, and divide the yard area; project the scanning point cloud data before and after the continuous frames of the same viewpoint to generate a depth map and compare the pixel depth at the same position. By judging the point cloud occlusion, the dynamic point cloud can be distinguished and deleted, and the global map can be updated by covering the corresponding position of the historical map with the current static map;

[0155] Step (3) Design a container segmentation algorithm that integrates prior size information to quickly detect the characteristics of the current frame container: Using the aspect ratio and normal vector direction of containers of different models as constraints, the RANSAC algorithm is used to segment the upper plane of the top container of each container position in the current frame point cloud based on the point cloud intensity information. The coordinates of the four corner points of each container position in the yard can be obtained through the top plane;

[0156] Step (4) In the special environment of the container yard, since the plane occupies most of the environmental features, the accurate posture relationship can be quickly obtained through surface feature matching; the yard SLAM front-end odometer based on the container surface features is designed to optimize the initial posture estimation: the drone and the container truck randomly select a preset number of n planes in the highest plane and the side plane of the yard, and extract the four corner points of each plane; by aligning the current frame plane with the four corner point positions of the nearest adjacent surface in the corresponding local map, a more accurate relative posture can be constrained; finally, by performing ICP alignment on the skeleton map after deleting the container plane points and ground points, an accurate relative posture is obtained;

[0157] Step (5) Design an air-ground collaborative loop detection method based on the topological relationship of adjacent slots to improve the efficiency of map construction: Use the global topological relationship of the yard as the global coordinate descriptor of each slot, and record the center coordinates, eight vertex coordinates, length, width and height ratio, container paint reflectivity and stacking height of each slot through a three-dimensional matrix; Convert the topological structure diagrams corresponding to the drone and the container truck from the top and side perspectives, calculate the loss value and detect the loop through the difference in topological relationship; This step not only has extremely high computational efficiency, but can also be updated during the map construction process, and can also constrain the map from the perspectives of drones and unmanned vehicles. At the same time, use drones to detect the relative position of container trucks in the yard, and provide the initial position of the container truck map when the maps at both ends of the open space are spliced;

[0158] Step (6) Design a visualization management platform based on the results of multi-machine collaborative SLAM to integrate the existing intelligent digital management system: use the altitude layer descriptor to calculate the danger level of each location and match the container position with the real coordinates and the container information in the digital management system, visualize the yard containers in the form of three-dimensional frames on the map, and construct a semantic map of the yard cargo distribution; integrate the semantic map with the information and visualize it on the upper platform.

[0159] After introducing the map construction method of the container yard according to the exemplary embodiment of the present disclosure, next, refer to Fig.12 A map constructing device 1200 for a container yard according to an exemplary embodiment of the present disclosure will be described.

[0160] refer to Fig.12The map construction device 1200 of the container yard includes: a hardware system construction module 1210, which is configured to construct an air-ground collaborative long-term SLAM system including a container truck and a drone; the drone is equipped with a first radar, an image acquisition device and a first inertial sensor at the bottom, and the container truck is equipped with a second radar and a second inertial sensor on the top; a data acquisition module 1220, which is configured to use the air-ground collaborative long-term SLAM system to collect point cloud data, sensor data and image data of the container yard, and construct an initial map of the container yard based on the point cloud data and the image data; a feature extraction module 1230, which is configured to determine the plane features of the container yard according to the size information of the containers in the container yard and the point cloud data; a map construction module 1240, which is configured to determine the target relative pose between the current frame and the nearest adjacent plane in the initial map according to the sensor data and the plane features of the container yard, and construct a target map of the container yard according to the target relative pose.

[0161] In one embodiment, the data acquisition module 1220 is configured as follows: the drone in the air-ground collaborative long-term SLAM system collects point cloud data above the container yard through the first radar, and collects image data of the container yard through the image acquisition device; the container truck in the air-ground collaborative long-term SLAM system collects point cloud data of the vertical plane of the containers in the container yard through the second radar; the point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the containers; the first inertial sensor and the second inertial sensor in the air-ground collaborative long-term SLAM system collect sensor data; the sensor data includes angular velocity and acceleration.

[0162] In one embodiment, the feature extraction module 1230 is configured to: delete the ground point cloud in the point cloud data; use a preset algorithm to fit the intensity information of the point cloud in the point cloud data with the size information of different types of containers and the direction of the plane normal vector as constraints to obtain the fitted point cloud data; segment the top container point cloud plane and the vertical container point cloud plane of each container position from the fitted point cloud data to obtain the target point cloud plane; determine the corner point coordinates of each container position in the container yard based on the target point cloud plane to obtain the plane features of the container yard.

[0163] In one embodiment, the map construction module 1240 is configured to: determine from the initial map the adjacent point cloud plane closest to the point cloud plane in the current frame; determine the initial relative pose between the point cloud plane in the current frame and the adjacent point cloud plane in the initial map based on the sensor data; match the point cloud plane in the current frame with the adjacent point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between the two.

[0164] In one embodiment, the map construction module 1240 is configured to: determine the corner point coordinates and normal vector direction of the point cloud plane in the initial map corresponding to the point cloud plane in the current frame; determine the translation amount according to the corner point coordinates, the corner point coordinates of a preset number of point cloud planes under the perspective of drones, and the corner point coordinates of a preset number of point cloud planes under the perspective of trucks; determine the rotation amount according to the normal vector direction, the normal vector directions of the point cloud planes under the perspective of drones, and the normal vector directions of the point cloud planes under the perspective of trucks; determine the relative position of the target according to the translation amount and the rotation amount.

[0165] In one embodiment, the map construction device 1200 also includes a posture optimization module, which is configured to: determine the edge point cloud from the point cloud plane, perform point cloud registration on the edge point cloud in the current frame and the edge point cloud in the initial map, and obtain an optimized target relative posture.

[0166] In one embodiment, the map construction device 1200 also includes a map update module, which is configured to: generate a depth image based on point cloud data of consecutive frames of the same viewpoint; compare the depth image with the pixel depth of the same position in the initial map, determine the dynamic point cloud through point cloud occlusion and delete it, and obtain the target point cloud; update the map of the container yard based on the target point cloud and the historical map.

[0167] In one embodiment, the map update module is configured to: determine the current map based on the target point cloud; compare the pixel depth of the current map with the pixel depth of the same position in the historical map, retain the dynamic point cloud of the current map relative to the historical map and the static point cloud in the historical map, and obtain an updated map of the container yard.

[0168] In one embodiment, the map construction device 1200 also includes a map optimization module, which is configured to: use an air-ground collaborative long-term SLAM system to collect location descriptors of the container yard, and construct the current topological relationship of the container yard through a checkerboard structure; the location descriptor includes characteristic data and location information of the container; determine the loop loss value based on the difference between the current topological relationship and the historical topological relationship of the container yard and a preset loop loss function; use a drone to detect the position of the container truck, and determine the position of the container truck as the initial position of the open space at both ends of the map for splicing the container yard; perform loop detection on the container yard according to the loop loss value, and optimize the map of the container yard according to the loop loss function, the initial position of the open space, the preset checkerboard topological relationship, and the preset peripheral ring distribution topological relationship.

[0169] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0170] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0171] Fig.13 A schematic diagram of a computer device provided in an embodiment of the present application. The device can be integrated into a terminal device or a chip of a terminal device. The terminal can be a computing device with data processing capabilities.

[0172] The device includes: a processor 1301 , a storage medium 1302 and a bus 1303 .

[0173] The storage medium 1302 stores program instructions executable by the processor 1301. When the computer device 1300 is running, the processor 1301 communicates with the storage medium 1302 via the bus 1303. The processor 1301 executes the program instructions to perform the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0174] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, comprising a program, which is used to execute the above method embodiment when executed by a processor.

[0175] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0178] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0179] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for constructing a container yard map, characterized in that: include: Constructing an air-ground collaborative long-term SLAM system including a container truck and an unmanned aerial vehicle; the bottom of the unmanned aerial vehicle is equipped with a first radar, an image acquisition device and a first inertial sensor, and the top of the container truck is equipped with a second radar and a second inertial sensor; The air-ground collaborative long-term SLAM system is used to collect point cloud data, sensor data and image data of the container yard, and an initial map of the container yard is constructed based on the point cloud data and the image data; Determining the plane features of the container yard according to the size information of the containers in the container yard and the point cloud data; According to the sensor data and the plane features of the container yard, a target relative pose between a current frame and the nearest adjacent plane in the initial map is determined, and a target map of the container yard is constructed according to the target relative pose.

2. The method according to claim 1, characterized in that The method of using the air-ground collaborative long-term SLAM system to collect point cloud data, sensor data and image data of the container yard includes: The drone in the air-ground collaborative long-term SLAM system collects point cloud data above the container yard through the first radar, and collects image data of the container yard through the image acquisition device; The container truck in the air-ground collaborative long-term SLAM system collects point cloud data of the vertical plane of the container in the container yard through the first radar; the point cloud data of the container yard includes point cloud data above the container yard and point cloud data of the vertical plane of the container; The first inertial sensor and the second inertial sensor in the air-ground collaborative long-term SLAM system collect sensing data; the sensing data includes angular rate and acceleration.

3. The method according to claim 1, characterized in that: The determining of the plane features of the container yard according to the size information of the containers in the container yard and the point cloud data comprises: Deleting the ground point cloud in the point cloud data; Using the size information of different types of containers and the direction of the plane normal vector as constraints, a preset algorithm is used to fit the intensity information of the point cloud in the point cloud data to obtain fitted point cloud data; Segment the top container point cloud plane and the vertical container point cloud plane of each container space from the fitted point cloud data to obtain a target point cloud plane; According to the target point cloud plane, the coordinates of the corner points of each container position in the container yard are determined to obtain the plane features of the container yard.

4. The method according to claim 1, characterized in that: The step of determining the relative position of the target between the current frame and the nearest adjacent plane in the initial map according to the sensor data and the plane features of the container yard comprises: Determine from the initial map an adjacent point cloud plane that is closest to the point cloud plane in the current frame; Determine an initial relative pose between the point cloud plane in the current frame and an adjacent point cloud plane in the initial map according to the sensor data; The point cloud plane in the current frame is matched with the adjacent point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between the two.

5. The method according to claim 4, characterized in that The matching of the point cloud plane in the current frame with the adjacent point cloud plane in the initial map to optimize the initial relative pose and obtain the target relative pose between the two includes: Determine the corner point coordinates and normal vector direction of the point cloud plane in the initial map corresponding to the point cloud plane in the current frame; Determine the translation amount according to the corner point coordinates, the corner point coordinates of a preset number of point cloud planes from the perspective of the drone, and the preset number of point cloud planes from the perspective of the truck; Determine the rotation amount according to the normal vector direction, the normal vector directions of a preset number of point cloud planes from the perspective of the drone, and the normal vector directions of a preset number of point cloud planes from the perspective of the truck; The relative position and orientation of the target is determined according to the translation amount and the rotation amount.

6. The method according to claim 4, characterized in that After obtaining the relative position and posture of the target between the two, the method further includes: The edge point cloud is determined from the point cloud plane, and the edge point cloud in the current frame is aligned with the edge point cloud in the initial map to obtain the optimized relative position and posture of the target.

7. The method according to claim 1, characterized in that After constructing the target map of the container yard according to the target relative position, the method further includes: Generate a depth image based on the point cloud data of consecutive frames of the same viewpoint; Comparing the depth image with the pixel depth at the same position in the initial map, determining the dynamic point cloud through point cloud occlusion and deleting it, and obtaining the target point cloud; The map of the container yard is updated according to the target point cloud and the historical map.

8. The method according to claim 7, characterized in that The updating of the map of the container yard according to the target point cloud and the historical map includes: Determine the current map according to the target point cloud; compare the pixel depths of the current map and the same position in the historical map, retain the dynamic point cloud of the current map relative to the historical map and the static point cloud in the historical map, and obtain an updated map of the container yard.

9. The method according to claim 1, characterized in that: After constructing the target map of the container yard according to the target relative position, the method further includes: The air-ground collaborative long-term SLAM system is used to collect the location descriptor of the container yard, and the current topological relationship of the container yard is constructed through a chessboard structure; the location descriptor includes characteristic data and location information of the container; Determining a loop loss value according to a difference between a current topological relationship and a historical topological relationship of the container yard and a preset loop loss function; The drone is used to detect the position of the container truck, and the position of the container truck is determined as the initial position of the open space at both ends of the map for splicing the container yard; The container yard is subjected to loop detection according to the loop loss value, and the map of the container yard is optimized according to the loop loss function, the initial position of the open space, the preset chessboard topological relationship, and the preset peripheral ring distribution topological relationship.

10. A container yard map construction device, characterized in that: include: The hardware system building module is configured to build an air-ground collaborative long-term SLAM system including a container truck and an unmanned aerial vehicle; the bottom of the unmanned aerial vehicle is equipped with a first radar, an image acquisition device and a first inertial sensor, and the top of the container truck is equipped with a second radar and a second inertial sensor; A data acquisition module is configured to use the air-ground collaborative long-term SLAM system to collect point cloud data, sensor data and image data of the container yard, and construct an initial map of the container yard based on the point cloud data and the image data; a feature extraction module configured to determine the plane features of the container yard according to the size information of the containers in the container yard and the point cloud data; The map construction module is configured to determine the target relative pose between the current frame and the nearest adjacent plane in the initial map according to the sensor data and the plane features of the container yard, and construct a target map of the container yard according to the target relative pose.

Images