Dense mapping method, crane and computer readable storage medium

Abstract

The invention discloses a dense mapping method, a crane and a readable storage medium, the arm head of the crane is provided with an image acquisition device and a point cloud data acquisition device, and the view fields of the image acquisition device and the point cloud data acquisition device cover the lower area of the arm head along the gravity direction. The method comprises the following steps: acquiring single-frame point cloud data and a first image which are respectively acquired by a point cloud data acquisition device and an image acquisition device at the same time; determining a pixel set of the load in the first image, and obtaining a first point cloud set corresponding to the pixel set in the single-frame point cloud data; and determining a grid map based on the position of the odometer in the world coordinate system, and determining a dense point cloud according to the grid map and the first point cloud set. By implementing the embodiment of the invention, the dense point cloud with relatively high precision and relatively good real-time performance can be generated in real time, so that the dense point cloud is used for load obstacle avoidance and planning.

Description

Technical Field

[0001] The present application relates to the technical field of engineering machinery, and in particular to a dense mapping method, a crane, and a computer-readable storage medium. Background Art

[0002] When the crane is in operation, the operator needs to pay attention to the surrounding environment of the load at all times to prevent the load from colliding with obstacles in the surrounding environment, causing safety accidents or damage to the transported load. The current common method is to observe the lifting area only through human visual observation or arm head camera, which is not only time-consuming and labor-intensive, but also has the risk of observation omissions. On the one hand, in order to assist manual avoidance of collision risks, it is necessary to densely map the scene around the load and obtain information about obstacles in the surrounding scene. On the other hand, the automation and intelligence of crane hoisting requires planning of hoisting operations and paths based on the information of obstacles in the surrounding scene and adjusting strategies in real time, which also puts forward requirements for the function of dense mapping.

[0003] The existing method of densely mapping the environment usually involves first obtaining a point cloud in an inertial coordinate system through SLAM technology, then filtering the point cloud to remove environmental noise and dynamic target points, and finally obtaining a dense point cloud by directly splicing the point cloud or calculating the hit probability of the grid. However, the above methods are difficult to meet the needs of practical applications in terms of mapping accuracy and real-time computing. Summary of the invention

[0004] The embodiments of the present application disclose a dense mapping method, a crane, and a computer-readable storage medium, which can generate a dense point cloud with high accuracy and good real-time performance in real time for use in load obstacle avoidance and planning.

[0005] To achieve the above objectives:

[0006] In a first aspect, an embodiment of the present application provides a dense mapping method, which is applied to a crane, wherein the boom head of the crane is provided with an image acquisition device and a point cloud data acquisition device, and the field of view of the image acquisition device and the point cloud data acquisition device covers the area below the boom head along the gravity direction, and the method comprises:

[0007] Acquire the single-frame point cloud data and the first image acquired simultaneously by the point cloud data acquisition device and the image acquisition device;

[0008] Determine a pixel set of the load in the first image, and obtain a first point cloud set corresponding to the pixel set in the single frame point cloud data;

[0009] Based on the position of the odometer in the world coordinate system, a grid map is determined, and based on the grid map and the first point cloud set, a dense point cloud is determined.

[0010] In one embodiment, determining a pixel set of the load in the first image and acquiring a first point cloud set corresponding to the pixel set in the single frame point cloud data includes:

[0011] Performing filtering processing on the single frame point cloud data to obtain first point cloud data;

[0012] Acquire an instance segmentation mask image of the payload based on the first image, and determine pixels of the instance segmentation mask image of the payload in the first image as a pixel set;

[0013] A first point cloud set corresponding to the first point cloud data and the pixel set is determined.

[0014] In one embodiment, determining the first point cloud set corresponding to the first point cloud data and the pixel set includes:

[0015] Acquire an intrinsic parameter matrix of the image acquisition device and an extrinsic parameter transformation matrix from the point cloud data acquisition device to the image acquisition device;

[0016] Determine a first transformation matrix from the point cloud data acquisition device to the coordinate system corresponding to the image acquisition device according to the intrinsic parameter matrix and the extrinsic parameter transformation matrix;

[0017] Based on the first transformation matrix, the first point cloud data is projected onto the plane where the first image is located, and a set of first point cloud data whose projection points fall into target pixels is determined as a first point cloud set; wherein the target pixel is any pixel in the pixel set.

[0018] In one embodiment, filtering the single frame point cloud data to obtain first point cloud data includes:

[0019] Filter out low-intensity points, outlier points, points outside the intersection of the effective field of view of the point cloud data acquisition device, and points outside the intersection of the effective field of view of the image acquisition device in the single-frame point cloud data to obtain first point cloud data;

[0020] Wherein, the low-intensity point is point cloud data whose point cloud intensity is lower than the intensity threshold;

[0021] The outlier point is the nearest neighbor N nei_0 The point cloud data whose average distance between points is greater than λ0 times the standard deviation of the distance between points in the entire region.

[0022] In one embodiment, before determining the grid map based on the position of the odometer in the world coordinate system and after obtaining the first point cloud set corresponding to the pixel set in the single frame point cloud data, the method further includes:

[0023] Based on the inertial odometer, a transformation matrix from the point cloud data acquisition device to a map coordinate system is obtained; wherein the origin of the map coordinate system coincides with the initial point of the inertial odometer, and the z-axis points upward along the anti-gravity direction;

[0024] Projecting the first point cloud set to the map coordinate system to obtain a projected point cloud set;

[0025] Projecting the first point cloud set to a preset plane of the world coordinate system, extracting the convex hull domain of the projection point set, and stretching it upward to the preset plane with the convex hull as the base to construct a load envelope area;

[0026] Points in the projected point cloud set that are within the load envelope area are filtered out to obtain a second point cloud set.

[0027] In one embodiment, determining a grid map based on the position of the odometer in the world coordinate system, and determining a dense point cloud based on the grid map and the first point cloud set includes:

[0028] Acquire a position vector of the inertial odometer in a world coordinate system, and acquire a target point cloud set centered on the position vector from the first point cloud set;

[0029] Taking the position vector as the center, the spatial region is divided into a set of two-dimensional square grids parallel to the vertical coordinate; wherein each of the grids contains an elevation;

[0030] A dense point cloud is generated according to the target point cloud set, the horizontal coordinate and the vertical coordinate of the center of each grid, and the mean grid elevation.

[0031] In one embodiment, generating a dense point cloud according to the target point cloud set, the horizontal coordinate and the vertical coordinate of the center of each grid, and the mean grid elevation includes:

[0032] Initializing the grid that first contains point cloud data according to the target point cloud set and each of the grids, and updating the grid that contains new point clouds;

[0033] A dense point cloud is generated according to the abscissa and ordinate of the center of each initialization grid, the abscissa and ordinate of the center of each updated grid, and the mean grid elevation.

[0034] In one embodiment, the grid includes a static grid and a dynamic grid;

[0035] Initializing the grid containing point cloud data for the first time according to the target point cloud set and each of the grids, and updating the grid containing the new point cloud, comprises:

[0036] Acquire an instance segmentation mask image of the payload based on the first image;

[0037] If the target point cloud data is within the instance segmentation mask image of the payload, the corresponding static grid is updated, initialized or skipped according to the initialization of the static grid corresponding to the target point cloud data; wherein the target point cloud data is any point cloud data in the target point cloud set;

[0038] If the target point cloud data is outside the instance segmentation mask image of the payload, initializing or updating the corresponding static grid according to the vertical coordinate of the target point cloud data and the elevation of the corresponding static grid;

[0039] If the target point cloud data is outside the instance segmentation mask image of the payload, the corresponding dynamic grid is initialized or updated according to the vertical coordinate of the target point cloud data, the elevation of the corresponding static grid and the elevation of the corresponding dynamic grid.

[0040] In a second aspect, an embodiment of the present application provides a crane, comprising: an image acquisition device and a point cloud data acquisition device arranged on a boom head, and the field of view of the image acquisition device and the point cloud data acquisition device covers the area below the boom head along the direction of gravity; and a processor and a memory storing a computer program, wherein when the processor runs the computer program, the dense mapping method as described above is implemented.

[0041] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned dense mapping method is implemented.

[0042] The dense mapping method, crane and computer-readable storage medium provided by the embodiment of the present application, the boom head of the crane is provided with an image acquisition device and a point cloud data acquisition device, and the field of view of the image acquisition device and the point cloud data acquisition device covers the area below the boom head along the gravity direction, the method includes: acquiring single-frame point cloud data and a first image acquired simultaneously by the point cloud data acquisition device and the image acquisition device, and determining a pixel set of the load in the first image, and then acquiring a first point cloud set corresponding to the pixel set in the single-frame point cloud data, and then determining a grid map based on the position of the odometer in the world coordinate system, and processing the grid map according to the first point cloud set to determine a dense point cloud. By acquiring point cloud data and images at the same time, the point cloud data belonging to the load in the point cloud data is determined according to the pixel part of the load in the image, and then by constructing a grid map, and processing the grid map according to the position between each point cloud data and the grid map, a dense point cloud of the scene around the load can be generated in real time for use in aspects such as load obstacle avoidance, planning and dynamic target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is an application scenario diagram of the dense mapping method disclosed in an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a flow chart of a dense mapping method disclosed in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a flow chart of another dense mapping method disclosed in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of a grid disclosed in an embodiment of the present invention;

[0047] Figure 5 The structure of the crane disclosed in the embodiment of the present invention is shown in FIG. Figure 1 ;

[0048] Figure 6 The structure of the crane disclosed in the embodiment of the present invention is shown in FIG. Figure 2 . DETAILED DESCRIPTION

[0049] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0050] It should be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0051] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Thus, “A, B, or C” or “A, B and / or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will occur only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0052] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0053] It should be noted that, in this article, step codes such as S101, S102, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the scope of protection of this application.

[0054] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0056] See also Figure 1 , Figure 1 FIG. 1 is an application scenario diagram of a dense mapping method in an embodiment. Figure 1 As shown, a crane 10 may be included, and the crane 10 at least includes a main body 11, a boom head 12 and a processor 13, and the processor 13 may be a computer or a computer cluster. The computer system in the processor 13 may be Windows, Linux, IOS or Unix, which is not specifically limited here. The processor 13 may be set on the main body 11 or the boom head 12 of the crane 10 in an external manner or an internal manner. The boom head 12 of the crane 10 is provided with an image acquisition device and a point cloud data acquisition device, and the field of view of the image acquisition device and the point cloud data acquisition device covers the lower area of ​​the boom head 12 along the gravity direction. The processor 13 of the crane 10 can obtain the single-frame point cloud data and the first image collected by the point cloud data acquisition device and the image acquisition device at the same time, respectively, determine the pixel set of the load in the first image, and obtain the first point cloud set corresponding to the pixel set in the single-frame point cloud data, determine the grid map based on the position of the odometer in the world coordinate system, and determine the dense point cloud according to the grid map and the first point cloud set.

[0057] See also Figure 2 , Figure 2 is a schematic diagram of a dense mapping method in one embodiment, which can be applied to Figure 1 In the application scenario shown, the crane 10 has an arm head 12 provided with an image acquisition device and a point cloud data acquisition device, and the field of view of the image acquisition device and the point cloud data acquisition device covers the area below the arm head 12 along the direction of gravity. Figure 2 As shown, the method may include the following steps:

[0058] 210. Acquire single-frame point cloud data and a first image acquired simultaneously by a point cloud data acquisition device and an image acquisition device respectively.

[0059] In the embodiment of the present application, it should be noted that the image acquisition device and the point cloud data acquisition device are arranged on the boom of the crane, and the field of view of the image acquisition device and the point cloud data acquisition device covers the area below the boom along the direction of gravity to ensure that the information of the area where the load is located can be acquired. Among them, the image acquisition device is used to acquire images of the area below the boom, and may include, for example, an RGB camera. The point cloud data acquisition device is used to acquire single-frame point cloud data corresponding to the area below the boom, and may include, for example, a solid-state laser radar, a depth camera, etc.

[0060] The processor can obtain the color image corresponding to the area below the arm head and the single-frame point cloud data corresponding to the area below the arm head respectively collected by the image acquisition device and the point cloud data acquisition device at the same time, so that the image and the single-frame point cloud data are synchronized in time. Here, the single-frame point cloud data includes at least one point cloud; the single-frame point cloud data and the image include information about the area where the load is located. The first image is an image collected by the image acquisition device without any processing.

[0061] 220. Determine a pixel set of the load in the first image, and obtain a first point cloud set corresponding to the pixel set in the single frame point cloud data.

[0062] In an embodiment of the present application, the processor can screen each pixel in the first image according to the information of the pixel corresponding to the load, so as to determine the pixel set of the load in the first image. For each point cloud data in a single frame of point cloud data, the processor can determine whether the point cloud data falls into each pixel of the load in the first image according to the coordinate information of the point cloud data corresponding to the first image. If so, the point cloud data is determined as the point cloud data in the first point cloud set; if not, the point cloud data is not added to the first point cloud set. Among them, the first point cloud set is a set of point cloud data corresponding to any pixel of the load in the first image in a single frame of point cloud data.

[0063] In some embodiments, the process in which the processor determines the pixel set of the load in the first image in step 220 and obtains the first point cloud set corresponding to the pixel set in the single frame point cloud data may further include the following steps:

[0064] Performing filtering processing on the single-frame point cloud data to obtain first point cloud data;

[0065] Acquire an instance segmentation mask image of the payload based on the first image, and determine a pixel set of the instance segmentation mask image of the payload in the first image;

[0066] A first point cloud set corresponding to the first point cloud data and the pixel set is determined.

[0067] In the embodiment of the present application, the processor may perform preliminary filtering on the single-frame point cloud data to filter out some environmental noise points and obtain first point cloud data, wherein the first point cloud data refers to the point cloud data obtained after filtering the single-frame point cloud data.

[0068] The processor can perform recognition processing on the first image through the trained model, identify and mark the instance segmentation mask image of the payload in the first image, that is, the instance segmentation mask image of the payload, and determine each pixel of the instance segmentation mask image of the payload in the first image, and then determine the pixel set of the instance segmentation mask image of the payload in the first image. Specifically, the processor uses OpenCV and camera intrinsic parameters and distortion parameters to dedistort the image, and uses a common depth model to obtain the instance segmentation mask image of the payload in the image, and determines the pixel set of the corresponding area.

[0069] The processor can project each first point cloud data onto the plane of the instance segmentation mask image of the payload, and determine whether each first point cloud data falls into the pixel set corresponding to the instance segmentation mask image of the payload. If so, it is determined to be the point cloud data contained in the first point cloud set; if not, the point cloud data can be filtered out.

[0070] By adopting the above-mentioned embodiment, the accuracy of point cloud data can be improved through filtering processing, and the amount of point cloud data to be processed can be effectively reduced, thereby improving real-time performance; by determining the relationship between the first point cloud data and the load through an instance segmentation mask image based on the load, the accuracy of dense mapping can be further improved.

[0071] In some embodiments, the processor may further include the following steps in the process of determining the first point cloud set corresponding to the first point cloud data and the pixel set:

[0072] Obtaining an intrinsic parameter matrix of the image acquisition device and an extrinsic parameter transformation matrix from the point cloud data acquisition device to the image acquisition device;

[0073] Determine a first transformation matrix from the point cloud data acquisition device to the coordinate system corresponding to the image acquisition device according to the intrinsic parameter matrix and the extrinsic parameter transformation matrix;

[0074] Based on the first transformation matrix, the first point cloud data is projected onto the plane where the first image is located, and a set of first point cloud data whose projection points fall into a target pixel is determined as a first point cloud set; wherein the target pixel is any pixel in the pixel set.

[0075] In the embodiment of the present application, the processor can obtain the camera intrinsic parameter matrix T through a common calibration method such as camera intrinsic parameter calibration or calibration plate calibration. int , distortion parameters and external parameter transformation from radar to camera T cl Then the processor calculates the first transformation matrix T corresponding to the coordinate system of the image acquisition device according to the above parameters. il ,in

[0076]

[0077] The processor projects the first point cloud data onto the plane where the first image is located using the first transformation matrix, and if the projection point falls into any pixel in the pixel set, the point cloud data is determined as the point cloud data in the first point cloud set.

[0078] By adopting the above embodiment, by acquiring the internal parameter data of the image acquisition device and the transformation matrix between the point cloud data acquisition device and the image acquisition device, the first point cloud data can be accurately projected onto the plane where the first image is located, so that the first point cloud set can be determined more accurately, further improving the accuracy of dense mapping.

[0079] In some embodiments, when the processor performs filtering processing on a single frame of point cloud data to obtain first point cloud data, the following steps may be included:

[0080] Filter out low-intensity points, outlier points, and points outside the intersection of the effective field of view of the point cloud data acquisition device and the image acquisition device in the single-frame point cloud data to obtain first point cloud data;

[0081] Among them, low-intensity points are point cloud data whose point cloud intensity is lower than the intensity threshold;

[0082] The outlier point is the nearest neighbor N nei_0 The point cloud data whose average distance between points is greater than λ0 times the standard deviation of the distance between points in the entire region.

[0083] In the embodiment of the present application, the specific method for the processor to filter the single-frame point cloud data is: filtering out the low-intensity points, outlier points, and points outside the intersection of the effective field of view of the point cloud data acquisition device and the image acquisition device in the single-frame point cloud data to obtain the first point cloud data. Among them, the low-intensity points are point cloud data with a point cloud intensity lower than the intensity threshold; the outlier points are points with the nearest N neighbors. nei_0 The point cloud data whose average distance between points is greater than λ0 times the standard deviation of the distance between points in the entire region.

[0084] By adopting the above embodiment and the above filtering method, point cloud data that has a greater impact on the accuracy of dense mapping in a single frame of point cloud data can be filtered out more accurately, thereby further improving the accuracy of dense mapping.

[0085] 230. Determine a grid map based on the position of the odometer in the world coordinate system, and determine a dense point cloud based on the grid map and the first point cloud set.

[0086] In an embodiment of the present application, the processor divides the spatial region based on the position of the odometer in the world coordinate system to determine one or more grid maps. For example, the spatial region can be divided into several grid maps with the position of the odometer in the world coordinate system as the center or boundary.

[0087] Then the processor can determine the dense point cloud based on each grid map and the first point cloud set. For example, the processor can determine whether the point cloud in the first point cloud set appears in each grid map. If yes, the grid map is retained; if not, the grid map is deleted; and the corresponding dense point cloud is generated according to the coordinate information of each retained grid map. For example, the processor can determine the coordinate information of the center point based on the horizontal and vertical coordinate information of each grid map, and use the coordinate information as the coordinate information of the dense point cloud. Alternatively, the processor can calculate the average value based on the coordinate information of each point cloud contained in each grid map, and use the average value as the coordinate information of the dense point cloud.

[0088] In some embodiments, the process of the processor executing step 230 to determine a grid map based on the position of the odometer in the world coordinate system, and determining a dense point cloud based on the grid map and the first point cloud set may include the following steps:

[0089] Obtain the position vector of the inertial odometer in the world coordinate system, and obtain the target point cloud set centered on the position vector from the first point cloud set;

[0090] Taking the position vector as the center, the spatial area is divided into a set of two-dimensional square grids parallel to the vertical coordinate; each grid contains elevation;

[0091] A dense point cloud is generated based on the target point cloud set, the horizontal and vertical coordinates of the center of each grid, and the mean grid elevation.

[0092] In the embodiment of the present application, the processor can obtain the position vector of the inertial odometer such as the visual inertial odometer (Visual-Inertial Odometry, VIO) in the world coordinate system, and record the position vector of the current odometer in the world coordinate system as c t ,but

[0093]

[0094] The processor then obtains the t The point cloud centered at {q i}

[0095] q i =p i -c t

[0096] And c t The spatial area is divided into a set of two-dimensional square grids parallel to the plane. Each grid can contain the following properties:

[0097] A. Elevation h ij , which represents the z coordinate of the object in the environment space corresponding to the grid, and is described by a one-dimensional Gaussian distribution

[0098] h ij ~N(μ ij ,σ ij )

[0099] Among them, μ i j and σ i j is initialized to a given value, and the specific update strategy is described later.

[0100] B. Timestamp t ij Indicates the system timestamp when the elevation corresponding to the Gaussian distribution was last updated

[0101] C. Number of consecutive hits N shot

[0102]

[0103] The processor screens the grids according to the target point cloud set, determines the grids that meet the conditions, and generates a dense point cloud based on the horizontal and vertical coordinates of the center of each grid that meets the conditions, as well as the mean value of the grid elevation.

[0104] By using the above embodiment, a grid between two-dimensional and three-dimensional can be constructed, and then a grid that meets the conditions can be determined based on the relationship between the grid and the point cloud. Then, a dense point cloud can be generated based on the coordinate information and the attributes contained in the grid that meets the conditions. A dense point cloud of the scene around the load can be generated in real time.

[0105] In some embodiments, the processor may generate a dense point cloud according to the target point cloud set, the horizontal coordinate and the vertical coordinate of the center of each grid, and the grid elevation mean value, by performing the following steps:

[0106] According to the target point cloud set and each grid, the grid containing the point cloud data for the first time is initialized, and the grid containing the new point cloud is updated;

[0107] A dense point cloud is generated according to the abscissa and ordinate of the center of each initialization grid, the abscissa and ordinate of the center of each updated grid, and the mean grid elevation.

[0108] (4) In the embodiment of the present application, for each grid, if it contains any point cloud data in the target point cloud set for the first time, the processor initializes the grid; if the grid already contains point cloud data and has point cloud data in the target point cloud set, that is, new point cloud data falls into the grid, the processor updates the grid. Then the processor can initialize each grid G ij Generate point η with its center x, y coordinates and grid elevation mean μ ij

[0109]

[0110] Then according to point η ij The collection of composes a dense point cloud.

[0111] By adopting the above embodiment and performing operations of initializing and / or updating the grid, the accuracy of generating a dense point cloud based on the grid can be improved.

[0112] By adopting the above-mentioned embodiment, the point cloud data is screened through a point cloud data acquisition device such as a laser radar in combination with an image acquisition device such as a camera, and a dense point cloud is generated by means of grid probability mapping to generate a static dense point cloud of the scene around the load in real time. The mapping accuracy and calculation real-time performance can be improved to meet actual needs. The obtained dense point cloud can be effectively used for load obstacle avoidance and planning.

[0113] In one embodiment, see Figure 3 , Figure 3 is a flowchart of another dense mapping method disclosed in an embodiment. The method can be applied to Figure 1 In the application scenario shown, the crane 10 has an arm head 12 provided with an image acquisition device and a point cloud data acquisition device, and the field of view of the image acquisition device and the point cloud data acquisition device covers the area below the arm head 12 along the direction of gravity. Figure 3 As shown, the method may include the following steps:

[0114] 310. Acquire single-frame point cloud data and a first image simultaneously acquired by a point cloud data acquisition device and an image acquisition device respectively.

[0115] 320. Determine a pixel set of the load in the first image, and obtain a first point cloud set corresponding to the pixel set in the single frame point cloud data.

[0116] 330. Based on the inertial odometer, obtain the transformation matrix from the point cloud data acquisition device to the map coordinate system; wherein the origin of the map coordinate system coincides with the initial point of the inertial odometer, and the z-axis points upward along the anti-gravity direction.

[0117] 340. Project the first point cloud set to the map coordinate system to obtain a projected point cloud set.

[0118] 350. Project the first point cloud set to a preset plane of the world coordinate system, extract the convex hull domain of the projection point set, and stretch it upward to the preset plane with it as the base to construct the load envelope area.

[0119] 360. Filter out the points in the load envelope area in the projected point cloud set to obtain a second point cloud set.

[0120] In the embodiment of the present application, the processor obtains the transformation T of the point cloud data acquisition device to the map coordinate system at the current moment based on the SLAM technology of the laser radar inertial odometer. wl Among them, the origin of the map coordinate system coincides with the initial point of the odometer, and the z-axis points upward along the anti-gravity direction.

[0121] Project the current point cloud to the map coordinate system to obtain the projected point cloud set

[0122] p i :=T wl p i

[0123] Among them, := means replacing the parameters on the right side of the equation with the parameters on the left side of the equation.

[0124] The processor collects the first point cloud {p i} seg Projected to the preset plane of the world coordinate system, such as z=min({p iz}), extract the convex hull domain S of the projection point set hull , and the convex hull domain S hull Stretch from bottom to top to the preset plane z=max({p iz} seg ), the prism domain formed is the load envelope area V hull .

[0125] The processor filters out the points in the load envelope area in the projected point cloud set to obtain the points in the load envelope area, i.e., the second point cloud set.

[0126]

[0127] By adopting the above embodiment, the point cloud data falling into the load envelope area can be filtered out, reducing the influence of the envelope on the dense mapping process, and further improving the accuracy of dense mapping.

[0128] 370. Determine a grid map based on the position of the odometer in the world coordinate system, and determine a dense point cloud based on the grid map and the first point cloud set.

[0129] Based on the same inventive concept as the above-mentioned embodiment, the above-mentioned embodiment is described in detail below through a specific example. In this example, the point cloud data acquisition device is a laser radar and the image acquisition device is an RGB camera.

[0130] The dense mapping method provided in this embodiment mainly includes the following steps:

[0131] 1. Deploy the LiDAR and RGB camera at the crane boom head, with the field of view covering the area below the boom head along the gravity direction. All collected single-frame point cloud data are located in the radar coordinate system (also called point cloud coordinate system), with the origin at the radar position and the x-axis pointing downward.

[0132] 2. Obtain the camera intrinsic parameter matrix T through common calibration methods int , distortion parameters and external parameter transformation from radar to camera T cl .

[0133] 3. Calculate the transformation matrix T from radar to image coordinate system il ,in

[0134]

[0135] 4. Obtain single-frame point cloud data of the lidar and RGB images with timestamps consistent with the lidar frame.

[0136] 5. Perform preliminary filtering on the radar frame point cloud to remove some environmental noise points and obtain the point cloud {P i}, optionally, filter out low intensity points (intensity threshold thr_int0), outliers (with the nearest N neighbors nei_0 The average distance of the points is greater than λ (0 times the distance standard deviation) of the probability distribution of the overall point spacing in the area, as well as points outside the intersection of the effective field of view of the radar and camera.

[0137] 6. Use OpenCV and camera intrinsic parameters and distortion parameters to dedistort the image, and use common depth models to obtain the instance segmentation mask image Mask of the payload in the image seg , and the pixel set of the corresponding area is recorded as S seg .

[0138] 7. Get and S seg The corresponding point cloud set {p i} seg Optionally, T il Each point in the point cloud is projected onto the camera image plane, and the projection point is determined by whether it is included in S seg Determine whether the point belongs to the load within the range

[0139]

[0140] p'=T il p

[0141]

[0142] 8. Based on the SLAM technology (LIO) of LiDAR inertial odometer, obtain the transformation T from the current radar to the map coordinate system wlAmong them, the origin of the map coordinate system coincides with the initial point of the odometer, and the z-axis points upward along the anti-gravity direction.

[0143] 9. Project the current point cloud to the map coordinate system

[0144] p i :=T wl p i

[0145] 10. i} seg Projected onto the world coordinate plane z=min({p iz}), extract the convex hull domain S of the projection point set hull , stretching upward to the plane z=max({p iz} seg ), the formed prism domain is the load envelope area V hull .

[0146] 11. Filter out the points within the load envelope area, that is

[0147]

[0148] 12. Create a dense point cloud

[0149] (1) The current odometer position vector in the world coordinate system is c t ,but

[0150]

[0151] (2) Get the c t The point cloud centered at {q i}

[0152] q i =p i -c t

[0153] (3) c t As the center, the spatial area is divided into a set of two-dimensional square grids parallel to the plane, such as Figure 4 As shown. Each grid side has a length of d, there are N grids on each side, and the total side length is 2D. Each grid G ij Initialization is performed when a point cloud is first included. i}When you arrive

[0154] Update. Contains the following properties:

[0155] D. Height h ij , which represents the z coordinate of the object in the environment space corresponding to the grid, and is described by a one-dimensional Gaussian distribution

[0156] h ij ~N(μ ij ,σ ij )

[0157] Among them, μ i j and σ i j is initialized to a given value, and the specific update strategy is described later.

[0158] E. Timestamp t ij Indicates the system timestamp when the elevation corresponding to the Gaussian distribution was last updated

[0159] F. Number of consecutive hits N shot

[0160]

[0161] (5) If there is a grid map, translate the grid map by r and delete the out-of-range

[0162] {-Dd,D+d} grid, and initialize the newly generated grid. Filter out the points outside the grid range, that is,

[0163]

[0164] (6) Two grid maps for environmental space and Description, representing static targets and dynamic targets respectively, the corresponding update strategies are as follows:

[0165] (5-1) Traverse each point in the input point cloud. If the input point cloud is within the input segmentation mask image of the payload, it includes:

[0166] If the corresponding static grid is not initialized and the input point cloud is the first frame point cloud, the given height h is used. ij0 and standard deviation σ min Initialize the static grid;

[0167] If the corresponding static grid has been initialized, skip the static grid.

[0168] (5-2) Traverse each point in the input point cloud. If the input point cloud is outside the input segmentation mask image of the payload, it includes:

[0169] If the corresponding static grid is not initialized, or the z coordinate of the input point cloud is less than the mean elevation of the corresponding static grid, the current z coordinate of the input point cloud and the standard deviation σ are used. min Initialize the static grid;

[0170] If the z coordinate of the input point cloud is higher than the static grid elevation, but the deviation is within thr ijIf the point is within the range, the corresponding static grid is updated with the current z coordinate of the input point cloud;

[0171] If the z coordinate of the input point cloud is higher than the static grid elevation, but the deviation is within thr ij Out of range, and there is no corresponding dynamic grid or the deviation from the corresponding dynamic grid is within the threshold thr_dyn ij In addition, the corresponding dynamic grid is initialized with the current z coordinate of the input point cloud;

[0172] If the z coordinate of the input point cloud is higher than the static grid elevation, but the deviation is within thr ij The range is outside, and the deviation from the corresponding dynamic grid is within the threshold thr_dyn ij If the point cloud is within the specified range, the corresponding dynamic grid is updated with the current z coordinate of the input point cloud.

[0173] (5-3) Traverse the dynamic grid. If the initialized dynamic grid is within the input segmentation mask image of the payload, the dynamic grid is set to uninitialized;

[0174] (5-4) Traverse the dynamic grid. If the initialized dynamic grid is outside the input segmentation mask image of the payload, it includes:

[0175] If the difference between the timestamp of the current point cloud frame and the timestamp of the dynamic grid is less than thr_t, the dynamic grid is set to uninitialized;

[0176] If the difference between the timestamp of the current point cloud frame and the timestamp of the dynamic grid is greater than or equal to thr_t, and the number of consecutive hits of the dynamic grid is greater than thr_N min , the dynamic grid covers the corresponding static grid and deletes the dynamic grid;

[0177] If the difference between the timestamp of the current point cloud frame and the timestamp of the dynamic grid is greater than or equal to thr_t, and the number of consecutive hits of the dynamic grid is less than or equal to thr_N min , then skip.

[0178] Among them, when the mean μ and standard deviation σ are given for initialization, the Gaussian distribution corresponding to the grid is directly updated to N(μ,σ 2 ).

[0179] When the grid is updated with the z coordinate,

[0180] μ:=μ+(z-μ) / N shot

[0181]

[0182] σ:=max(σ,σ min )

[0183] h ij0 Used to initialize the static grid, which is obtained by performing the nearest neighbor bilinear interpolation on the static grid map; σ min Indicates the preset lower bound of the standard deviation; thr ij Indicates the preset static neighbor deviation distance; thr_dyn ij Indicates the preset dynamic neighbor deviation distance; thr_t indicates the preset maximum unupdated time interval of the dynamic grid. The dynamic grid that is not updated after this time will be set as uninitialized; thr_N min Indicates the preset minimum number of consecutive hits for dynamic-static conversion. When the number of consecutive hits of the dynamic grid exceeds this threshold, the dynamic grid is converted to a static grid and the original dynamic grid is set to uninitialized.

[0184] (7) Each grid G ij Generate point η with its center x, y coordinates and grid elevation mean μ ij

[0185]

[0186] Static raster map Generate dense point cloud of static obstacles Dynamic raster map Generating dynamic object dense point cloud

[0187] By adopting the above embodiment, according to the crane scene operation and anti-collision requirements, a two-dimensional Gaussian map can be used to describe the three-dimensional scene, and two Gaussian maps are used to describe static and dynamic targets respectively, and the two Gaussian maps can be converted to each other according to the strategy. In addition, the update of a single grid is constrained by setting a hit threshold and updating the lower bound of the standard deviation.

[0188] In one embodiment, the grid includes a static grid and a dynamic grid;

[0189] According to the target point cloud set and each grid, the grid containing the point cloud data for the first time is initialized, and the grid containing the new point cloud is updated, including:

[0190] Obtain an instance segmentation mask image of the payload based on the first image;

[0191] If the target point cloud data is within the instance segmentation mask image of the payload, the corresponding static grid is updated, initialized or skipped according to the initialization of the static grid corresponding to the target point cloud data; wherein the target point cloud data is any point cloud data in the target point cloud set;

[0192] If the target point cloud data is outside the instance segmentation mask image of the payload, the corresponding static grid is initialized or updated according to the vertical coordinate of the target point cloud data and the elevation of the corresponding static grid;

[0193] If the target point cloud data is outside the instance segmentation mask image of the payload, the corresponding dynamic grid is initialized or updated according to the vertical coordinate of the target point cloud data, the elevation of the corresponding static grid, and the elevation of the corresponding dynamic grid.

[0194] In an embodiment of the present application, the processor updates, initializes or skips processing of static grids and dynamic grids according to the target point cloud set and each grid, as shown in the update strategy of steps (5-1) to (5-4) above, which will not be repeated here.

[0195] Based on the same inventive concept as the above embodiments, Figure 5 As shown, an embodiment of the present invention further provides a crane, comprising:

[0196] An image acquisition device 1 and a point cloud data acquisition device 2 are arranged on the boom head of the crane, and the fields of view of the image acquisition device 1 and the point cloud data acquisition device 2 cover the area below the boom head along the gravity direction; and

[0197] The processor 3 and the memory 4 storing the computer program implement the dense mapping method as described above when the processor 3 runs the computer program.

[0198] Based on the same inventive concept as the above-mentioned embodiment, the embodiment of the present invention further provides a crane, such as Figure 6 As shown, the crane includes: a processor 610 and a memory 611 storing a computer program; wherein, Figure 6 The processor 610 shown in the figure is not used to indicate that the number of the processor 610 is one, but is only used to indicate the position relationship of the processor 610 relative to other devices. In actual applications, the number of the processor 610 may be one or more; similarly, Figure 6 The memory 611 shown in the figure has the same meaning, that is, it is only used to refer to the position relationship of the memory 611 relative to other devices. In practical applications, the number of memories 611 can be one or more. When the processor 610 runs the computer program, the above-mentioned dense mapping method is implemented.

[0199] The crane may also include: at least one network interface 612. The various components in the crane are coupled together via a bus system 613. It is understood that the bus system 613 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 613 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 613 is not used in the following examples. Figure 6Various buses are labeled as bus system 613.

[0200] The memory 611 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 611 described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0201] The memory 611 in the embodiment of the present invention is used to store various types of data to support the operation of the crane. Examples of these data include: any computer programs used to operate on the crane, such as operating systems and applications; contact data; phone book data; messages; pictures; videos, etc. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program can include various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. Here, the program that implements the method of the embodiment of the present invention can be included in the application program.

[0202] Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer-readable storage medium, in which a computer program is stored. The computer-readable storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); it may also be various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer-readable storage medium is executed by the processor, the above-mentioned dense mapping method is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to Figure 2 The description of the illustrated embodiment will not be repeated here.

[0203] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0204] In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

[0205] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dense mapping method, applied to cranes, characterized in that: The boom head of the crane is provided with an image acquisition device and a point cloud data acquisition device, and the fields of view of the image acquisition device and the point cloud data acquisition device cover the area below the boom head along the gravity direction. The method comprises: Acquire the single-frame point cloud data and the first image acquired simultaneously by the point cloud data acquisition device and the image acquisition device; Determine a pixel set of the load in the first image, and obtain a first point cloud set corresponding to the pixel set in the single frame point cloud data; Based on the position of the odometer in the world coordinate system, a grid map is determined, and based on the grid map and the first point cloud set, a dense point cloud is determined.

2. The dense mapping method according to claim 1, characterized in that: The step of determining a pixel set of the load in the first image and acquiring a first point cloud set corresponding to the pixel set in the single frame point cloud data includes: Performing filtering processing on the single frame point cloud data to obtain first point cloud data; Acquire an instance segmentation mask image of the payload based on the first image, and determine a pixel set of the instance segmentation mask image of the payload in the first image; A first point cloud set corresponding to the first point cloud data and the pixel set is determined.

3. The dense mapping method according to claim 2, characterized in that: The determining a first point cloud set corresponding to the first point cloud data and the pixel set includes: Acquire an intrinsic parameter matrix of the image acquisition device and an extrinsic parameter transformation matrix from the point cloud data acquisition device to the image acquisition device; Determine a first transformation matrix from the point cloud data acquisition device to the coordinate system corresponding to the image acquisition device according to the intrinsic parameter matrix and the extrinsic parameter transformation matrix; Based on the first transformation matrix, the first point cloud data is projected onto the plane where the first image is located, and a set of first point cloud data whose projection points fall into target pixels is determined as a first point cloud set; wherein the target pixel is any pixel in the pixel set.

4. The dense mapping method according to claim 2 or 3, characterized in that: The filtering process of the single frame point cloud data to obtain first point cloud data includes: Filter out low-intensity points, outlier points, and points outside the intersection of the effective fields of view of the point cloud data acquisition device and the image acquisition device in the single-frame point cloud data to obtain first point cloud data; Wherein, the low-intensity point is point cloud data whose point cloud intensity is lower than the intensity threshold; The outlier point is the nearest neighbor N nei_0 The point cloud data whose average distance between points is greater than λ0 times the standard deviation of the distance between points in the entire region.

5. The dense mapping method according to claim 1, characterized in that: Before determining the grid map based on the position of the odometer in the world coordinate system, and after obtaining the first point cloud set corresponding to the pixel set in the single frame point cloud data, the method further includes: Based on the inertial odometer, a transformation matrix from the point cloud data acquisition device to a map coordinate system is obtained; wherein the origin of the map coordinate system coincides with the initial point of the inertial odometer, and the z-axis points upward along the anti-gravity direction; Projecting the first point cloud set to the map coordinate system to obtain a projected point cloud set; Projecting the first point cloud set to a preset plane of the world coordinate system, extracting the convex hull domain of the projection point set, and stretching it upward to the preset plane with the convex hull as the base to construct a load envelope area; Points in the projected point cloud set that are within the load envelope area are filtered out to obtain a second point cloud set.

6. The dense mapping method according to claim 1, characterized in that: The step of determining a grid map based on the position of the odometer in the world coordinate system, and determining a dense point cloud according to the grid map and the first point cloud set, includes: Acquire a position vector of the inertial odometer in a world coordinate system, and acquire a target point cloud set centered on the position vector from the first point cloud set; Taking the position vector as the center, the spatial region is divided into a set of two-dimensional square grids parallel to the vertical coordinate; wherein each of the grids contains an elevation; A dense point cloud is generated according to the target point cloud set, the horizontal coordinate and the vertical coordinate of the center of each grid, and the mean grid elevation.

7. The dense mapping method according to claim 6, characterized in that: Generating a dense point cloud according to the target point cloud set, the horizontal coordinate and the vertical coordinate of the center of each grid, and the grid elevation mean, includes: Initializing the grid that first contains point cloud data according to the target point cloud set and each of the grids, and updating the grid that contains new point clouds; A dense point cloud is generated according to the abscissa and ordinate of the center of each initialization grid, the abscissa and ordinate of the center of each updated grid, and the mean grid elevation.

8. The dense mapping method according to claim 7, characterized in that: The grid includes a static grid and a dynamic grid; Initializing the grid containing point cloud data for the first time according to the target point cloud set and each of the grids, and updating the grid containing the new point cloud, comprises: Acquire an instance segmentation mask image of the payload based on the first image; If the target point cloud data is within the instance segmentation mask image of the payload, the corresponding static grid is updated, initialized or skipped according to the initialization of the static grid corresponding to the target point cloud data; wherein the target point cloud data is any point cloud data in the target point cloud set; If the target point cloud data is outside the instance segmentation mask image of the payload, initializing or updating the corresponding static grid according to the vertical coordinate of the target point cloud data and the elevation of the corresponding static grid; If the target point cloud data is outside the instance segmentation mask image of the payload, the corresponding dynamic grid is initialized or updated according to the vertical coordinate of the target point cloud data, the elevation of the corresponding static grid and the elevation of the corresponding dynamic grid.

9. A crane, characterized in that: include: An image acquisition device and a point cloud data acquisition device are arranged on the boom head of the crane, and the fields of view of the image acquisition device and the point cloud data acquisition device cover the area below the boom head along the gravity direction; as well as, A processor and a memory storing a computer program, when the processor runs the computer program, implements the dense mapping method as described in any one of claims 1 to 8 above.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the dense mapping method as described in any one of claims 1 to 8 is implemented.

Images