Point cloud data processing method, device, system, storage medium and product

Abstract

The present disclosure relates to a point cloud data processing method, device, system, storage medium and product, and relates to the technical field of warehousing. The point cloud data processing method comprises: acquiring point cloud data corresponding to a storage location on a shelf, wherein the point cloud data is obtained by scanning a first specified region of the storage location using a single-line laser radar; determining a feature line segment corresponding to the point cloud data based on a data feature of each data point in the point cloud data, the feature line segment being a line segment formed by adjacent data points with the same data feature in the point cloud data; and determining a state of the storage location based on a relative position of the feature line segment and the single-line laser radar.

Description

Technical Field

[0001] This disclosure relates to the field of warehousing technology, and in particular to a point cloud data processing method, apparatus, system, storage medium, and product. Background Technology

[0002] With the development of artificial intelligence technology, robots are now being used in the warehousing and logistics industry to complete the retrieval and storage of goods on shelves. Detecting the presence or absence of items in storage locations is crucial for robots to function effectively in these processes. Summary of the Invention

[0003] One of the technical problems that this disclosure aims to solve is: how to improve the accuracy of determining the state of the storage location.

[0004] According to some embodiments of this disclosure, a method for processing point cloud data is provided, including: acquiring point cloud data corresponding to storage locations on a shelf, wherein the point cloud data is obtained by scanning a first designated area of ​​the storage location using a single-line lidar; determining feature line segments corresponding to the point cloud data based on the data characteristics of each data point in the point cloud data, wherein the feature line segments are line segments formed by adjacent data points with the same data characteristics in the point cloud data; and determining the state of the storage location based on the relative position of the feature line segments and the single-line lidar.

[0005] In some embodiments, the data features include at least one of the following: the direction of the normal, corner points, and curvature.

[0006] In some embodiments, determining the feature line segment corresponding to the point cloud data based on the data features of each data point in the point cloud data includes: determining one or more feature intervals based on the data features of each data point in the point cloud data, wherein the feature interval includes multiple data points, the multiple data points are adjacent and the data features of the multiple data points are consistent; determining a target feature interval in one or more feature intervals according to the number of data points included in the one or more feature intervals; and determining the line segment formed by the data points included in the target feature interval as the feature line segment corresponding to the point cloud data.

[0007] In some embodiments, determining a target feature interval among one or more feature intervals based on the number of data points included in one or more feature intervals includes: determining the feature interval with the most data points among one or more feature intervals as the target feature interval.

[0008] In some embodiments, the data feature is the direction of the normal, and the processing method further includes: for each data point, determining the neighboring data points of the data point; and determining the direction of the normal of the data point based on the neighboring data points.

[0009] In some embodiments, the data points in the point cloud data are encoded in a specified order, and determining the neighboring data points of a data point includes: determining the neighboring data points of a data point based on a preset number of data points on both sides before and after the encoding of the data point.

[0010] In some embodiments, determining the neighboring data points of a data point based on a preset number of data points on both sides of the data point's encoding includes: when the number of data points on the first side of the data point's encoding is less than a preset number, selecting all the data points on the first side of the data point's encoding and a preset number of data points on the second side of the data point's encoding as the neighboring data points of the data point.

[0011] In some embodiments, determining the neighboring data points of a data point includes: using a nearest neighbor search algorithm to determine the neighboring data points of the data point.

[0012] In some embodiments, the state of the storage location includes no material bin, a material bin present and properly placed, or a material bin present and improperly placed.

[0013] In some embodiments, the relative position includes relative distance and relative angle. Determining the state of the storage location based on the relative position of the feature line segment and the single-line lidar includes: determining the relative distance from the feature line segment to the single-line lidar; determining the relative angle between the feature line segment and the scanning direction of the single-line lidar; and determining the state of the storage location based on the relative distance and the relative angle.

[0014] In some embodiments, determining the state of a storage location based on relative distance and relative angle includes: when the relative distance is within a first specified range, determining that the storage location is in the presence of a material bin.

[0015] In some embodiments, determining the state of the storage location based on relative distance and relative angle further includes: when the relative distance is within a first specified range and the relative angle is within a second specified range, determining that the storage location has a material box and that the presence of the material box indicates that the placement is qualified.

[0016] In some embodiments, determining the state of the storage location based on relative distance and relative angle further includes: if the relative distance is within a first specified range and the angle is not within a second specified range, determining that the storage location has a material box and that the presence of the material box indicates improper placement.

[0017] In some embodiments, determining the state of the storage location based on relative distance and relative angle includes: if the relative distance is not within a first specified range, determining that the storage location is in a state where there is no hopper.

[0018] In some embodiments, the aforementioned processing method further includes at least one of the following: when the storage location is in the state of having a material box and the business scenario is storing a material box, not storing a material box in the storage location; when the storage location is in the state of having a material box and the business scenario is retrieving a material box, retrieving a material box from the storage location; when the storage location is in the state of having a material box but the placement is not up to standard and the business scenario is storing a material box, rearranging the material box; when the storage location is in the state of not having a material box and the business scenario is storing a material box, storing a material box in the storage location; or when the storage location is in the state of not having a material box and the business scenario is retrieving a material box, not retrieving a material box from the storage location.

[0019] In some embodiments, obtaining point cloud data corresponding to storage locations on a shelf includes: clustering point cloud data scanned by a single-line lidar to obtain multiple sets of point cloud data; determining the distance from each set of point cloud data to the single-line lidar based on the center of each set of point cloud data; determining a target set of point cloud data among the multiple sets of point cloud data based on the distance from each set of point cloud data to the single-line lidar, and identifying the target set of point cloud data as the point cloud data corresponding to the storage location.

[0020] In some embodiments, obtaining point cloud data corresponding to storage locations on the shelf further includes: determining point cloud data within a second specified area from the initial point cloud data based on the position of a single-line lidar, and re-determining the point cloud data within the second specified area as the initial point cloud data, wherein the distance between the position corresponding to the point cloud data within the second specified area and the single-line lidar does not exceed a specified distance.

[0021] In some embodiments, the first designated area is determined based on the size of the bin.

[0022] In some embodiments, a single-line lidar is mounted on a storage and retrieval device, which is used to store bins in a storage location and retrieve at least one of the bins in the storage location.

[0023] According to a second aspect of some embodiments of this disclosure, a point cloud data processing apparatus is provided, comprising: an acquisition module configured to acquire point cloud data corresponding to a storage location on a shelf, wherein the point cloud data is obtained by scanning a first designated area of ​​the storage location using a single-line lidar; a first determination module configured to determine a feature line segment corresponding to the point cloud data based on the data characteristics of each data point in the point cloud data, wherein the feature line segment is a line segment formed by adjacent data points having the same data characteristics in the point cloud data; and a second determination module configured to determine the state of the storage location based on the relative position of the feature line segment and the single-line lidar.

[0024] According to a third aspect of some embodiments of the present disclosure, a point cloud data processing apparatus is provided, comprising: a processor; and a memory coupled to the processor for storing instructions, wherein when the instructions are executed by the processor, the processor performs any of the aforementioned storage state processing methods.

[0025] According to a fourth aspect of some embodiments of the present disclosure, a point cloud data processing system is provided, comprising: any of the aforementioned point cloud data processing devices; and a single-line lidar configured to scan a first designated area of ​​a storage location.

[0026] In some embodiments, the processing system further includes: an access device equipped with a single-line lidar and configured to store a bin at a storage location and retrieve at least one of the following: storing a bin at a storage location and retrieving a bin from a storage location.

[0027] According to a fifth aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided having computer instructions stored thereon, wherein the instructions, when executed by a processor, implement any of the aforementioned point cloud data processing methods.

[0028] According to a sixth aspect of some embodiments of the present disclosure, a computer program product is provided, including instructions that, when executed by a processor, cause the processor to perform any of the aforementioned point cloud data processing methods.

[0029] This disclosure identifies characteristic line segments in the point cloud data corresponding to the storage location by performing feature analysis. These characteristic line segments are line segments formed by adjacent data points with the same data characteristics in the point cloud data. Using characteristic line segments to determine the location information of the point cloud data can reduce the influence of noisy data points. Therefore, based on the relative position of the characteristic line segments and the single-line lidar, the accuracy of determining the storage location's status can be improved.

[0030] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a method for processing point cloud data according to some embodiments of the present disclosure is shown.

[0033] Figure 2A schematic diagram is shown illustrating the determination of point cloud data corresponding to storage locations according to some embodiments of the present disclosure.

[0034] Figure 3 A schematic diagram of pass-through filtering of initial point cloud data according to some embodiments of the present disclosure is shown.

[0035] Figure 4 A schematic diagram illustrating the determination of data characteristics for each data point of point cloud data according to some embodiments of the present disclosure is shown.

[0036] Figure 5 A flowchart illustrating a method for processing point cloud data according to other embodiments of the present disclosure is shown.

[0037] Figure 6 A schematic diagram of a point cloud data processing apparatus according to some embodiments of the present disclosure is shown.

[0038] Figure 7 A schematic diagram of a point cloud data processing apparatus according to other embodiments of the present disclosure is shown.

[0039] Figure 8 A schematic diagram of the structure of a point cloud data processing apparatus according to some embodiments of the present disclosure is shown.

[0040] Figure 9 A schematic diagram of the structure of a point cloud data processing system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0041] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0042] In related technologies, a single-point laser is used to detect the presence or absence of a material hopper at a storage location. The presence of a hopper is confirmed when the distance measured by the single-point laser is within a certain range. However, the single-point distance determined by the single-point laser is prone to error and cannot determine whether the material hopper is tilted, thus failing to ascertain the accurate state of the storage location.

[0043] Based on this, this disclosure provides a method for processing point cloud data. Figure 1 A flowchart illustrating a method for processing point cloud data according to some embodiments of the present disclosure is shown. Figure 1As shown, the method of this embodiment includes steps S102 to S106.

[0044] In step S102, point cloud data corresponding to the storage location on the shelf is obtained. The point cloud data is obtained by scanning the first designated area of ​​the storage location using a single-line lidar.

[0045] The first designated area is determined based on the dimensions of the hopper, specifically its height range within the storage location. A single-line lidar is used to scan the area within the corresponding height range of the hopper in the storage location, obtaining point cloud data, which is two-dimensional planar data.

[0046] In some embodiments, a single-line lidar is mounted on a storage and retrieval device for storing bins at storage locations and retrieving at least one of the bins at storage locations. The storage and retrieval device may be, for example, a robot that uses lifting and walking devices to perform inbound or outbound tasks at various storage locations within the shelving.

[0047] Since the point cloud data from a single-line lidar scan is two-dimensional planar data, it may include point cloud data from other storage locations or nearby areas. Therefore, after obtaining the point cloud data from the single-line lidar scan, further filtering is required to obtain the point cloud data corresponding to the storage location.

[0048] In some embodiments, obtaining point cloud data corresponding to storage locations on a shelf includes: clustering point cloud data scanned by a single-line lidar to obtain multiple sets of point cloud data; determining the distance from each set of point cloud data to the single-line lidar based on the center of each set of point cloud data; determining a target set of point cloud data among the multiple sets of point cloud data based on the distance from each set of point cloud data to the single-line lidar, and identifying the target set of point cloud data as the point cloud data corresponding to the storage location.

[0049] Single-line lidar scans the storage site from a location that is directly facing and inclined towards it. Therefore, in the point cloud data scanned by single-line lidar, the point cloud data corresponding to the storage site has the closest relative distance to the single-line lidar compared to point cloud data from other areas. Thus, clustering results based on the point cloud data scanned by single-line lidar can quickly and accurately determine the point cloud data corresponding to the storage site, reducing interference from other storage sites or other areas on detection accuracy.

[0050] Figure 2 A schematic diagram illustrating the determination of point cloud data corresponding to storage locations according to some embodiments of this disclosure is shown. For example... Figure 2 As shown, a single-line LiDAR is located between two rows of storage positions, assuming four material bins are placed in the two rows. A coordinate system can be established with the position of the single-line LiDAR as the origin (or the robot's position as the origin when the single-line LiDAR is mounted on the robot), and the orientation as the Y-axis, as shown below. Figure 2 As shown. That is, the positive Y-axis points in front of the single-line lidar, and the negative Y-axis points behind the single-line lidar.

[0051] A single-line LiDAR scans storage location A to obtain initial point cloud data. After clustering the initial point cloud data, four sets of point cloud data (21, 22, 23, and 24) are obtained. Further, based on the distance from the center of these four sets of point cloud data to the single-line LiDAR, and since the cluster selection is based on proximity to the robot (because the robot typically aligns with the storage location to be inspected on the shelf beforehand), the point cloud data corresponding to set 22 is taken as the point cloud data corresponding to storage location A. The center of each set of point cloud data is, for example, the average coordinates of the data points in that set.

[0052] To reduce the workload of processing point cloud data, the initial point cloud data can be preliminarily screened before clustering to delineate the area where the storage location is located. For example, the point cloud data on both sides of the single-line lidar can be used as the point cloud data for clustering, that is, the initial point cloud data can be redefined.

[0053] Alternatively, the storage location area can be determined from the initial point cloud data using a pass-through filtering method. Figure 3 A schematic diagram illustrating pass-through filtering of initial point cloud data according to some embodiments of the present disclosure is shown. For example... Figure 3 As shown, a coordinate system is established with the location of the single-line lidar as the origin and its orientation as the Y-axis. The figure illustrates some data points. The filtering range is then determined by setting the coordinates (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4). Figure 3 As shown, it can filter out the point cloud data of the two rectangular areas on both sides of the single-line lidar and re-determine the filtered point cloud data as the initial point cloud data.

[0054] In some embodiments, obtaining point cloud data corresponding to storage locations on the shelf further includes: determining point cloud data within a second specified area from the initial point cloud data based on the position of a single-line lidar, and re-determining the point cloud data within the second specified area as the initial point cloud data, wherein the distance between the position corresponding to the point cloud data within the second specified area and the single-line lidar does not exceed a specified distance.

[0055] Single-line lidar can be used to obtain two-dimensional planar data of the storage location. Compared with single-point data, two-dimensional planar data has more features. It can not only determine the presence or absence of material bins in the storage location, but also further determine the status of the material bins in the storage location, such as whether the placement is qualified. This will be described in the following steps.

[0056] In step S104, based on the data characteristics of each data point in the point cloud data, the feature line segment corresponding to the point cloud data is determined. The feature line segment is the line segment formed by adjacent data points with the same data characteristics in the point cloud data.

[0057] After determining the point cloud data corresponding to the storage location, feature analysis is performed on the data points included in the point cloud data, and the feature line segments corresponding to the point cloud data are determined based on the data characteristics of each data point in the point cloud data.

[0058] In some embodiments, determining the feature line segment corresponding to the point cloud data based on the data features of each data point in the point cloud data includes: determining one or more feature intervals based on the data features of each data point in the point cloud data, wherein the feature interval includes multiple data points, the multiple data points are adjacent and the data features of the multiple data points are consistent; determining a target feature interval in one or more feature intervals according to the number of data points included in the one or more feature intervals; and determining the line segment formed by the data points included in the target feature interval as the feature line segment corresponding to the point cloud data.

[0059] In some embodiments, determining a target feature interval among one or more feature intervals based on the number of data points included in one or more feature intervals includes: determining the feature interval with the most data points among one or more feature intervals as the target feature interval.

[0060] Feature segments reflect the largest consistent feature interval in point cloud data, thus enabling them to characterize features such as location information. Characterizing the location information of point cloud data using feature segments can remove the influence of noisy data points, such as those caused by uneven surfaces (e.g., sharp edges) in the hopper. Therefore, feature segments can accurately characterize the location information of the hopper.

[0061] Alternatively, depending on the actual shape of the hopper, the method of determining the feature line segment is not limited to the feature interval with the most data points mentioned above. You can also select the feature interval with the most data points and the second most data points, and use the line segments formed by the data points in these two feature intervals to generate the feature line segment corresponding to the point cloud data. For example, take the line segment located in the middle position of the two line segments as the feature line segment.

[0062] Featured line segments can reduce the impact of noisy data points in point cloud data on the subsequent determination of the storage location status. That is, featured line segments can provide high distance accuracy, referring to the distance accuracy from the bin to the storage device (e.g., a robot). In some embodiments, the distance accuracy can reach 2.5 mm, meaning the error between the robot's monitored distance to the bin in the storage location and the actual distance is within ±2.5 mm. This allows the robot to accurately determine whether the outbound and inbound operations have been completed correctly.

[0063] Furthermore, point cloud data has low computational complexity and high efficiency. For example, the number of data points in point cloud data is denoted as... (Approximately 1200), therefore, the maximum complexity of the point cloud data processing method disclosed herein is... This refers to the complexity during point cloud clustering. In other processing steps, the complexity is less than or equal to... Therefore, the time to compute a frame on a typical embedded platform (such as RK3568) is less than 100 milliseconds, which can provide real-time computing performance for the access device, helping the access device to complete outbound or inbound tasks quickly and accurately.

[0064] This disclosure does not limit the specific type of data features used to determine feature segments. In some embodiments, data features include at least one of the direction of the normal, corner points, and curvature.

[0065] When the data feature is the direction of the normal, the above processing method further includes: for each data point, determining the neighboring data points of the data point; and based on the neighboring data points, determining the direction of the normal of the data point.

[0066] Point cloud data can be ordered or unordered. When the point cloud data is ordered, the data points in the point cloud data can be encoded in a specified order, and neighboring data points can be determined based on the encoding of the data points.

[0067] In some embodiments, the data points in the point cloud data are encoded in a specified order, and determining the neighboring data points of a data point includes: determining the neighboring data points of a data point based on a preset number of data points on both sides before and after the encoding of the data point.

[0068] For example, for a data point coded as i, select N points before and after it, that is, select the data points with the encoding range of [iN, i-1] and [i+1, i+N] as the neighboring data points of the data point coded as i.

[0069] Determining the neighboring data points of a data point based on a preset number of data points on both sides of the data point's encoding includes: when the number of data points on the first side of the data point's encoding is less than the preset number, selecting all the data points on the first side of the data point's encoding and a preset number of data points on the second side of the data point's encoding as the neighboring data points of the data point.

[0070] In some embodiments, determining the neighboring data points of a data point includes using a nearest neighbor search algorithm. That is, for unordered point cloud data, various nearest neighbor search algorithms (such as the kd-tree algorithm) can be used to determine the neighboring data points of each data point. Of course, ordered point cloud data can also use nearest neighbor search algorithms to determine the neighboring data points of each data point.

[0071] After identifying the neighboring data points of each data point, principal component analysis or least squares method is used to fit the data points and their neighbors to determine the data characteristics of each data point. When the data characteristic is the direction of the normal, the direction of the normal for each data point is standardized; for example, the direction of the normal for each data point is set to follow the direction of the normal. Figure 2 The positive Y-axis in the diagram.

[0072] Figure 4 A schematic diagram illustrating the determination of data characteristics for each data point of point cloud data according to some embodiments of the present disclosure is shown. Figure 4 The point cloud data and the orientation of its normals are shown. Figure 4 In line segment k, the directions of the normals to the data points are consistent, therefore Figure 4 The feature line segment in the point cloud data is k. Data points other than feature line segment k correspond to points on the uneven surface of the material box, such as edges or various decorative patterns. In practical applications, the material box surface generally corresponds to a single line segment, such as... Figure 4 As shown, if the feature line segment k is not used to characterize the surface of the bin, there will be a large error in determining the distance between the robot and the bin, which can reach the centimeter level. This does not meet the requirements of the application scenario because the uneven points on the surface of the bin will reduce the accuracy of the calculation results.

[0073] By performing feature analysis on the data points in the point cloud data to determine the feature line segments, it is possible to identify data points that can characterize the features of the point cloud data. In other words, the feature line segments are data points after removing noisy data points from the point cloud data. This results in higher accuracy in determining the state of the storage location based on the feature line segments.

[0074] In step S106, the state of the storage location is determined based on the relative position of the feature line segment and the single-line lidar.

[0075] Relative position includes relative distance and relative angle. Relative distance is used to determine whether there is a material bin in the storage location, while relative angle is used to determine whether the material bins in the storage location are placed correctly.

[0076] In some embodiments, determining the state of the storage location based on the relative position of the feature line segment and the single-line lidar includes: determining the relative distance from the feature line segment to the single-line lidar; determining the relative angle between the feature line segment and the scanning direction of the single-line lidar; and determining the state of the storage location based on the relative distance and the relative angle.

[0077] The status of the storage location includes no material bin, material bin present and properly placed, or material bin present and improperly placed.

[0078] In some embodiments, determining the state of a storage location based on relative distance and relative angle includes: when the relative distance is within a first specified range, determining that the storage location is in the presence of a material bin.

[0079] Determining the status of a storage location based on relative distance and relative angle also includes: when the relative distance is within a first specified range and the relative angle is within a second specified range, determining that the storage location has a material box and that the material box is properly placed.

[0080] Determining the storage location status based on relative distance and relative angle also includes: if the relative distance is within a first specified range and the angle is not within a second specified range, determining the storage location status as having a material bin, and further specifying that the presence of a material bin indicates improper placement. In this case, improper placement means that the material bin is placed in a crooked position.

[0081] Based on relative distance and relative angle, the state of the storage location is determined as follows: if the relative distance is not within the first specified range, the state of the storage location is determined to be that there is no material box.

[0082] The scanning direction of a single-line lidar is, for example, Figure 2 The Y-axis direction in the diagram. Alternatively, it can be determined by the relationship between the feature line segment and... Figure 2 The relative angle along the X-axis is used to determine whether the storage bins are properly placed based on whether this relative angle falls within the third specified range.

[0083] In some embodiments, when the relative distance is negative, in the business scenario of retrieving the material box, it can be set that there is no material box in the storage location, or in the business scenario of storing the material box, the state of the storage location is determined to be a failure to place the box, such as the box falling midway.

[0084] If the relative distance is not negative but not within the first specified range, in the business scenario of retrieving the material box, it is determined that there is no material box in the storage location, or in the business scenario of storing the material box, it is determined that the storage location is in a state of unqualified placement and needs to be rearranged.

[0085] In some embodiments, the default values ​​for relative distance and relative angle can be set to negative numbers, representing no result.

[0086] By adding a check to determine whether the relative distance is negative, abnormal situations can be detected quickly, facilitating emergency response.

[0087] Based on the characteristic line segments, it can be determined whether there are material bins in the storage location, and if so, whether they are placed correctly, so that the material bins can be rearranged if they are not placed correctly.

[0088] In some embodiments, the above processing method further includes at least one of the following: when the storage location is in the state of having a material box and the business scenario is storing a material box, not storing a material box in the storage location, for example, the robot searches for a new storage location; when the storage location is in the state of having a material box and the business scenario is retrieving a material box, retrieving the material box from the storage location; when the storage location is in the state of having a material box but the placement is not up to standard and the business scenario is storing a material box, rearranging the material box; when the storage location is in the state of not having a material box and the business scenario is storing a material box, storing a material box in the storage location; or when the storage location is in the state of not having a material box and the business scenario is retrieving a material box, not retrieving the material box from the storage location.

[0089] The point cloud processing method disclosed herein is applicable to scenarios involving bin inbound and outbound operations, facilitating shelf organization and improving warehouse consistency. When using a robot to perform bin inbound or outbound tasks, after the robot reaches any storage location on a high-level shelf, the data scanned by a single-line LiDAR can be used to calculate the distance from the robot's center to the bin plane, as well as the bin's tilt angle within the storage location. This solves the problem of the robot detecting distance and angle deviations of bins in storage locations.

[0090] Figure 5 A flowchart illustrating a method for processing point cloud data according to other embodiments of this disclosure is shown. Figure 5 As shown, the processing method of this embodiment includes steps S502 to S506.

[0091] In step S502, the initial point cloud data obtained by single-line lidar scanning is preprocessed, including pass-through filtering, point cloud clustering, and point cloud filtering.

[0092] Direct filtering involves determining point cloud data within a second specified region from the initial point cloud data, and then re-determining the point cloud data within the second specified region as the initial point cloud data. Point cloud clustering involves clustering the initial point cloud data to obtain multiple sets of point cloud data. Point cloud filtering involves selecting the point cloud data corresponding to the storage location based on the distance from each set of point cloud data to the single-line lidar.

[0093] In step S504, the distance and angle of the material box are calculated.

[0094] Based on feature analysis of data points in the point cloud data corresponding to the storage location, characteristic line segments corresponding to the point cloud data are determined. Based on the relative position of the characteristic line segments to the single-line lidar, it is determined whether there is a material box at the storage location, and if there is a material box at the storage location, the relative distance and relative angle between the material box and the single-line lidar are determined.

[0095] In step S506, the status of the storage location is determined.

[0096] The status of the storage location is determined based on the relative position of the feature line segment and the single-line lidar. The status of the storage location includes any one of the following: no material bin, material bin present and properly placed, or material bin present and improperly placed.

[0097] This disclosure identifies characteristic line segments in the point cloud data corresponding to the storage location by performing feature analysis. These characteristic line segments are line segments formed by adjacent data points with the same data characteristics in the point cloud data. Using characteristic line segments to determine the location information of the point cloud data can reduce the influence of noisy data points. Therefore, based on the relative position of the characteristic line segments and the single-line lidar, the accuracy of determining the storage location's status can be improved.

[0098] Figure 6 A schematic diagram of a point cloud data processing apparatus according to some embodiments of the present disclosure is shown. Figure 6 As shown, the point cloud data processing device 40 in this embodiment includes modules 610-630.

[0099] The acquisition module 610 is configured to acquire point cloud data corresponding to the storage location on the shelf, wherein the point cloud data is obtained by scanning a first designated area of ​​the storage location using a single-line lidar; the first determination module 620 is configured to determine the feature line segment corresponding to the point cloud data based on the data characteristics of each data point in the point cloud data, wherein the feature line segment is a line segment formed by adjacent data points with the same data characteristics in the point cloud data; the second determination module 630 is configured to determine the status of the storage location based on the relative position of the feature line segment and the single-line lidar.

[0100] In some embodiments, the data features include at least one of the following: the direction of the normal, corner points, and curvature.

[0101] In some embodiments, the first determining module 620 is configured to determine one or more feature intervals based on the data characteristics of each data point in the point cloud data, wherein the feature interval includes multiple data points, the multiple data points are adjacent and the data characteristics of the multiple data points are consistent; determine a target feature interval in one or more feature intervals according to the number of data points included in the one or more feature intervals; and determine the line segment formed by the data points included in the target feature interval as the feature line segment corresponding to the point cloud data.

[0102] In some embodiments, the first determining module 620 is configured to determine the feature interval with the most data points among one or more feature intervals as the target feature interval.

[0103] In some embodiments, the data feature is the direction of the normal, and the point cloud data processing device 40 is further configured to determine the neighboring data points of each data point; and determine the direction of the normal of the data point based on the neighboring data points.

[0104] In some embodiments, the data points in the point cloud data are encoded in a specified order, and the point cloud data processing device 60 is further configured to determine the neighboring data points of the data points based on a preset number of data points on both sides before and after the encoding of the data points.

[0105] In some embodiments, the point cloud data processing apparatus 60 is further configured to select all data points on the first side of the data point's encoding and a preset number of data points on the second side of the data point's encoding as neighboring data points when the number of data points on the first side of the data point's encoding is less than a preset number.

[0106] In some embodiments, the point cloud data processing apparatus 60 is further configured to determine the neighboring data points of the data points using a nearest neighbor search algorithm.

[0107] In some embodiments, the state of the storage location includes no material bin, a material bin present and properly placed, or a material bin present and improperly placed.

[0108] In some embodiments, the second determining module 630 is configured to determine the relative distance between the feature line segment and the single-line lidar; determine the relative angle between the feature line segment and the scanning direction of the single-line lidar; and determine the state of the storage location based on the relative distance and the relative angle.

[0109] In some embodiments, the second determining module 630 is configured to determine that the storage location is in the presence of a hopper when the relative distance is within a first specified range.

[0110] In some embodiments, when the relative distance is within a first specified range and the relative angle is within a second specified range, the state of the storage location is determined to be that a material box exists, and the state of the material box being present is that the placement is qualified.

[0111] In some embodiments, when the relative distance is within a first specified range and the angle is not within a second specified range, the state of the storage location is determined to be that a material box exists, and the state of the material box is that the placement is unqualified.

[0112] In some embodiments, if the relative distance is not within a first specified range, the state of the storage location is determined to be that there is no hopper.

[0113] In some embodiments, the point cloud data processing device 60 is further configured to: not store a material box in the storage location when the storage location is in a state where a material box exists and the business scenario is storing a material box; remove a material box from the storage location when the storage location is in a state where a material box exists and the business scenario is retrieving a material box; rearrange the material box when the storage location is in a state where a material box exists but is not properly placed and the business scenario is storing a material box; store a material box in the storage location when the storage location is in a state where a material box does not exist and the business scenario is storing a material box; or not retrieve a material box from the storage location when the storage location is in a state where a material box does not exist and the business scenario is retrieving a material box.

[0114] In some embodiments, the acquisition module 610 is configured to cluster the point cloud data scanned by a single-line lidar to obtain multiple sets of point cloud data; determine the distance from each set of point cloud data to the single-line lidar based on the center of each set of point cloud data; and determine the target set of point cloud data in the multiple sets of point cloud data based on the distance from each set of point cloud data to the single-line lidar, and identify the target set of point cloud data as the point cloud data corresponding to the storage location.

[0115] In some embodiments, the acquisition module 610 is configured to determine point cloud data within a second specified area from the initial point cloud data based on the position of the single-line lidar, and to re-determine the point cloud data within the second specified area as the initial point cloud data, wherein the distance between the position corresponding to the point cloud data within the second specified area and the single-line lidar does not exceed a specified distance.

[0116] In some embodiments, the first designated area is determined based on the size of the bin.

[0117] In some embodiments, a single-line lidar is mounted on a storage and retrieval device, which is used to store bins in a storage location and retrieve at least one of the bins in the storage location.

[0118] The point cloud data processing device disclosed herein analyzes the features of the point cloud data corresponding to the storage location to determine the feature line segments corresponding to the point cloud data. These feature line segments are line segments formed by adjacent data points with the same data features in the point cloud data. Using feature line segments to determine the location information of the point cloud data can reduce the influence of noisy data points in the point cloud data. Therefore, based on the relative position of the feature line segments and the single-line lidar, the accuracy of determining the state of the storage location can be improved.

[0119] The point cloud data processing apparatus in the embodiments of this disclosure can be implemented by various computing devices or computer systems, as described below. Figure 7 as well as Figure 8 Describe it.

[0120] Figure 7A schematic diagram of a point cloud data processing apparatus according to other embodiments of the present disclosure is shown. For example... Figure 7 As shown, the apparatus 70 of this embodiment includes a memory 710 and a processor 720 coupled to the memory 710. The processor 720 is configured to execute a point cloud data processing method in any of the embodiments of this disclosure based on instructions stored in the memory 710.

[0121] The memory 710 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, a database, and other programs.

[0122] Figure 8 A schematic diagram of a point cloud data processing apparatus according to some embodiments of the present disclosure is shown. Figure 8 As shown, the device 80 in this embodiment includes a memory 810 and a processor 820, which are similar to the memory 710 and processor 720, respectively. It may also include an input / output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850, and the memory 810 and processor 820 can be connected, for example, via a bus 860. The input / output interface 830 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 840 provides a connection interface for various networked devices, such as connecting to a database server or cloud storage server. The storage interface 850 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0123] Figure 9 A schematic diagram of the structure of a point cloud data processing system according to some embodiments of the present disclosure is shown. Figure 9 As shown, the processing system 90 of this embodiment includes a point cloud data processing device 910 as described above; and a single-line lidar 920 configured to scan a first designated area of ​​the storage location.

[0124] In some embodiments, the processing system 90 further includes an access device equipped with a single-line lidar and configured to store bins at a storage location and retrieve at least one of the following: storing bins at a storage location and retrieving bins from a storage location.

[0125] Embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements any of the aforementioned point cloud data processing methods.

[0126] Embodiments of this disclosure also provide a computer program product, including instructions that, when executed by a processor, cause the processor to perform a processing method for any of the aforementioned point cloud data.

[0127] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] The above description is only a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for processing point cloud data, comprising: Obtain point cloud data corresponding to the storage location on the shelf, wherein the point cloud data is obtained by scanning a first designated area of ​​the storage location using a single-line lidar. Based on the data characteristics of each data point in the point cloud data, the feature line segment corresponding to the point cloud data is determined. The feature line segment is a line segment formed by adjacent data points with the same data characteristics in the point cloud data. The state of the storage location is determined based on the relative position of the feature line segment and the single-line lidar.

2. The processing method according to claim 1, wherein, The data features include at least one of the following: direction of normal, corner point, and curvature.

3. The processing method according to claim 1, wherein, The step of determining the feature line segment corresponding to the point cloud data based on the data features of each data point in the point cloud data includes: Based on the data characteristics of each data point in the point cloud data, one or more feature intervals are determined, wherein the feature intervals include multiple data points, the multiple data points are adjacent, and the data characteristics of the multiple data points are consistent; The target feature interval is determined based on the number of data points included in the one or more feature intervals. The line segments formed by the data points included in the target feature interval are determined as the feature line segments corresponding to the point cloud data.

4. The processing method according to claim 3, wherein, Determining the target feature interval among the one or more feature intervals based on the number of data points included in the one or more feature intervals includes: The feature interval with the most data points among the one or more feature intervals is determined as the target feature interval.

5. The processing method according to claim 1, wherein, The data feature is the direction of the normal, and the processing method further includes: For each data point, determine the neighboring data points of that data point; Based on the neighboring data points, the direction of the normal to the data points is determined.

6. The processing method according to claim 5, wherein, The data points in the point cloud data are encoded in a specified order, and determining the neighboring data points of the data points includes: Based on a preset number of data points on both sides of the encoding of the data point, the neighboring data points of the data point are determined.

7. The processing method according to claim 6, wherein, The step of determining the neighboring data points of the data point based on a preset number of data points before and after the encoding of the data point includes: If the number of data points on the first side of the encoding of a data point is less than the preset number, all data points on the first side of the encoding of the data point and a preset number of data points on the second side of the encoding of the data point are selected as the neighboring data points of the data point.

8. The processing method according to claim 5, wherein, The step of determining the neighboring data points of the data point includes: The nearest neighbor search algorithm is used to determine the neighboring data points of the data point.

9. The processing method according to claim 1, wherein, The status of the storage location includes no material bin, a material bin present and properly placed, or a material bin present and improperly placed.

10. The processing method according to claim 9, wherein, The relative position includes relative distance and relative angle, and determining the state of the storage location based on the relative position of the feature line segment and the single-line lidar includes: Determine the relative distance from the feature line segment to the single-line lidar; Determine the relative angle between the feature line segment and the scanning direction of the single-line lidar; The state of the storage location is determined based on the relative distance and the relative angle.

11. The processing method according to claim 10, wherein, Determining the state of the storage location based on the relative distance and the relative angle includes: When the relative distance is within a first specified range, the state of the storage location is determined to be that a material bin is present.

12. The processing method according to claim 11, wherein, Determining the state of the storage location based on the relative distance and the relative angle further includes: When the relative distance is within the first specified range and the relative angle is within the second specified range, the storage location is determined to be in a state where a material box exists, and the state where the material box exists is that it is placed correctly.

13. The processing method according to claim 11, wherein, Determining the state of the storage location based on the relative distance and the relative angle further includes: If the relative distance is within the first specified range and the angle is not within the second specified range, the storage location is determined to be in a state where a material box exists, and the state where the material box exists is that the placement is unqualified.

14. The processing method according to claim 10, wherein, Determining the state of the storage location based on the relative distance and the relative angle includes: If the relative distance is not within the first specified range, the storage location is determined to be in a state where no hopper exists.

15. The processing method according to claim 9, further comprising at least one of the following: If the storage location is in the presence of a material bin and the business scenario is storing the material bin, then no material bin will be stored in the storage location. If the storage location is in the state of having a material box and the business scenario is to remove the material box, then remove the material box from the storage location. If the storage location is in a state where there are material boxes but they are not placed properly, and the business scenario is storing material boxes, then the material boxes shall be rearranged. If the storage location is empty of material bins and the business scenario involves storing material bins, then a material bin is stored in the storage location; or If the storage location is empty of material bins and the business scenario involves retrieving material bins, the material bins will not be retrieved from the storage location.

16. The processing method according to claim 1, wherein, The acquisition of point cloud data corresponding to the storage locations on the shelf includes: Cluster the point cloud data scanned by the single-line lidar to obtain multiple sets of point cloud data; Based on the center of each set of point cloud data in the multiple sets of point cloud data, the distance from each set of point cloud data to the single-line lidar is determined; Based on the distance from each set of point cloud data to the single-line lidar, the target set of point cloud data in the multiple sets of point cloud data is determined, and the target set of point cloud data is identified as the point cloud data corresponding to the storage location.

17. The processing method according to claim 16, wherein, The process of obtaining the point cloud data corresponding to the storage locations on the shelf also includes: Based on the position of the single-line lidar, point cloud data within a second designated area is determined from the initial point cloud data, and the point cloud data within the second designated area is redefined as the initial point cloud data. The distance between the position corresponding to the point cloud data within the second designated area and the single-line lidar does not exceed a designated distance.

18. The processing method according to any one of claims 1-17, wherein, The first designated area is determined based on the size of the hopper.

19. The processing method according to any one of claims 1-17, wherein, The single-line lidar is installed on the storage and retrieval device, which is used to store at least one of the material bins in the storage location and retrieve at least one of the material bins in the storage location.

20. A point cloud data processing apparatus, comprising: The acquisition module is configured to acquire point cloud data corresponding to the storage location on the shelf, wherein the point cloud data is obtained by scanning a first designated area of ​​the storage location using a single-line lidar. The first determining module is configured to determine the feature line segment corresponding to the point cloud data based on the data features of each data point in the point cloud data. The feature line segment is a line segment formed by adjacent data points with the same data features in the point cloud data. The second determining module is configured to determine the state of the storage location based on the relative position of the feature line segment and the single-line lidar.

21. A point cloud data processing apparatus, comprising: processor; as well as A memory coupled to the processor is used to store instructions that, when executed by the processor, cause the processor to perform a storage state processing method as described in any one of claims 1-19.

22. A point cloud data processing system, comprising: The point cloud data processing apparatus as described in claim 20 or 21; A single-line lidar is configured to scan a first designated area of ​​the storage location.

23. The processing system according to claim 22, further comprising: The storage and retrieval device, equipped with the single-line lidar, is configured to store a bin at the storage location and retrieve at least one of the following:

24. A computer-readable storage medium having computer instructions stored thereon, wherein, When executed by the processor, this instruction implements the method for processing point cloud data as described in any one of claims 1-19.

25. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform a method for processing point cloud data as described in any one of claims 1-19.

Images