Point cloud data processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a point cloud data processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a plurality of obstacle point clouds, and extracting wall point clouds from the plurality of obstacle point clouds; projecting the obstacle point clouds except the wall point clouds in the plurality of obstacle point clouds to the ground to obtain a first point cloud vector; projecting the wall surface point cloud to the ground, and performing linear fitting on the projected wall surface point cloud to obtain a straight line; screening the projected wall point cloud based on the straight line to obtain a second point cloud vector; and combining the first point cloud vector and the second point cloud vector into edge semantic information. According to the method and the device, more accurate edge semantic information can be obtained, so that the robot has better obstacle avoidance capability.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a point cloud data processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] Cleaning robots (such as sweeping robots) and some other types of robots may need to strictly move along the edge (wall). In related technologies, the point cloud projected by the laser radar on the wall is usually used as the edge semantics to plan and control the robot to perform edge actions. However, due to the insufficient three-dimensional perception capability of the laser radar, the accuracy of the acquired edge semantic information is low. Summary of the invention

[0003] The embodiments of the present application provide a point cloud data processing method, device, electronic device, computer-readable storage medium and computer program product, which can obtain more accurate edge semantic information, thereby enabling the robot to have better obstacle avoidance capabilities.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a point cloud data processing method, including:

[0006] Acquire a plurality of obstacle point clouds, and extract a wall point cloud from the plurality of obstacle point clouds;

[0007] Projecting obstacle point clouds other than the wall point cloud among the plurality of obstacle point clouds onto the ground to obtain a first point cloud vector;

[0008] Projecting the wall point cloud onto the ground, and performing linear fitting on the projected wall point cloud to obtain a straight line;

[0009] Filtering the projected wall point cloud based on the straight line to obtain a second point cloud vector;

[0010] The first point cloud vector and the second point cloud vector are combined into edge semantic information.

[0011] The present application provides a point cloud data processing device, including:

[0012] An extraction module, used to obtain a plurality of obstacle point clouds and extract a wall point cloud from the plurality of obstacle point clouds;

[0013] A first processing module, configured to project obstacle point clouds other than the wall point cloud among the plurality of obstacle point clouds onto the ground to obtain a first point cloud vector;

[0014] A fitting module, used for projecting the wall point cloud onto the ground, and performing linear fitting on the projected wall point cloud to obtain a straight line;

[0015] A second processing module, configured to filter the projected wall point cloud based on the straight line to obtain a second point cloud vector;

[0016] A combining module is used to combine the first point cloud vector and the second point cloud vector into edge semantic information.

[0017] In the above scheme, the extraction module is also used to obtain multiple three-dimensional point clouds obtained by scanning the wall with a depth camera; from the multiple three-dimensional point clouds, multiple three-dimensional point clouds with heights greater than a height threshold are selected as multiple obstacle point clouds.

[0018] In the above scheme, the extraction module is also used to determine the edge strategy that the robot needs to execute based on the relative position relationship between the edge of the wall and the robot; when the edge strategy is the right edge, multiple three-dimensional point clouds with a height greater than a height threshold and located on the right side of the depth camera are selected from the multiple three-dimensional point clouds as multiple obstacle point clouds; when the edge strategy is the left edge, multiple three-dimensional point clouds with a height greater than a height threshold and located on the left side of the depth camera are selected from the multiple three-dimensional point clouds as multiple obstacle point clouds.

[0019] In the above scheme, the device also includes a third processing module, which is used to, when the number of the multiple obstacle point clouds is less than a quantity threshold, re-scan the wall through the depth camera and obtain the three-dimensional point cloud obtained after the re-scanning by the depth camera; and extract the obstacle point cloud from the three-dimensional point cloud obtained after the re-scanning.

[0020] In the above scheme, the extraction module is also used to perform plane fitting on the multiple obstacle point clouds to obtain a plane; from the multiple obstacle point clouds, the obstacle point clouds whose distance from the plane is less than a first distance threshold are selected as wall point clouds.

[0021] In the above scheme, the first processing module is also used to perform the following processing for each obstacle point cloud among the multiple obstacle point clouds except the wall point cloud: determine the distance from the obstacle point cloud to the ground; based on the distance and the normal vector of the ground, determine the projection point of the obstacle point cloud on the ground; combine multiple projection points to obtain a first point cloud vector.

[0022] In the above scheme, the second processing module is also used to select the wall point cloud whose distance to the straight line is less than a second distance threshold from the projected wall point cloud as the wall line point cloud; and generate a second point cloud vector based on the wall line point cloud.

[0023] In the above scheme, the second processing module is further used to identify outlier point clouds from the wall line point clouds; delete the outlier point clouds from the wall line point clouds, and combine the remaining wall line point clouds into a second point cloud vector.

[0024] An embodiment of the present application provides an electronic device, including:

[0025] A memory for storing computer executable instructions or computer programs;

[0026] The processor is used to implement a point cloud data processing method provided in an embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0027] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which are used to implement a point cloud data processing method provided in an embodiment of the present application when executed by a processor.

[0028] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, a point cloud data processing method provided in an embodiment of the present application is implemented.

[0029] The embodiments of the present application have the following beneficial effects:

[0030] On the one hand, by processing the obstacle point cloud, the wall point cloud in the obstacle point cloud is determined, the wall point cloud is projected to the ground, and the projected wall point cloud is linearly fitted. The projected wall point cloud is screened according to the fitted straight line to obtain the second point cloud vector to obtain the edge information of the wall, so that the obtained edge semantic information is more accurate, which effectively solves the problem of the non-straight edge semantics obtained in the related technology; on the other hand, the obstacle point clouds other than the wall point cloud are projected to the ground and saved to obtain the first point cloud vector, and the first point cloud vector and the second point cloud vector are combined to obtain the edge semantic information, so that the obtained edge semantic information is more comprehensive. Since the obstacle point cloud information is retained in the edge semantic information, the robot will consider whether there are obstacles in the path when performing edge motion according to the edge semantic information, that is, the robot can have better obstacle avoidance ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 11 is a schematic diagram of the architecture of a point cloud data processing system 100 provided in an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application;

[0033] Figure 3 It is a flowchart of a point cloud data processing method provided in an embodiment of the present application;

[0034] Figure 4 It is a flowchart of a point cloud data processing method provided in an embodiment of the present application;

[0035] Figure 5 It is a flowchart of a point cloud data processing method provided in an embodiment of the present application;

[0036] Figure 6 It is a process schematic diagram of the point cloud data processing method provided in an embodiment of the present application;

[0037] Figure 7 It is a schematic diagram of the effect of the point cloud data processing method provided in the embodiment of the present application;

[0038] Figure 8 It is a schematic diagram of the effect of the point cloud data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0040] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0041] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0042] In the following description, the terms "first\second\..." involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0044] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0045] 1) LiDAR (Light Detection and Ranging): LiDAR is a remote sensing technology used to measure distance. It determines the distance of a target by emitting laser pulses to the target and measuring the time difference or phase change of the reflected light waves. A LiDAR system usually includes a laser transmitter, a receiver (which can be a photodiode or an optical lens), and a processor. Common LiDARs include pulsed LiDAR, continuous wave LiDAR, multi-frequency LiDAR and other types. LiDAR is widely used in many fields such as autonomous driving, robot navigation, and security monitoring.

[0046] 2) Depth camera: A depth camera is an imaging device that can perceive the three-dimensional information of an object. It can obtain the depth information of a scene through different technical means, such as Time of Flight (TOF), Structured Light, Stereo Vision, etc. Compared with LiDAR, a depth camera has a higher three-dimensional perception capability.

[0047] 3) Edge semantic information: Edge semantic information refers to the semantic labels or category information related to the edge in the image or point cloud data. This information is usually obtained through edge detection algorithms and subsequent semantic segmentation techniques.

[0048] 4) Outlier point clouds: Outlier point clouds refer to those points in the point cloud dataset that are significantly different from other points. These points may be caused by measurement errors, anomalies in the data collection process, or actual objects in the scene. The detection and removal of outlier point clouds is an important step in point cloud processing. Outlier point clouds usually have the characteristics of long distance, low field density, and non-compliance with the overall trend.

[0049] Cleaning and some other types of robots may need to move strictly along the edges (walls).

[0050] There are two main edge solutions in related technologies. One solution is to use the point cloud projected by the laser radar on the wall as the edge semantics to provide a basis for the robot to plan and control the execution of edge actions. However, this method has two problems. On the one hand, due to the low accuracy of the laser radar at close range, the measured distance will fluctuate, resulting in the obtained edge semantics being not straight and inaccurate. If the edge is too close to the wall, it is easy for the machine to hit the wall. If it is too far, strict edge cannot be achieved, which makes planning and control difficult. On the other hand, cleaning robots that generally require edge functions will only be equipped with two-dimensional laser radars, which can only perceive the edge semantic information at the height of the radar installation. When there are obstacles at other heights, it is easy for the robot to hit the obstacles. Another solution is that the relevant deployment personnel directly push the robot along the path to form a teaching path. This solution generally does not hit the wall, but the deployment cost is high.

[0051] In other words, due to the insufficient three-dimensional perception capability of the laser radar, the relevant laser edge technology is prone to colliding with obstacles at heights that are not perceived by the laser radar, and due to the fluctuation of the point cloud, the accuracy of the acquired edge semantic information is easily low, making the robot's edge movements incoherent.

[0052] In view of this, the embodiments of the present application provide a point cloud data processing method, device, electronic device, computer-readable storage medium and computer program product, which can obtain more accurate edge semantic information, thereby enabling the robot to have better obstacle avoidance capabilities. The electronic device provided in the embodiments of the present application can be implemented as a server, or implemented by a server and a terminal in collaboration. The following is an example of the point cloud data processing method provided in the embodiments of the present application being implemented in collaboration by a server and a terminal.

[0053] For example, see Figure 1 , Figure 1 is a schematic diagram of the architecture of a point cloud data processing system 100 provided in an embodiment of the present application, in order to realize supporting a point cloud data processing application, such as Figure 1 As shown, the point cloud data processing system 100 includes: a server 200, a network 300, and a terminal 400 (such as various types of robots). The terminal 400 is connected to the server 200 via the network 300, wherein the network 300 can be a local area network or a wide area network, or a combination of the two.

[0054] In some embodiments, after acquiring multiple obstacle point clouds, the terminal 400 can transmit the multiple obstacle point clouds to the server 200 through the network 300, and then the server 200 extracts the wall point cloud from the multiple obstacle point clouds; then, the server 200 projects the obstacle point clouds other than the wall point cloud in the multiple obstacle point clouds to the ground to obtain a first point cloud vector; then, the server 200 projects the wall point cloud to the ground, and performs linear fitting on the projected wall point cloud to obtain a straight line; then, the server 200 filters the projected wall point cloud based on the straight line to obtain a second point cloud vector; finally, the server 200 combines the first point cloud vector and the second point cloud vector into edge semantic information, and returns the edge semantic information to the terminal 400 through the network 300, so that the terminal 400 can perform edge tasks according to the edge semantic information.

[0055] Of course, the above process can also be implemented by terminal 400 alone. For example, terminal 400 can determine edge semantic information based on its own computing power and perform edge motion according to the edge semantic information. The embodiment of the present application does not make specific limitations on this.

[0056] It should be noted that the technical solution provided in this application can be applied to various scenarios, such as household cleaning, warehousing and logistics, lawn mowing, security patrols and other robot edge movement scenarios.

[0057] In some embodiments, taking the edge movement scenario of a household cleaning robot as an example, first, the household cleaning robot obtains multiple obstacle point clouds; then, extracts wall point clouds from the multiple obstacle point clouds; then, projects the obstacle point clouds other than the wall point clouds to the ground to obtain a first point cloud vector; then, projects the wall point cloud to the ground, and performs linear fitting on the projected wall point cloud to obtain a straight line; then, the projected wall point cloud is screened based on the straight line to obtain a second point cloud vector; finally, the first point cloud vector and the second point cloud vector are combined into edge semantic information to control the household cleaning robot to perform edge movement according to the edge semantic information to complete the household cleaning work.

[0058] In some embodiments, taking the scene of a lawn mowing robot moving along the edge as an example, first, the lawn mowing robot scans the obstacles along the edge of the lawn through a depth camera to obtain multiple three-dimensional point clouds; then, multiple three-dimensional point clouds with heights greater than a height threshold are selected from the multiple three-dimensional point clouds as multiple obstacle point clouds; then, the obstacle point clouds except the wall point clouds in the multiple obstacle point clouds are projected to the ground to obtain a first point cloud vector; then, the wall point cloud is projected to the ground, and the projected wall point cloud is linearly fitted to obtain a straight line; then, the projected wall point cloud is screened based on the straight line to obtain a second point cloud vector; finally, the first point cloud vector and the second point cloud vector are combined into edge semantic information to control the lawn mowing robot to perform edge movement according to the edge semantic information to complete the lawn mowing task.

[0059] In other embodiments, the embodiments of the present application can also be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0060] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0061] For example, Figure 1 The server 200 in the example may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and various types of robots (such as a sweeping robot, a lawn mowing robot, etc.), but is not limited thereto. The terminal 400 and the server 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0062] The structure of the electronic device provided in the embodiment of the present application is further described below. Taking the electronic device as a server as an example, see Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, Figure 2The electronic device 500 shown includes: at least one processor 510, a memory 540, and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 530 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 530 is not described in detail. Figure 2 Various buses are labeled as bus system 530 .

[0063] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 540 may optionally include one or more storage devices that are physically remote from the processor 510.

[0065] The memory 540 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 540 described in the embodiments of the present application is intended to include any suitable type of memory.

[0066] In some embodiments, memory 540 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0067] Operating system 541, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0068] A network communication module 542, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB);

[0069] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The point cloud data processing device 543 stored in the memory 540 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an extraction module 5431, a first processing module 5432, a fitting module 5433, a second processing module 5434, a combination module 5435 and a third processing module 5436. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. It should be pointed out that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the point cloud data processing device 543 excludes the implementation of only including the extraction module 5431, the first processing module 5432, the fitting module 5433, the second processing module 5434 and the combination module 5435. The functions of each module will be explained below.

[0070] The point cloud data processing method provided in the embodiment of the present application will be specifically described below in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0071] See also Figure 3 , Figure 3 is a flow chart of the point cloud data processing method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0072] In step 101, a plurality of obstacle point clouds are obtained.

[0073] In some embodiments, obstacles may be directly scanned using sensors such as lidar, cameras, depth cameras, or radars to obtain spatial point cloud data of the obstacles.

[0074] For example, assuming that a lidar is installed on a robot operating platform, the robot can use the lidar to scan obstacles and capture the spatial point cloud data of the obstacles, and process the acquired obstacle data to help the robot complete the obstacle avoidance task; assuming that a radar sensor is installed on a car, the car can obtain the spatial point cloud data of the obstacles through the radar, and realize intelligent obstacle avoidance operations during autonomous driving by processing the spatial point cloud data.

[0075] In some embodiments, the above-mentioned step 101 can also be implemented in the following way: obtain multiple three-dimensional point clouds obtained by scanning the wall with a depth camera; from the multiple three-dimensional point clouds, select multiple three-dimensional point clouds with heights greater than a height threshold as multiple obstacle point clouds.

[0076] It should be noted that the depth camera can be a built-in camera of the robot itself, or an external camera connected later as needed, and no specific limitation is made here.

[0077] In some embodiments, a depth camera (such as Kinect, Intel RealSense, etc.) is used to scan the wall to capture the depth information of the current scene environment and obtain multiple three-dimensional point cloud data; then, the three-dimensional spatial coordinates of each point cloud are obtained (that is, the coordinates of the point cloud corresponding to the x-axis, y-axis and z-axis in the spatial rectangular coordinate system); then, the point cloud is filtered according to its height information (that is, the z-axis coordinate), and the three-dimensional point clouds whose height information is greater than a height threshold (for example, the threshold is 0, that is, the point clouds above the ground are regarded as obstacle point clouds) are filtered out to obtain obstacle point clouds.

[0078] Here, the height threshold may be manually set by a user, or may be determined by obtaining the height information of the obstacle through a depth camera or other sensor and dynamically analyzing the height information of the obstacle, and is not specifically limited here.

[0079] For example, taking a household cleaning robot as an example, first, the household cleaning robot scans the wall with the loaded depth camera, captures the depth information of the current scene environment, and obtains multiple three-dimensional point cloud data; then, the three-dimensional spatial coordinates of each point cloud data are obtained, and a point cloud data set containing multiple three-dimensional coordinate points is generated; then, the height threshold is determined to be 100 pixels according to the height of the obstacle point cloud, that is, the point cloud with a z-axis coordinate greater than 100 pixels is considered to be an obstacle point cloud, and the point cloud data set is filtered according to the height threshold, and only the point cloud with a z-axis coordinate height greater than 100 pixels is filtered out, so as to obtain the corresponding obstacle point cloud.

[0080] In some embodiments, the above-mentioned selection of multiple three-dimensional point clouds with heights greater than a height threshold from multiple three-dimensional point clouds as multiple obstacle point clouds can also be achieved in the following way: based on the relative position relationship between the edge of the wall and the robot, determining the edge strategy that the robot needs to execute; when the edge strategy is the right edge, selecting multiple three-dimensional point clouds with heights greater than the height threshold and located on the right side of the depth camera from multiple three-dimensional point clouds as multiple obstacle point clouds; when the edge strategy is the left edge, selecting multiple three-dimensional point clouds with heights greater than the height threshold and located on the left side of the depth camera from multiple three-dimensional point clouds as multiple obstacle point clouds.

[0081] In some embodiments, the relative position between the edge of the wall and the robot is obtained by a sensor. When the distance between the left side of the robot and the edge of the wall is greater than the distance between the right side of the robot and the edge of the wall, it is determined to adopt a right edge strategy (i.e., control the right side of the robot to move along the edge of the wall). When the distance between the left side of the robot and the edge of the wall is less than the distance between the right side of the robot and the edge of the wall, it is determined to adopt a left edge strategy (i.e., control the left side of the robot to move along the edge of the wall). When the edge strategy is the right edge, the point cloud with a y-axis coordinate greater than 0 and a z-axis coordinate greater than a height threshold in the three-dimensional coordinates of the obstacle point cloud is saved to obtain the corresponding obstacle point cloud. When the edge strategy is the left edge, the point cloud with a y-axis coordinate less than 0 and a z-axis coordinate greater than a height threshold in the three-dimensional coordinates of the obstacle point cloud is saved to obtain the corresponding obstacle point cloud.

[0082] For example, taking a household cleaning robot as an example, assuming that the distance between the edge of the wall and the left side of the robot is 100 pixels, and the distance between the edge of the wall and the right side of the robot is 300 pixels, by comparison, it can be determined that the robot needs to adopt a left edge strategy, and filter out the point clouds with y-axis coordinates less than 0 and z-axis coordinates greater than a height threshold (for example, 100 pixels) in the point cloud dataset, to obtain the corresponding obstacle point cloud.

[0083] In some embodiments, when the number of multiple obstacle point clouds is less than a quantity threshold, the wall is scanned again by the depth camera, and a three-dimensional point cloud obtained after the re-scanning by the depth camera is obtained; and the obstacle point cloud is extracted from the three-dimensional point cloud obtained after the re-scanning.

[0084] Here, the quantity threshold may be manually set by the user or dynamically set by the machine according to scene conditions, and is not specifically limited here.

[0085] For example, assuming that the number of multiple obstacle point clouds initially acquired by the robot is 50, which is less than the quantity threshold of 100, the robot's depth camera is controlled to adjust the angle and scan the wall again to acquire point clouds for areas that have not been scanned before, and obtain a three-dimensional point cloud after the rescan; then, the three-dimensional point cloud after the rescan is filtered to obtain the corresponding obstacle point cloud.

[0086] In some embodiments, when the number of multiple obstacle point clouds is less than a quantity threshold, the robot can also be controlled to use the mounted laser radar device to scan the wall to obtain three-dimensional point cloud data of the current scene; then, the collected point cloud data is preprocessed (such as denoising, filtering, and removing abnormal points) to improve the point cloud quality; then, a machine learning or deep learning method is used to classify the point cloud data to determine the wall point cloud; then, an edge detection algorithm (such as the Canny operator, or calculating the curvature of the point cloud midpoint to identify the edge point) is used to detect the edge or contour of the wall point cloud to obtain the wall edge point cloud; then, the wall edge point cloud data is converted into the robot coordinate system to ensure that the robot can understand and utilize the edge point cloud data of the wall; finally, according to the edge point cloud of the wall, edge semantic information is generated to plan the control system according to the edge semantic information to formulate the robot's walking path and action strategy.

[0087] For example, suppose there is a robot walking along a wall in an indoor environment. The number of obstacle point clouds obtained by the robot after scanning and processing with a depth camera is 70, which is less than the number threshold of 100. The robot controls the laser radar it carries to scan the wall and collects three-dimensional point cloud data of the current scene; then, the computer system inside the robot pre-processes the collected point cloud data to remove abnormal point clouds caused by environmental interference or equipment errors; then, a machine learning algorithm is used to classify the point cloud data to determine the wall point cloud; then, the robot further analyzes the wall point cloud, detects the edge of the wall, and obtains the edge point cloud of the wall; then, the robot converts the coordinate system of the laser radar into the coordinate system of the robot itself; finally, the robot generates edge semantic information based on the edge point cloud of the wall, and determines the robot's walking path and wall walking strategy based on the edge semantic information.

[0088] In step 102, a wall point cloud is extracted from a plurality of obstacle point clouds.

[0089] It should be noted that a point cloud segmentation algorithm can be used to segment wall point clouds from multiple obstacle point clouds, a clustering algorithm can be used to filter out wall point clouds from multiple obstacle point clouds, and a machine learning or deep learning algorithm can be used to classify obstacle point clouds to obtain wall point clouds, without specific limitation here.

[0090] In some embodiments, see Figure 4 , Figure 4 is a flow chart of a point cloud data processing method provided in an embodiment of the present application, such as Figure 4 As shown, Figure 3 Step 102 shown may be performed by Figure 4 Steps 1021 to 1022 shown in the figure are implemented by combining Figure 4 The steps shown are explained.

[0091] In step 1021, plane fitting is performed on multiple obstacle point clouds to obtain a plane.

[0092] In some embodiments, the above-mentioned plane fitting of multiple obstacle point clouds to obtain a plane can also be achieved in the following way: select part of the point clouds from the multiple obstacle point clouds; construct a fitting plane equation (for example, Ax+By+Cz+D=0) based on the screened part of the point clouds; calculate the distance between the obstacle point clouds other than the screened part of the point clouds and the fitting plane, and determine the error of the fitting plane; when the error exceeds the error range, re-select a different point cloud from the obstacle point cloud for plane fitting, or adjust the parameters of the fitting plane equation.

[0093] It should be noted that the method of selecting some point clouds from multiple obstacle point clouds can be random selection or selection based on a certain standard (such as point distribution uniformity), which is not specifically limited here.

[0094] For example, suppose there is a set of obstacle point cloud data {(x1, y1, z1), (x2, y2, z3), ···, (x n ,y n , z n )}, there are n groups of obstacle point cloud data in total, m point cloud data are randomly selected from them, where m is a positive integer less than n; a fitting plane equation Ax+By+Cz+D=0 is constructed according to the selected m point cloud data, the distances between the nm remaining obstacle point clouds and the fitting plane are calculated, and the error of the fitting plane is determined. When the error exceeds the error range, a different point cloud is re-selected from the obstacle point cloud for plane fitting, or the parameters of the fitting plane equation are adjusted.

[0095] In some embodiments, the above-mentioned plane fitting of multiple obstacle point clouds to obtain the plane can be achieved in the following manner: setting the maximum number of iterations, the inlier threshold and the inlier number threshold; randomly selecting a minimum number (for example, 3) of point cloud fitting planes from multiple obstacle point clouds; using the selected point cloud data to calculate the model parameters of the fitting plane; calculating the distance between the obstacle point cloud other than the randomly selected point cloud and the fitting plane, and marking the point cloud with a distance less than the inlier threshold as an inlier; determining the number of inliers, and when the number of inliers is greater than the inlier number threshold, determining the current model as the best model; continuously iterating until the maximum number of iterations is reached, and selecting the plane with the most inliers as the best plane model.

[0096] For example, suppose there is a set of obstacle point cloud data, set the maximum number of iterations to 100, the inlier threshold to 10 pixels, and the inlier number threshold to 30 points. In each iteration, three points are randomly selected to fit a plane. For each fitted plane, the distances from all points to the plane are calculated, and the points with a distance less than 10 pixels are marked as inliers. The number of inliers is checked. If there are more than 30 and the model quality is better than the previous model, the best model is updated (that is, the current model replaces the previous model). Iterate continuously until the maximum number of iterations is 100, and after all iterations are completed, the plane with the most inliers is selected as the best plane model.

[0097] In some embodiments, the above-mentioned plane fitting of multiple obstacle point clouds to obtain a plane can be achieved in the following way: calculating the covariance matrix corresponding to the multiple obstacle point clouds; determining the first N principal components in the covariance matrix, wherein the principal component represents the main variance direction of the obstacle point cloud, and N is a positive integer greater than 0; constructing a plane equation based on the first N principal components to obtain a fitting plane.

[0098] It should be noted that in the process of determining the first N principal components in the covariance matrix, the user can manually set the value of N, or dynamically select the principal components greater than the contribution rate threshold according to the contribution rate of the set principal components, which is not specifically limited here.

[0099] For example, suppose there is a set of obstacle point cloud data, calculate the covariance matrix corresponding to the obstacle point cloud, perform eigenvalue decomposition on the covariance matrix, and obtain the corresponding eigenvalues ​​and eigenvectors, where the size of the eigenvalue represents the variance of the data distribution in each direction, and the eigenvector represents these directions. Filter out the first N eigenvectors with larger eigenvalues. The directions corresponding to these eigenvectors are the main variation directions of the data, that is, the normal vectors of the plane, and construct the fitting plane equation based on the first N eigenvectors. Calculate the distance from all obstacle point clouds to the fitting plane to evaluate the quality of the fitting. The fitting plane equation can be improved by adjusting the parameters, and finally determine the corresponding fitting plane.

[0100] In step 1022, an obstacle point cloud whose distance to the plane is less than a first distance threshold is selected from the plurality of obstacle point clouds as a wall point cloud.

[0101] In some embodiments, the distance between each obstacle point cloud and the plane is calculated; a first distance threshold is set; and obstacle point clouds with a distance less than the first distance threshold are screened out as wall point clouds.

[0102] It should be noted that the first distance threshold may be a preset value or may be dynamically set according to the requirements of different application scenarios, and is not specifically limited here.

[0103] For example, assuming that there is an obstacle point cloud P (x0, y0, z0), and the equation of the fitting plane is Ax+By+Cz+D=0, then the distance d between the obstacle point cloud P and the fitting plane can be calculated using formula (1); assuming that the first distance threshold is d0, when d is less than d0, the obstacle point cloud P is regarded as a wall point cloud.

[0104]

[0105] In step 103, the obstacle point clouds other than the wall point cloud among the plurality of obstacle point clouds are projected onto the ground to obtain a first point cloud vector.

[0106] In some embodiments, when the plane corresponding to the ground is perpendicular to any coordinate axis in the spatial coordinate system, the point cloud coordinates after point cloud projection can be obtained by changing the coordinate values ​​of the point cloud in the spatial coordinate system, thereby obtaining the first point cloud vector.

[0107] For example, assuming that the plane corresponding to the ground is perpendicular to the z-axis in the spatial coordinate system, the plane equation corresponding to the ground is z=z3, and there is an obstacle point cloud Q(x4, y4, z4) other than the wall point cloud. The point cloud coordinates Q0 of the obstacle point cloud Q after being projected to the ground are (x4, y4, z3), and all the projected point clouds are merged into a point cloud data set to obtain the first point cloud vector.

[0108] In some embodiments, the above-mentioned step 103 can also be implemented in the following manner: for each obstacle point cloud except the wall point cloud in multiple obstacle point clouds, perform the following processing: determine the distance from the obstacle point cloud to the ground; based on the distance and the normal vector of the ground, determine the projection point of the obstacle point cloud on the ground; combine multiple projection points to obtain a first point cloud vector.

[0109] In some embodiments, the distance from the obstacle point cloud to the ground can be determined by formula (1); then, the normal vector corresponding to the ground is calculated according to the ground equation; based on the distance from the obstacle point cloud to the ground and the normal vector corresponding to the ground, the projection point of the obstacle point cloud on the ground is determined; finally, multiple projection points are combined to obtain the first point cloud vector.

[0110] For example, suppose there is an obstacle point cloud L(x5, y5, z5), and the plane equation corresponding to the ground is Ax+By+Cz+D=0. The distance d1 from the obstacle point cloud to the ground can be calculated by formula (1). The normal vector n of the ground equation is (A, B, C). Next, the equation of the perpendicular line from the point cloud L to the ground equation is calculated as Finally, according to the distance d1, the ground equation Ax+By+Cz+D=0, and the vertical line equation Determine the projection point L0 corresponding to the obstacle point cloud on the ground plane; combine the projection points of all obstacle point clouds to obtain the first point cloud vector.

[0111] In step 104, the wall point cloud is projected onto the ground.

[0112] It should be noted that the implementation of step 104 in the embodiment of the present application is similar to the implementation of the above-mentioned step 103. For details, please refer to the implementation of step 103, and the embodiment of the present application will not be repeated here.

[0113] In step 105, a linear fit is performed on the projected wall point cloud to obtain a straight line.

[0114] It should be noted that after obtaining the projected wall point cloud, the projected wall point cloud is located on the same two-dimensional plane, so a two-dimensional coordinate system can be established for the projected wall point cloud, that is, the three-dimensional point cloud is reduced to two dimensions, with only x-coordinates and y-coordinates.

[0115] In some embodiments, an appropriate straight line fitting method is selected, and the selected straight line fitting method is used to perform straight line fitting on the two-dimensional wall point cloud data to obtain a corresponding fitting straight line.

[0116] For example, taking the least squares method as an example, assuming that the two-dimensional wall point cloud data {(x1, y1), (x2, y2), ···, (x n ,y n )}, first calculate the mean of all x- and y-coordinates, Then, through the formula The slope a of the fitted line is calculated; then the mean coordinates of x and y and the slope a are substituted into the fitted line y=ax+b to obtain the intercept b of the fitted line, that is, the corresponding fitted line is obtained.

[0117] For example, taking the Random Sample Consensus (RANSAC) algorithm as an example, assuming that the two-dimensional wall point cloud data {(x1, y1), (x2, y2), ···, (x n ,y n )}, first set the number of iterations k = 1000, the minimum number of inliers d = 2, and the distance threshold t = 10 pixels; then perform the following iterative process, randomly select two points (x1, y1) and (x2, y2), and calculate the slope based on the two points The intercept b=y1-a*x1, calculate the distance between all point clouds and the straight line, and mark the point clouds with distances less than the distance threshold t as inliers, count the number of inliers, and if the number of inliers is greater than the number of inliers in the current best model and greater than the minimum number of inliers d, update the best model and the inlier set; continue to iterate until the number of iterations reaches k; finally, select the model with the largest number of inliers as the final fitting line.

[0118] In step 106, the projected wall point cloud is screened based on the straight line to obtain a second point cloud vector.

[0119] It should be noted that the first point cloud vector and the second point cloud vector are only used to distinguish point cloud vectors storing different data, and the storage types of the two point cloud vectors may be the same or different, which is not specifically limited here.

[0120] In some embodiments, see Figure 5 , Figure 5 is a flow chart of a point cloud data processing method provided in an embodiment of the present application, such as Figure 5 As shown, Figure 3 Step 106 shown may be performed by Figure 5 Steps 1061 to 1062 shown in the figure are implemented by combining Figure 5 The steps shown are explained.

[0121] In step 1061 , a wall point cloud whose distance to the straight line is less than a second distance threshold is selected from the projected wall point cloud as a wall line point cloud.

[0122] In some embodiments, the distance between each projected wall point cloud and the straight line is calculated; a second distance threshold is set; and the wall point clouds with a distance less than the second distance threshold are screened out as wall line point clouds.

[0123] It should be noted that the first distance threshold and the second distance threshold are used to distinguish different distance thresholds. The two values ​​can be the same or different. The second distance threshold can be a preset value or dynamically set according to the requirements of different application scenarios. No specific limitation is made here.

[0124] For example, assuming that there is a projected wall point cloud M (x0, y0), and the equation of the fitting line is Ax+By+C=0, then the distance d2 between the projected wall point cloud M and the fitting line can be calculated using formula (2); assuming that the second distance threshold is d1, when d2 is less than d1, the projected wall point cloud is used as the wall line point cloud.

[0125]

[0126] In step 1062 , a second point cloud vector is generated based on the wall line point cloud.

[0127] In some embodiments, the above step 1062 can also be implemented in the following manner: identifying outlier point clouds from the wall line point clouds; deleting the outlier point clouds from the wall line point clouds, and combining the remaining wall line point clouds into a second point cloud vector.

[0128] For example, suppose there is a set of wall point cloud data {(1,2,0), (1.1,2.1,0), (1.2,2.2,0), (1.3,2.3,0), (10,10,0), (1.4,2.4,0), (1.5,2.5,0)}. Here, most of the point clouds are tightly clustered together, while the point cloud (10,10,0) is obviously far away from other point clouds. It may be a measurement error or an outlier point cloud. The average distance from each point cloud to the fitting line can be calculated, and then Those point clouds whose distance exceeds the set threshold are regarded as outlier point clouds. Assuming the set threshold is 0.2, based on this threshold, the point cloud (10,10,0) is identified as an outlier point cloud. The outlier point cloud is deleted, and the remaining point cloud data is {(1,2,0), (1.1,2.1,0), (1.2,2.2,0), (1.3,2.3,0), (1.4,2.4,0), (1.5,2.5,0)}. Finally, the remaining point cloud data are combined to obtain the second point cloud vector.

[0129] In step 107 , the first point cloud vector and the second point cloud vector are combined into edge semantic information.

[0130] In some embodiments, the point cloud data in the first point cloud vector is combined with the point cloud data in the second point cloud vector to obtain edge semantic information that contains the point cloud data in both point cloud vectors, so as to perform further processing based on the edge semantic information (such as three-dimensional modeling, path planning, or obstacle detection, etc.).

[0131] For example, assuming that the first point cloud vector contains obstacle point clouds other than wall point clouds, and the second point cloud vector contains filtered wall line point clouds, the two vectors are combined, and a semantic label is assigned to each point to create edge semantic information, i.e., the combined edge semantic information, wherein each point cloud vector in the edge semantic information clearly identifies the type of point cloud (wall line point cloud or obstacle point cloud other than wall point clouds), and the edge semantic information can be further processed subsequently (such as 3D modeling, navigation path planning, or obstacle detection operations) to achieve precise control of the robot's motion.

[0132] Next, an exemplary application of the embodiment of the present application in a practical application scenario will be described. The exemplary application describes the specific implementation process of the point cloud data processing method in a robot moving along the edge scenario.

[0133] LiDAR edge technology is prone to colliding with obstacles that are not at the laser installation height due to the insufficient three-dimensional perception capability of LiDAR. In addition, due to the low accuracy of the point cloud, the acquired edge semantic information is not straight, resulting in incoherence when the robot performs edge actions. Compared with LiDAR, the three-dimensional point cloud of the depth camera has higher perception accuracy at close range and can perceive obstacles at different heights, thereby obtaining more comprehensive edge semantic information.

[0134] The present application performs plane segmentation on the obstacle point cloud by using a typical plane segmentation algorithm, segments the wall point cloud and the protruding obstacle point cloud, and projects the wall point cloud to the ground for fitting to obtain a fitting straight line, and retains the point cloud data within a certain distance of the fitting straight line, and combines the point cloud data within a certain distance of the fitting straight line with the protruding obstacle point cloud to obtain edge semantic information. The protruding obstacle point cloud is retained in the edge semantic information, which can achieve a higher level of machine obstacle avoidance.

[0135] See also Figure 6 , Figure 6 This is a schematic diagram of the process of the point cloud data processing method provided in the embodiment of the present application. Figure 6 The specific process of the point cloud data processing method is explained.

[0136] In step 201, three-dimensional point cloud data collected by a depth camera is obtained.

[0137] For example, a depth camera (such as Kinect, Intel RealSense, etc.) is used to scan the wall, capture the depth information of the current scene environment, and obtain multiple three-dimensional point cloud data.

[0138] In step 202, the ground point cloud is removed, and the point cloud on one side of the depth camera is filtered out according to the edge strategy to obtain the obstacle point cloud.

[0139] For example, first save the point cloud whose height is greater than the height threshold (that is, the z coordinate of the point cloud pointcloud.z>Z0), then obtain the relative position of the edge of the wall and the robot through the sensor, when the distance between the left side of the robot and the edge of the wall is greater than the distance between the right side of the robot and the edge of the wall, it is determined to adopt the right edge strategy, and save the point cloud with a y-axis coordinate greater than 0 in the three-dimensional coordinates of the obstacle point cloud to obtain the corresponding obstacle point cloud, when the distance between the left side of the robot and the edge of the wall is less than the distance between the right side of the robot and the edge of the wall, it is determined to adopt the left edge strategy, and save the point cloud with a y-axis coordinate less than 0 in the three-dimensional coordinates of the obstacle point cloud to obtain the corresponding obstacle point cloud.

[0140] In step 203 , it is determined whether the number of obstacle point clouds is greater than 100. If so, step 204 is executed; if not, step 205 is executed.

[0141] For example, if the number of obstacle point clouds is greater than 100, the subsequent step 204 is executed; if the number of obstacle point clouds is less than 100, it means that the number of obstacle point clouds is too small to form sufficiently long edge semantic information, and step 205 is executed.

[0142] In step 204, the plane model is used to extract internal points as wall point clouds.

[0143] For example, the SACMODEL_PLANE model is used to fit the plane using the RANSAC algorithm to obtain the fitted plane, and the SACSegmentation method of opencv is used to extract the inner points as the wall point cloud.

[0144] In step 205, it is returned that the extraction of edge semantic information fails.

[0145] For example, when the number of obstacle point clouds is less than 100, a prompt message indicating that the current point cloud data processing method fails to extract edge semantic information is returned.

[0146] In step 206 , the z values ​​of all wall point clouds are set to 0.

[0147] For example, see Figure 7 , Figure 7 : is a schematic diagram of the effect of the point cloud data processing method provided in the embodiment of the present application. All wall point clouds are projected onto the ground to obtain the following Figure 7 The projected wall point cloud is shown.

[0148] In step 207, the z values ​​of the remaining obstacle point clouds are set to 0 and stored in the curve point cloud vector.

[0149] Here, curve point cloud vectors are used to store point cloud information of obstacles other than wall point clouds, which can effectively realize intelligent obstacle avoidance of the robot.

[0150] In step 208, the projected wall point cloud is fitted into a straight line.

[0151] For example, the SACMODEL_LINE model is used to perform linear fitting using the RANSAC algorithm to obtain a fitted straight line.

[0152] In step 209, the inner points are extracted as the wall line point cloud.

[0153] For example, see Figure 8 , Figure 8This is a schematic diagram of the effect of the point cloud data processing method provided in the embodiment of the present application. The SACSegmentation method of opencv is used to extract the inner points as the wall line point cloud to achieve the following Figure 8 The effect shown.

[0154] In step 210, the outlier point cloud is removed and saved as a linePC point cloud vector.

[0155] For example, the RadiusOutlierRemoval method of OpenCV is used to remove outliers, and the result is saved as a point cloud vector linePC.

[0156] In step 211, the linePC point cloud vector and the curve point cloud vector are combined to obtain complete edge semantic information.

[0157] For example, two point cloud vectors are merged to obtain the final edge semantic information, so as to further process the edge semantic information (such as 3D modeling, navigation path planning or obstacle detection) to achieve precise control of the robot movement.

[0158] The following is a description of an exemplary structure of the point cloud data processing device 543 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the point cloud data processing device 543 of the memory 540 may include: an extraction module 5431 , a first processing module 5432 , a fitting module 5433 , a second processing module 5434 and a combination module 5435 .

[0159] The extraction module 5431 is used to obtain multiple obstacle point clouds and extract wall point clouds from the multiple obstacle point clouds; the first processing module 5432 is used to project the obstacle point clouds other than the wall point clouds in the multiple obstacle point clouds to the ground to obtain a first point cloud vector; the fitting module 5433 is used to project the wall point cloud to the ground and perform linear fitting on the projected wall point cloud to obtain a straight line; the second processing module 5434 is used to filter the projected wall point cloud based on the straight line to obtain a second point cloud vector; the combination module 5435 is used to combine the first point cloud vector and the second point cloud vector into edge semantic information.

[0160] In some embodiments, the extraction module 5431 is also used to obtain multiple three-dimensional point clouds obtained by scanning the wall with a depth camera; from the multiple three-dimensional point clouds, multiple three-dimensional point clouds with heights greater than a height threshold are selected as multiple obstacle point clouds.

[0161] In some embodiments, the extraction module 5431 is also used to determine the edge strategy that the robot needs to execute based on the relative position relationship between the edge of the wall and the robot; when the edge strategy is the right edge, multiple three-dimensional point clouds with a height greater than a height threshold and located on the right side of the depth camera are selected from multiple three-dimensional point clouds as multiple obstacle point clouds; when the edge strategy is the left edge, multiple three-dimensional point clouds with a height greater than a height threshold and located on the left side of the depth camera are selected from multiple three-dimensional point clouds as multiple obstacle point clouds.

[0162] In some embodiments, the point cloud data processing device also includes a third processing module 5436, which is used to re-scan the wall through the depth camera when the number of multiple obstacle point clouds is less than the number threshold, and obtain the three-dimensional point cloud obtained after the re-scanning by the depth camera; extract the obstacle point cloud from the three-dimensional point cloud obtained after the re-scanning.

[0163] In some embodiments, the extraction module 5431 is further used to perform plane fitting on multiple obstacle point clouds to obtain a plane; from the multiple obstacle point clouds, an obstacle point cloud whose distance to the plane is less than a first distance threshold is selected as a wall point cloud.

[0164] In some embodiments, for each obstacle point cloud except the wall point cloud in multiple obstacle point clouds, the first processing module 5432 is also used to determine the distance from the obstacle point cloud to the ground; based on the distance and the normal vector of the ground, determine the projection point of the obstacle point cloud on the ground; and combine multiple projection points to obtain a first point cloud vector.

[0165] In some embodiments, the second processing module 5434 is further used to select, from the projected wall point cloud, a wall point cloud whose distance from the straight line is less than a second distance threshold as a wall line point cloud; and generate a second point cloud vector based on the wall line point cloud.

[0166] In some embodiments, the second processing module 5434 is further used to identify outlier point clouds from the wall line point clouds; delete the outlier point clouds from the wall line point clouds, and combine the remaining wall line point clouds into a second point cloud vector.

[0167] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3 , Figure 4 ,or Figure 5 The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0168] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the point cloud data processing method described in the embodiment of the present application.

[0169] The present application embodiment provides a computer-readable storage medium in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the point cloud data processing method provided by the present application embodiment, for example, Figure 3 , Figure 4 ,or Figure 5 The point cloud data processing method is shown.

[0170] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM) and other memories; it may also be various devices including one or any combination of the above memories.

[0171] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0172] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0173] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0174] To summarize, the embodiment of the present application effectively solves the problem of non-straight edge semantics obtained in the related art, and can obtain more comprehensive edge semantic information, which can make the control planning algorithm run more smoothly when using the edge semantic information obtained by the present application. At the same time, because the edge semantic information retains the obstacle point cloud information, the robot will consider whether there are obstacles in the path when performing edge motion according to the edge semantic information, that is, the robot can have better obstacle avoidance ability, effectively avoiding the robot from colliding with obstacles of different heights during edge motion.

[0175] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A point cloud data processing method, characterized in that: The method comprises: Acquire a plurality of obstacle point clouds, and extract a wall point cloud from the plurality of obstacle point clouds; Projecting obstacle point clouds other than the wall point cloud among the plurality of obstacle point clouds onto the ground to obtain a first point cloud vector; Projecting the wall point cloud onto the ground, and performing linear fitting on the projected wall point cloud to obtain a straight line; Filtering the projected wall point cloud based on the straight line to obtain a second point cloud vector; The first point cloud vector and the second point cloud vector are combined into edge semantic information.

2. The method according to claim 1, characterized in that The step of extracting a wall point cloud from the plurality of obstacle point clouds comprises: Performing plane fitting on the plurality of obstacle point clouds to obtain a plane; From the plurality of obstacle point clouds, an obstacle point cloud whose distance from the plane is less than a first distance threshold is selected as a wall point cloud.

3. The method according to claim 1, characterized in that The step of projecting the obstacle point clouds other than the wall point cloud from the plurality of obstacle point clouds onto the ground to obtain a first point cloud vector comprises: For each obstacle point cloud in the plurality of obstacle point clouds except the wall point cloud, the following processing is performed: Determine the distance from the obstacle point cloud to the ground; Determine a projection point of the obstacle point cloud on the ground based on the distance and a normal vector of the ground; A plurality of the projection points are combined to obtain a first point cloud vector.

4. The method according to claim 1, characterized in that: The step of screening the projected wall point cloud based on the straight line to obtain a second point cloud vector includes: From the projected wall point cloud, select a wall point cloud whose distance to the straight line is less than a second distance threshold as a wall line point cloud; A second point cloud vector is generated based on the wall line point cloud.

5. The method according to claim 4, characterized in that The generating a second point cloud vector based on the wall line point cloud comprises: Identifying outlier point clouds from the wall line point cloud; The outlier point cloud is deleted from the wall line point cloud, and the remaining wall line point clouds are combined into a second point cloud vector.

6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining a plurality of obstacle point clouds comprises: Obtain multiple three-dimensional point clouds obtained by scanning the wall with a depth camera; From the multiple three-dimensional point clouds, multiple three-dimensional point clouds with heights greater than a height threshold are selected as multiple obstacle point clouds.

7. The method according to claim 6, characterized in that The step of selecting, from the plurality of three-dimensional point clouds, a plurality of three-dimensional point clouds whose heights are greater than a height threshold as a plurality of obstacle point clouds comprises: Based on the relative position relationship between the edge of the wall and the robot, determining the edge strategy that the robot needs to execute; When the edge strategy is the right edge, selecting, from the multiple three-dimensional point clouds, multiple three-dimensional point clouds whose heights are greater than a height threshold and are located on the right side of the depth camera as multiple obstacle point clouds; When the edge strategy is the left edge, a plurality of three-dimensional point clouds whose heights are greater than a height threshold and are located on the left side of the depth camera are selected from the plurality of three-dimensional point clouds as a plurality of obstacle point clouds.

8. The method according to claim 6 or 7, characterized in that: The method further comprises: When the number of the plurality of obstacle point clouds is less than the number threshold, rescanning the wall by the depth camera, and acquiring a three-dimensional point cloud obtained by the rescanning by the depth camera; An obstacle point cloud is extracted from the three-dimensional point cloud obtained after the re-scanning.

9. A point cloud data processing device, characterized in that: The device comprises: An extraction module, used to obtain a plurality of obstacle point clouds and extract a wall point cloud from the plurality of obstacle point clouds; A first processing module, configured to project obstacle point clouds other than the wall point cloud among the plurality of obstacle point clouds onto the ground to obtain a first point cloud vector; A fitting module, used for projecting the wall point cloud onto the ground, and performing linear fitting on the projected wall point cloud to obtain a straight line; A second processing module, configured to filter the projected wall point cloud based on the straight line to obtain a second point cloud vector; A combining module is used to combine the first point cloud vector and the second point cloud vector into edge semantic information.

10. An electronic device, characterized in that: include: A memory for storing computer executable instructions or computer programs; The processor is used to implement the point cloud data processing method described in any one of claims 1 to 8 when executing the computer executable instructions or computer programs stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the point cloud data processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the point cloud data processing method according to any one of claims 1 to 8 is implemented.

Images