Robot point cloud processing method, apparatus, device, medium, and program product

Abstract

The robot point cloud processing method, device, equipment, medium and program product of the present disclosure relate to the technical field of robots. In some embodiments of the present disclosure, the joint state of the robot arm and the environment perception data are obtained, and the joint state of the robot arm is mapped into the camera coordinate system corresponding to the environment perception data based on the coordinate conversion relationship; based on the joint state of the robot arm in the camera coordinate system, a candidate segmentation region corresponding to the robot arm is constructed; taking a seed point as a starting point, region generation is performed in the candidate segmentation region to obtain a segmented point cloud region of the robot arm and a grasped object; from the panoramic environment point cloud generated in the current frame, the self point cloud cluster corresponding to the segmented point cloud region is removed to obtain a target point cloud, the identification and removal of the robot arm and the handheld object are realized, a clean environment point cloud not containing the interference of the robot itself is obtained, and the accuracy of stripping the point cloud of the robot itself from the environment perception data is improved.

Description

Technical Field

[0001] This disclosure relates to the field of robotics technology, and in particular to a method, apparatus, device, medium, and program product for processing robot point clouds. Background Technology

[0002] With the rapid development of robotics technology, collaborative robots and mobile manipulators are increasingly being used in industrial automation, logistics sorting, and home services. To achieve autonomous obstacle avoidance and environmental interaction, these robots are typically equipped with visual sensors such as RGB-D cameras on their heads or shoulders. They collect environmental data in real time to construct high-precision 3D point cloud maps, providing a basis for path planning and task execution. However, in actual operation, the objects grasped by the robot's robotic arm and end effector often frequently enter the camera's field of view (FOV). If these visual data are not processed in a targeted manner, the perception system will misjudge the robot's own arm and the objects it holds as obstacles in the external environment, leading to a series of technical problems: First, the system is prone to falsely triggering obstacle avoidance mechanisms, causing the robot to mistakenly determine that its path is blocked because it "sees" its own parts, thus failing to perform its intended task; Second, these point cloud data belonging to the robot itself will form false obstacle markers in the grid map or distance field, making it impossible for the global or local planner to calculate an effective path; In addition, as the robotic arm continues to move, the position and shape of these "pseudo-obstacles" in space constantly change, greatly increasing the complexity and uncertainty of environmental modeling.

[0003] Traditional point cloud cropping methods typically employ simple geometric bounding boxes or predefined fixed masks to remove their own point clouds. However, these methods have significant limitations: on the one hand, their cropping accuracy is low, and they cannot adapt to the changing joint shapes of robotic arms under complex movements; on the other hand, for "handheld objects" with unknown or constantly changing shapes, predefined methods are even more difficult to effectively identify and remove.

[0004] Currently, the accuracy of extracting the robot's own point cloud from environmental perception data is relatively low. Summary of the Invention

[0005] This disclosure provides a robot point cloud processing method, apparatus, device, medium, and program product to at least solve the problem of low accuracy in existing methods for extracting robot point clouds from environmental perception data.

[0006] The technical solution disclosed herein is as follows: This disclosure provides a robot point cloud processing method, including: The robot acquires the joint states of its robotic arm and environmental perception data, and maps the joint states of the robotic arm to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships. Based on the joint states of the robotic arm in the camera coordinate system, candidate segmentation regions corresponding to the robotic arm are constructed. Starting from the seed point, region generation is performed within the candidate segmentation area to obtain the segmentation point cloud region of the robotic arm and the grasped object; The target point cloud is obtained by removing the point cloud clusters corresponding to the segmented point cloud regions from the panoramic environment point cloud generated in the current frame.

[0007] Optionally, the method further includes: The robot is controlled to perform path planning operations based on the target point cloud.

[0008] Optionally, the environmental perception data includes: depth map and camera intrinsic parameters. The step of acquiring the robot's manipulator joint states and environmental perception data, and mapping the manipulator joint states to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships, includes: The robot's robotic arm joint states are obtained, and the spatial poses of the robotic arm's links and end effector are calculated by calling the forward kinematics model. Obtain the depth map and the camera intrinsic parameters; Based on the depth map, the camera intrinsic parameters, and the extrinsic parameters between the camera and the robot base, the joint state of the robotic arm is mapped to the camera coordinate system corresponding to the environmental perception data.

[0009] Optionally, constructing the candidate segmentation region corresponding to the robotic arm based on the joint states of the robotic arm in the camera coordinate system includes: The robotic arm is represented as a combination of multiple simplified geometric shapes, thus obtaining the geometric model corresponding to the robotic arm; The geometric model is projected onto the depth image plane to obtain the theoretical area occupied by the robotic arm in the image. The theoretically occupied region is expanded to obtain candidate segmentation regions.

[0010] Optionally, the step of performing region generation within the candidate segmentation region, starting from the seed point, to obtain the segmentation point cloud region of the robotic arm and the grasped object, includes: Depth difference threshold and spatial distance threshold are set as the criteria for determining region growth; Starting from the seed point, search for neighboring pixels within the candidate segmentation region; If the depth difference between the neighboring pixel and the boundary of the current growth region is less than the depth difference threshold, or the distance between the neighboring pixel in three-dimensional space is less than the spatial distance threshold, and the neighboring pixel is always within the preset prior constraint range of the robotic arm model, then the neighboring pixel is included in the current growth region and marked as visited. Repeat the neighborhood search and judgment operation until no neighboring pixels that meet the growth conditions can be found, and obtain the segmented point cloud region of the robotic arm and the grasped object.

[0011] Optionally, the step of removing the self-point cloud clusters corresponding to the segmented point cloud region from the panoramic environment point cloud generated in the current frame to obtain the target point cloud includes: The segmented point cloud region is back-projected into three-dimensional space to obtain its own point cloud cluster; Remove the point cloud clusters of the same type from the panoramic environment point cloud to obtain candidate point clouds; The candidate point cloud is filled with point cloud holes to obtain the target point cloud.

[0012] This disclosure also provides a robot point cloud processing device, including: The acquisition module is used to acquire the robot's robotic arm joint states and environmental perception data, and to map the robotic arm joint states to the camera coordinate system corresponding to the environmental perception data based on the coordinate transformation relationship. The construction module is used to construct candidate segmentation regions corresponding to the robotic arm based on the joint states of the robotic arm in the camera coordinate system. The generation module is used to perform region generation within the candidate segmentation area, starting from the seed point, to obtain the segmentation point cloud region of the robotic arm and the grasped object; The culling module is used to remove the self-point cloud clusters corresponding to the segmented point cloud region from the panoramic environment point cloud generated in the current frame to obtain the target point cloud.

[0013] This disclosure also provides an electronic device, including: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the steps in the above method.

[0014] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0015] This disclosure also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the method described above.

[0016] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: In some embodiments of this disclosure, the robot's robotic arm joint states and environmental perception data are acquired, and the robotic arm joint states are mapped to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships. Based on the robotic arm joint states in the camera coordinate system, candidate segmentation regions corresponding to the robotic arm are constructed. Starting from a seed point, region generation is performed within the candidate segmentation regions to obtain segmentation point cloud regions of the robotic arm and the grasped object. The self-point cloud clusters corresponding to the segmentation point cloud regions are removed from the panoramic environmental point cloud generated in the current frame to obtain the target point cloud, thereby realizing the identification and removal of the robotic arm and its grasped object, obtaining a clean environmental point cloud without interference from the robot itself, and improving the accuracy of removing the robot's own point cloud from the environmental perception data.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 A flowchart illustrating a robot point cloud processing method provided as an exemplary embodiment of this disclosure; Figure 2 A schematic diagram of the structure of a robot point cloud processing device provided for an exemplary embodiment of this disclosure; Figure 3 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0021] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0022] It should be noted that the user information involved in this disclosure includes, but is not limited to, user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] To address the aforementioned technical issues, in some embodiments of this disclosure, the robot's robotic arm joint states and environmental perception data are acquired, and the robotic arm joint states are mapped to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships. Based on the robotic arm joint states in the camera coordinate system, candidate segmentation regions corresponding to the robotic arm are constructed. Starting from a seed point, region generation is performed within the candidate segmentation regions to obtain segmented point cloud regions of the robotic arm and the grasped object. The self-point cloud clusters corresponding to the segmented point cloud regions are removed from the panoramic environmental point cloud generated in the current frame to obtain the target point cloud, thereby realizing the identification and removal of the robotic arm and its grasped object, obtaining a clean environmental point cloud without interference from the robot itself, and improving the accuracy of removing the robot's own point cloud from the environmental perception data.

[0024] The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating a robot point cloud processing method provided as an exemplary embodiment of this disclosure. Figure 1 As shown, the method includes: S101: Acquire the robot's robotic arm joint states and environmental perception data, and map the robotic arm joint states to the camera coordinate system corresponding to the environmental perception data based on the coordinate transformation relationship; S102: Construct candidate segmentation regions for the robotic arm based on the joint states of the robotic arm in the camera coordinate system; S103: Starting from the seed point, perform region generation within the candidate segmentation region to obtain the segmentation point cloud region of the robotic arm and the grasped object; S104: Remove the point cloud clusters corresponding to the segmented point cloud regions from the panoramic environment point cloud generated in the current frame to obtain the target point cloud.

[0026] In this embodiment, the entity executing the above method can be a robot or a server. The type of robot is not limited; it can be any type of robot, including robotic arms.

[0027] In this embodiment, the implementation form of the server is not limited. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server devices. The server mainly consists of a processor, hard disk, memory, system bus, and other common computer architecture types.

[0028] In this embodiment, the robot's robotic arm joint states and environmental perception data are acquired, and the robotic arm joint states are mapped to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships. Based on the robotic arm joint states in the camera coordinate system, candidate segmentation regions corresponding to the robotic arm are constructed. Starting from a seed point, region generation is performed within the candidate segmentation regions to obtain segmentation point cloud regions of the robotic arm and the grasped object. The self-point cloud clusters corresponding to the segmentation point cloud regions are removed from the panoramic environmental point cloud generated in the current frame to obtain the target point cloud, thereby realizing the identification and removal of the robotic arm and its grasped object, obtaining a clean environmental point cloud without interference from the robot itself, and improving the accuracy of removing the robot's own point cloud from the environmental perception data.

[0029] In some embodiments of this disclosure, the robot's manipulator joint states and environmental perception data are acquired, and the manipulator joint states are mapped to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships. One possible approach is to acquire the robot's manipulator joint states, call a forward kinematics model to calculate the spatial poses of the manipulator's links and end effector; acquire a depth map and camera intrinsic parameters; and map the manipulator joint states to the camera coordinate system corresponding to the environmental perception data based on the depth map, camera intrinsic parameters, and extrinsic parameters between the camera and the robot base.

[0030] It's important to note that the joint states of a robotic arm are the core set of physical quantities describing the real-time motion configuration of the robot body. These are typically collected in real-time by encoders or torque sensors installed at each joint. Mathematically, they are primarily represented as a vector containing multiple degrees of freedom. For a rotary joint, the state data is the angle of rotation around its axis; for a translating joint, the state data is the linear displacement along its axis. In practical control and algorithms, joint states not only include position and angle information but also often encompass dynamic parameters such as angular velocity, angular acceleration, and output torque. They are the fundamental input for the robot's forward kinematics calculations. It is based on this real-time joint state data that the system can accurately calculate the absolute pose of each link and end effector in three-dimensional space.

[0031] Environmental perception data refers to the observational information collected by a robot through its external sensors (such as RGB-D depth cameras and LiDAR) to describe the characteristics of the surrounding physical world. It primarily refers to the depth map acquired by the depth camera and the corresponding camera intrinsic parameters. The depth map records the distance information from each pixel within the field of view to the camera's optical center in the form of a two-dimensional matrix, while the camera intrinsic parameters include optical attributes such as focal length and principal point coordinates, used to backproject the two-dimensional pixel coordinates into three-dimensional spatial coordinates. After processing, this data is typically transformed into a 3D point cloud or occupancy grid map, providing the robot with the geometric and semantic information needed for obstacle avoidance, path planning, and object recognition; it is the robot's "eyes" for perceiving the external environment.

[0032] Specifically, to address the self-occlusion and false obstacle problems faced by the RGB-D camera mounted on the shoulder of the mobile robot, the system executes the following processing flow: At the beginning of each control cycle, the controller first obtains the real-time joint angle vectors of each rotation / translation joint of the robotic arm through the robot driver; calls the forward kinematics model to calculate the spatial pose of the robotic arm's links and end effector; acquires the depth map and camera intrinsic parameters collected by the depth camera; based on the depth map, camera intrinsic parameters, and extrinsic parameters between the camera and the robot base, maps the joint state of the robotic arm to the camera coordinate system corresponding to the environmental perception data; and establishes a three-dimensional spatial representation of the current posture of the robotic arm in the camera coordinate system to achieve spatial consistency between the prior geometric information of the robotic arm and the current depth observation.

[0033] In some embodiments of this disclosure, candidate segmentation regions corresponding to the robotic arm are constructed based on the joint states of the robotic arm in the camera coordinate system. One possible approach is to represent the robotic arm as a combination of multiple simplified geometric shapes to obtain a geometric model corresponding to the robotic arm; project the geometric model onto the depth image plane to obtain the theoretically occupied area of ​​the robotic arm in the image; and expand the theoretically occupied area to obtain candidate segmentation regions. Specifically, the robotic arm is represented as a combination of multiple simplified geometric shapes, such as a sphere, cylinder, or capsule; based on the robotic arm pose, the aforementioned geometric model is projected onto the depth image plane to obtain the theoretically occupied area of ​​the robotic arm in the image; the projected area is expanded within a certain range to form candidate regions to accommodate model errors and sensing errors; and the neighborhood of the end effector is further expanded to cover the spatial area directly in contact with or attached to the end effector of the robotic arm. This disclosure transforms the robotic arm pose information into prior constraints in the image domain, narrowing the scope of subsequent processing.

[0034] In some embodiments of this disclosure, starting from a seed point, region generation is performed within the candidate segmentation region to obtain the segmented point cloud region of the robotic arm and the grasped object. One possible approach is to set a depth difference threshold and a spatial distance threshold as the criteria for region growth; starting from the seed point, neighboring pixels are searched within the candidate segmentation region; if the depth difference between a neighboring pixel and the boundary of the current growth region is less than the depth difference threshold, or the distance between a neighboring pixel in three-dimensional space is less than the spatial distance threshold, and the neighboring pixel is always within the preset prior constraint range of the robotic arm model, then the neighboring pixel is included in the current growth region and marked as visited; the neighboring search and judgment operation is repeated until no neighboring pixels that meet the growth conditions can be found, thus obtaining the segmented point cloud region of the robotic arm and the grasped object. It should be noted that this disclosure does not limit the depth difference threshold and the spatial distance threshold, and adjustments can be made according to the actual situation.

[0035] Specifically, starting from a seed point, a region growing algorithm is executed within the candidate region to obtain the robotic arm's own region (i.e., the segmented point cloud region). The system iteratively expands the seed point within the image neighborhood or local spatial neighborhood. When adjacent pixels meet the condition that the depth difference is less than a preset threshold or the 3D spatial distance is less than a preset range, they are included in the current region. During this process, prior constraints of the robotic arm are introduced to guide region expansion to occur preferentially in regions close to the robotic arm model. When a pixel no longer meets the above growth conditions, growth in that direction stops until the entire region no longer expands. Since the handheld object and the end effector usually have a continuous geometric connection in space, the region growing process naturally extends from the robotic arm to the object it grasps, thereby achieving adaptive segmentation of objects with unknown shapes.

[0036] In some embodiments of this disclosure, the target point cloud is obtained by removing the self-point cloud clusters corresponding to the segmented point cloud regions from the panoramic environmental point cloud generated in the current frame. One possible approach is to back-project the segmented point cloud regions into 3D space to obtain self-point cloud clusters; remove the self-point cloud clusters from the panoramic environmental point cloud to obtain candidate point clouds; and fill the point cloud holes in the candidate point clouds to obtain the target point cloud. Specifically, the pixel clusters identified by region growing in the depth map are back-projected back into 3D space using camera intrinsics to generate self-point cloud clusters. These self-point cloud clusters precisely include the robotic arm entity and any objects it is currently grasping. By using region growing rather than simple geometric clipping, more environmental point clouds close to the arm's edge can be preserved, improving the accuracy of environmental perception. Removing self-point cloud clusters from the panoramic environmental point cloud generated in the current frame yields candidate point clouds; filling the point cloud holes in the removed regions in the spatiotemporal domain, and using historical data from one or more previous frames for background completion, reduces map information loss due to occlusion.

[0037] In some embodiments of this disclosure, the robot performs path planning operations based on a target point cloud. The target point cloud is output to a navigation module, which then searches for a path based on the point cloud free from its own interference. The obstacle avoidance module no longer reacts to the robotic arm itself or the object it holds, effectively preventing accidental triggering.

[0038] Figure 2 This is a schematic diagram of the structure of a robot point cloud processing device 20 provided for an exemplary embodiment of this disclosure. Figure 2 As shown, the robot point cloud processing device 20 includes: an acquisition module 21, a construction module 22, a generation module 23, and a rejection module 24.

[0039] The acquisition module 21 is used to acquire the robot's robotic arm joint state and environmental perception data, and to map the robotic arm joint state to the camera coordinate system corresponding to the environmental perception data based on the coordinate transformation relationship. Module 22 is used to construct candidate segmentation regions corresponding to the robotic arm based on the joint states of the robotic arm in the camera coordinate system. The generation module 23 is used to generate a region within the candidate segmentation area, starting from the seed point, to obtain the segmentation point cloud region of the robotic arm and the grasped object. The culling module 24 is used to remove the self-point cloud clusters corresponding to the segmented point cloud regions from the panoramic environment point cloud generated in the current frame to obtain the target point cloud.

[0040] Optionally, the removal module 24 can also be used for: The robot is controlled to perform path planning operations based on the target point cloud.

[0041] Optionally, the environmental perception data includes: depth map and camera intrinsic parameters. When acquiring the robot's manipulator joint states and environmental perception data, and mapping the manipulator joint states to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships, the acquisition module 21 is used for: The robot's robotic arm joint states are obtained, and the spatial poses of the robotic arm's links and end effector are calculated by calling the forward kinematics model. Obtain depth map and camera intrinsics; Based on the depth map, camera intrinsic parameters, and extrinsic parameters between the camera and the robot base, the joint states of the robotic arm are mapped to the camera coordinate system corresponding to the environmental perception data.

[0042] Optionally, when constructing candidate segmentation regions corresponding to the robotic arm based on the joint states of the robotic arm in the camera coordinate system, the construction module 22 is used for: The robotic arm is represented as a combination of multiple simplified geometric shapes, thus obtaining the corresponding geometric model of the robotic arm; The geometric model is projected onto the depth image plane to obtain the theoretical area occupied by the robotic arm in the image. The theoretically occupied region is expanded to obtain candidate segmentation regions.

[0043] Optionally, when generating the region within the candidate segmentation region, starting from the seed point, the generation module 23 is used to: Depth difference threshold and spatial distance threshold are set as the criteria for determining region growth; Starting from the seed point, search for neighboring pixels within the candidate segmentation region; If the depth difference between a neighboring pixel and the boundary of the current growth region is less than the depth difference threshold, or the distance between a neighboring pixel in three-dimensional space is less than the spatial distance threshold, and the neighboring pixel is within the preset prior constraint range of the robotic arm model at any time, then the neighboring pixel is included in the current growth region and marked as visited. Repeat the neighborhood search and judgment operation until no neighboring pixels that meet the growth conditions can be found, and obtain the segmented point cloud region of the robotic arm and the grasped object.

[0044] Optionally, when the culling module 24 removes its own point cloud clusters corresponding to the segmented point cloud regions from the panoramic environment point cloud generated from the current frame to obtain the target point cloud, it is used to: The segmented point cloud region is back-projected into three-dimensional space to obtain its own point cloud cluster; Remove the point cloud clusters of the same type from the panoramic environment point cloud to obtain candidate point clouds; Fill the gaps in the candidate point cloud to obtain the target point cloud.

[0045] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0046] Figure 3 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. For example... Figure 3 As shown, the electronic device includes a memory 31 and a processor 32. Additionally, the electronic device also includes a power supply component 33 and a communication component 34.

[0047] Memory 31 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.

[0048] The memory 31 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0049] Communication component 34 is used for data transmission with other devices.

[0050] The processor 32 can execute computer instructions stored in the memory 31 to: acquire the robot's robotic arm joint states and environmental perception data, and map the robotic arm joint states to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships; construct candidate segmentation regions corresponding to the robotic arm based on the robotic arm joint states in the camera coordinate system; perform region generation within the candidate segmentation regions starting from the seed point to obtain the segmentation point cloud regions of the robotic arm and the grasped object; and remove the self-point cloud clusters corresponding to the segmentation point cloud regions from the panoramic environmental point cloud generated in the current frame to obtain the target point cloud.

[0051] Accordingly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores a computer program, and the computer program is executed by one or more processors, it causes one or more processors to perform... Figure 1 Each step in the method embodiment.

[0052] Accordingly, embodiments of this disclosure also provide a computer program product, which includes a computer program / instructions that are executed by a processor. Figure 1 Each step in the method embodiment.

[0053] The above Figure 3 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0054] The above Figure 3 The power supply component provides power to the various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0055] The aforementioned electronic devices also include a display screen and audio components.

[0056] The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions, but also the duration and pressure associated with the touch or swipe operation.

[0057] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0058] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0063] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0064] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0065] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0066] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing point clouds in a robot, characterized in that, include: The robot acquires the joint states of its robotic arm and environmental perception data, and maps the joint states of the robotic arm to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships. Based on the joint states of the robotic arm in the camera coordinate system, candidate segmentation regions corresponding to the robotic arm are constructed. Starting from the seed point, region generation is performed within the candidate segmentation area to obtain the segmentation point cloud region of the robotic arm and the grasped object; The target point cloud is obtained by removing the point cloud clusters corresponding to the segmented point cloud regions from the panoramic environment point cloud generated in the current frame.

2. The method according to claim 1, characterized in that, The method further includes: The robot is controlled to perform path planning operations based on the target point cloud.

3. The method according to claim 1, characterized in that, The environmental perception data includes: depth map and camera intrinsic parameters. The process of acquiring the robot's robotic arm joint states and environmental perception data, and mapping the robotic arm joint states to the camera coordinate system corresponding to the environmental perception data based on coordinate transformation relationships, includes: The robot's robotic arm joint states are obtained, and the spatial poses of the robotic arm's links and end effector are calculated by calling the forward kinematics model. Obtain the depth map and the camera intrinsic parameters; Based on the depth map, the camera intrinsic parameters, and the extrinsic parameters between the camera and the robot base, the joint state of the robotic arm is mapped to the camera coordinate system corresponding to the environmental perception data.

4. The method according to claim 1, characterized in that, The process of constructing candidate segmentation regions corresponding to the robotic arm based on the joint states of the robotic arm in the camera coordinate system includes: The robotic arm is represented as a combination of multiple simplified geometric shapes, thus obtaining the geometric model corresponding to the robotic arm; The geometric model is projected onto the depth image plane to obtain the theoretical area occupied by the robotic arm in the image. The theoretically occupied region is expanded to obtain candidate segmentation regions.

5. The method according to claim 1, characterized in that, Starting from the seed point, region generation is performed within the candidate segmentation area to obtain the segmentation point cloud region of the robotic arm and the grasped object, including: Depth difference threshold and spatial distance threshold are set as the criteria for determining region growth; Starting from the seed point, search for neighboring pixels within the candidate segmentation region; If the depth difference between the neighboring pixel and the boundary of the current growth region is less than the depth difference threshold, or the distance between the neighboring pixel in three-dimensional space is less than the spatial distance threshold, and the neighboring pixel is always within the preset prior constraint range of the robotic arm model, then the neighboring pixel is included in the current growth region and marked as visited. Repeat the neighborhood search and judgment operation until no neighboring pixels that meet the growth conditions can be found, and obtain the segmented point cloud region of the robotic arm and the grasped object.

6. The method according to claim 1, characterized in that, The step of removing the point cloud clusters corresponding to the segmented point cloud region from the panoramic environment point cloud generated in the current frame to obtain the target point cloud includes: The segmented point cloud region is back-projected into three-dimensional space to obtain its own point cloud cluster; Remove the point cloud clusters of the same type from the panoramic environment point cloud to obtain candidate point clouds; The candidate point cloud is filled with point cloud holes to obtain the target point cloud.

7. A robot point cloud processing device, characterized in that, include: The acquisition module is used to acquire the robot's robotic arm joint states and environmental perception data, and to map the robotic arm joint states to the camera coordinate system corresponding to the environmental perception data based on the coordinate transformation relationship. The construction module is used to construct candidate segmentation regions corresponding to the robotic arm based on the joint states of the robotic arm in the camera coordinate system. The generation module is used to perform region generation within the candidate segmentation area, starting from the seed point, to obtain the segmentation point cloud region of the robotic arm and the grasped object; The culling module is used to remove the self-point cloud clusters corresponding to the segmented point cloud region from the panoramic environment point cloud generated in the current frame to obtain the target point cloud.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-6.

Images