Method, Medium, System, and Robot for Displaying a Robot Operating Environment

Abstract

The present disclosure provides a method for displaying an operating environment of a robot. The method includes: determining, based on sensing information acquired by the robot, an object type of each of one or more objects in the operating environment of the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least partially based on the object type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface.

Description

Technical Field

[0001] The present invention relates to the field of robotics, and more particularly to a method, a non-transitory computer-readable storage medium, a system, and a robot for displaying an operating environment of a robot. Background Art

[0002] In recent years, robotics has made significant progress, leading to a revolution in automation across multiple industries. A key aspect of robotics is robot modeling, which involves creating a mathematical representation of the robot and its behavior. This enables precise simulation and optimization of robot operations. In the context of industrial automation, robot modeling plays a crucial role in enhancing the manufacturing process by simulating and optimizing robot motion. Real-time simulation using a robot model helps identify and solve potential problems, thereby improving productivity and operational workflows in complex manufacturing environments.

[0003] However, traditional robot modeling techniques mainly focus on kinematics, dynamics, and control algorithms, which are essential for understanding robot behavior and improving performance. However, other factors such as obstacles, terrain changes, and environmental conditions should not be overlooked. Due to the interaction between the robot and its surrounding environment, these factors can significantly affect the robot's decision-making, navigation capabilities, and overall operational efficiency. Therefore, there is a need to develop a method and / or device for displaying the operating environment of a robot. Summary of the Invention

[0004] An object of the present invention is to overcome the drawbacks in the related art and provide a method, a non-transitory computer-readable storage medium, a system, and a robot for displaying an operating environment of a robot.

[0005] According to one aspect of the present disclosure, there is provided a method for displaying an operating environment of a robot. The method includes: determining, based on sensing information obtained by the robot, a type of each of one or more objects in the operating environment of the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least partially based on the type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface.

[0006] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided. Instructions are stored on the non-transitory computer-readable storage medium, and when executed by one or more processors, the instructions cause the one or more processors to perform the following operations, which include: determining an object type of each of one or more objects in an operating environment of the robot based on sensing information acquired by the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least partially based on the object type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface.

[0007] According to yet another aspect of the present disclosure, a system for displaying an operating environment of a robot is provided. The system includes one or more sensors; one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, which include: determining an object type of each of one or more objects in an operating environment of the robot based on sensing information acquired by the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least partially based on the object type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface.

[0008] According to still another aspect of the present disclosure, a robot is provided. The robot includes a robotic arm; one or more sensors, the one or more sensors including an electronic skin; a controller; a display; and one or more storage devices storing instructions that, when executed, cause the controller to perform operations, which include: determining an object type of each of one or more objects in an operating environment of the robot based on sensing information acquired by the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least partially based on the object type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To facilitate a clearer and more direct understanding of the technical solutions presented in the embodiments of the present disclosure, the following will briefly introduce the drawings that are helpful for understanding the embodiments. It should be noted that the drawings discussed in conjunction with the following detailed description are only some illustrations of the present disclosure. Those of ordinary skill in the art can obtain other drawings based on these drawings without additional creative efforts.

[0010] Figure 1 FIG. shows a schematic diagram of an exemplary device according to some embodiments of the present disclosure.

[0011] Figure 2The figure shows a schematic diagram of an exemplary system capable of performing the methods disclosed herein according to some embodiments of the present disclosure.

[0012] Figure 3 The figure shows a schematic flowchart of a method for displaying an operating environment of a robot according to some embodiments of the present disclosure.

[0013] Figure 4 The figure shows a schematic diagram of a robot control system according to some embodiments of the present disclosure, which includes a real physical scene and a corresponding 3D virtual environment according to some embodiments of the present disclosure.

[0014] Figure 5 The figure shows a schematic diagram of a 3D virtual environment according to some embodiments of the present disclosure, in which a representation of the proximity of a non-preset object to the robot is depicted.

[0015] Figure 6 The figure shows a schematic diagram of a 3D virtual environment according to some embodiments of the present disclosure, in which a window for a user to select items to be displayed is shown.

[0016] Figure 7 The figure shows a schematic diagram of a 3D virtual environment according to some embodiments of the present disclosure, in which a process of a robot avoiding a non-preset object is shown.

[0017] Figure 8 The figure shows a schematic diagram of a real physical scene and a corresponding 3D virtual environment according to some embodiments of the present disclosure.

[0018] Figure 9 The figure shows another schematic diagram of a real physical scene and a corresponding 3D virtual environment according to some embodiments of the present disclosure.

[0019] For purposes of simplicity and clarity of illustration, the elements shown in the figures may not be drawn to scale. For clarity, some elements may be exaggerated in size relative to other elements. Additionally, reference numerals may be repeated throughout the figures to indicate corresponding or similar elements as appropriate. Detailed Description

[0020] The terms "first", "second", and "third" used in the embodiments of the present disclosure are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features being discussed. Thus, features described as "first", "second", or "third" may explicitly or implicitly include one or more such features. In the present disclosure, unless otherwise specified, "a plurality" means at least two items, such as two, three, or more. Additionally, the terms "comprising", "including", and "having" and their variants are intended to cover non-exclusive inclusion. For example, a process, method, system, article, or device that includes a series of steps or units is not composed solely of the listed steps or units, but may also optionally include other unstated steps or units, or implicitly include other steps or units inherent to such process, method, article, or device.

[0021] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with the embodiment may be present in at least one embodiment of the present disclosure. The phrase "in one embodiment" that appears at various places in the specification does not necessarily refer to the same embodiment, and separate or alternative embodiments are not mutually exclusive. A person of ordinary skill in the art can understand that the embodiments described herein can be combined with other embodiments found throughout the specification.

[0022] Please refer to Figure 1 , which shows a schematic diagram of an exemplary device according to some embodiments of the present disclosure. The device can be a mechanical device, such as a robot 1000. The robot 1000 can include at least one joint 1010, at least one robotic arm 1020, and at least one electronic skin 1030.

[0023] It should be understood that Figure 1 schematically shows an electronic skin 1030 covering a part of the surface of the robot 1000, but the electronic skin 1030 can also cover the entire surface of the robot 1000.

[0024] The robot 1000 can include a base 1040, which is connected to at least one joint 1010 or robotic arm 1020. The base 1040 can be arranged on or fixed to an operating platform or surface to provide stable operating conditions for the robot 1000. Additionally, the base 1040 can also be movable. For example, one or more drive wheels can be installed at the bottom of the base 1040 to facilitate the mobility of the robot 1000, thereby allowing it to adapt to non-stationary operating situations and increasing its flexibility.

[0025] In this embodiment, at least one end of the robotic arm 1020 is connected to the joint 1010. Each joint 1010 may include, for example, at least one actuating component (not shown), and the actuating component may swing the robotic arm 1020 connected thereto. The robot 1000 may include one joint 1010 and one robotic arm 1020, or may include multiple joints 1010 and multiple robotic arms 1020. The number of joints 1010 and robotic arms 1020 may depend on the design and purpose of the robot 1000 and is not limited herein. When multiple robotic arms 1020 are included, two of the robotic arms may be rotatably connected by joints 1010 connected at their respective ends, and the movement of the robotic arm 1020 may be achieved by the relative rotation of the joints 1010.

[0026] In some embodiments, each robotic arm 1020 may include a metal bracket (not shown) and a robot control system (not shown). The robot control system may be coupled to the electronic skin 1030, and the electronic skin 1030 may be wrapped around the outer surface of the metal bracket. The metal bracket may serve as the metal frame or housing of the robotic arm 1020, providing a place for connecting the electronic skin 1030. It should be noted that the metal bracket is grounded to ensure the normal operation of the robot 1000. The electronic skin 1030 cooperates with the robot control system and is capable of controlling various operations of the robotic arm 1020, such as rotation, swinging, obstacle avoidance, etc.

[0027] In other embodiments, each joint 1010 may include a joint bracket (not shown) and a robot control system (not shown). The robot control system may be coupled to the electronic skin 1030, and the electronic skin 1030 may be wrapped around the outer surface of the joint bracket. Optionally, the material of the joint bracket may be a conductive material such as metal, and the joint bracket may be grounded to ensure the normal operation of the robot 1000. Of course, the rotation and actuation of the robotic arm 1020 may also be achieved through the joint 1010. In this case, the electronic skin 1030 and the robot control system together control both the joint 1010 and the robotic arm 1020 to perform operations such as rotation, swinging, obstacle avoidance, etc.

[0028] It should be understood that the outer surfaces of the joint brackets of at least one joint 1010 and the outer surfaces of the metal brackets of at least one robotic arm 1020 may both be wrapped with at least one electronic skin 1030. Additionally, the robot control system is generally arranged independently of the robot and may include a display and a controller; or it may also be partially arranged, such as the controller, in the base 1040, the robotic arm 1020, and / or the joint 1010.

[0029] Based on research on the modeling technology of robots in related technologies, the inventors of the present disclosure have recognized that modeling the operating environment of a robot is equally important as modeling the robot itself, which typically includes the kinematics, dynamics, and behavior patterns of the robot. This is because the interaction between the robot and its surrounding environment significantly affects the operation, motion trajectory, and performance of the robot. The importance of modeling the operating environment should not be underestimated, as it provides many benefits in various fields such as industrial automation, healthcare and rehabilitation, defense and security, exploration and space missions, entertainment and education, etc.

[0030] Accurate modeling of the operating environment enables the robot to adapt to and effectively respond to its surrounding environment, facilitating obstacle understanding, navigation, collision avoidance, and motion optimization. For example, in industrial automation, accurate modeling of the factory floor layout, equipment arrangement, and potential obstacles enables the robot to effectively handle materials, perform assembly tasks, or participate in quality control. Similarly, in healthcare and rehabilitation, modeling the patient's environment and physical surroundings can play an important role in assisting with daily life mobility, rehabilitation exercises, and activities. The defense and security sectors can also benefit from environmental modeling. By simulating different terrains, building layouts, and potential threats, robots can be deployed efficiently and effectively for surveillance, reconnaissance, or search and rescue missions. In exploration and space missions, accurate environmental modeling can help in planning and executing tasks in remote and inhospitable environments. In summary, environmental modeling optimizes the robot's trajectory and decision-making process, thereby enhancing their capabilities and applications in various fields.

[0031] Figure 2 The figure shows a schematic diagram of an exemplary system capable of performing the methods disclosed herein according to some embodiments of the present disclosure. As Figure 2 shown, the system 200 may include a perception information acquisition module 2100 for obtaining perception information, an object type determination module 2300, a three-dimensional (3D) virtual environment construction module 2500, and a 3D environment display module 2800. The perception information acquisition module 2100 may include an electronic skin (E-skin) unit 2110, a LiDAR unit 2120, a camera unit 2130, a proximity sensing unit 2140, an acoustic detection unit 2150, a motion sensing unit 2160, and a joint motor encoder 2170. In some examples, each of these units may be integrated into the robot, for example, Figure 1 the robot 1 shown. In other examples, some units may be integrated into the robot, for example, Figure 1 the robot 1 shown, while some other units may be arranged separately from the robot.

[0032] The E-skin unit 2110 may be included in the robotic arm of the robot (for example, as Figure 1One or more electronic skins arranged in an array on the (as shown) for sensing the shape of one or more objects present around the robot.

[0033] In the present disclosure, the electronic skin can be interpreted as an electrode array. The electrode array can include at least one electrode unit arranged in an array. The principle of detecting an object with an electrode is that when an object approaches, the capacitance value of the corresponding electrode unit changes, which can be detected by the configured LC oscillation circuit. Initially, each electrode unit can have a specific baseline capacitance value. The proximity of the object can be associated with the difference between the detected capacitance value of the corresponding electrode and the baseline capacitance value. For example, the closer the object is, the larger the detected capacitance value of the corresponding electrode. To better determine the shape of the object, each electrode unit can correspond to a corresponding LC oscillation circuit, thus allowing the capacitance value of each electrode unit to be detected independently. The principle of detecting the shape of an object with an electrode array is that each electrode unit in the electrode array has its own coordinates. When an object approaches, the capacitance values of the electrode units that have changed and their respective coordinates can be used to obtain the fitted shape of the object.

[0034] Specifically, based on the configuration files of all electrode units in the array, including the deviation of the detected capacitance value from the corresponding baseline capacitance value and the coordinate information of the electrode units, a set of point cloud data capable of describing the position coordinates and depth information of the object can be derived to obtain (e.g., calculate) the fitted shape (especially the surface shape) of the object. As a person skilled in the art, depending on the layout number and density of the electrode array, it should be understood that the point cloud data here can also be expressed as position data, stereo grid data. The depth information refers to the distance between the surface of the object (or a part of the object) located within the detection range of the electrode array facing the electrode array and each electrode unit that generates a capacitance change signal.

[0035] In addition, due to the limited obstacle detection range of the electronic skin, in many cases, only a part of the object within the detection range of the electronic skin can be detected, rather than the entire object. Therefore, in order to achieve better 3D modeling of the objects around the robot, it is necessary to delimit and smooth the edges of the detection range. For example, a distance threshold can be predetermined such that the capacitance value of each detected electrode unit is selected only when the distance from the detected electrode unit to the object (e.g., an obstacle) is less than the distance threshold. If some detected distances exceed the distance threshold, it can be indicated that there is a low confidence level for the capacitance values of the relevant electrode units, and these capacitance values can be discarded.

[0036] Although the present disclosure describes the process of detecting an object with a capacitive electronic skin, it should be understood that the electronic skin can take any other suitable form, such as a resistive electronic skin, a conductive electronic skin, an inductive electronic skin, etc. The present disclosure does not impose any restrictions in this regard.

[0037] In some examples, if the E - skin unit 2110 is only used to detect the proximity of an object, such detection can also be accomplished by a single electrode unit.

[0038] In some examples, the LiDAR unit 2120 can be used to help determine the depth information of the objects around the robot or the closest distance to the robot (e.g., the closest component).

[0039] In some examples, the camera unit 2130 can be used to help determine the depth information of the various objects in the environment around the robot, especially when the camera unit 2130 includes a depth camera, a stereo camera, an infrared camera, a time - of - flight (TOF) camera, etc. or a combination thereof that can capture information about depth of field, object distance, shape, and / or position to enable tasks such as object detection, pose estimation, and scene reconstruction in various application scenarios. In an example, the viewing / capturing / sensing range of the camera unit 2130 can be greater than the detection range of the E - skin unit. It can be understood that the purpose of including the LiDAR unit 2120 and / or the camera unit 2130 can be, for example, to replace obtaining a modeled representation of the operating environment by importing a modeling file or manually inputting parameters by the user, and the operating environment where the robot is located can be scanned on - site by means of such sensing units for pre - modeling of the operating environment.

[0040] In some examples, the proximity sensing unit 2140 can be used to help determine the proximity of the surrounding objects to the robot. In some cases, the proximity sensing unit 2140 can be equipped with advanced algorithms for detecting and tracking moving objects, thus allowing the robot to dynamically adjust its path and behavior to ensure effective and reliable interaction with its surrounding environment.

[0041] In some examples, the acoustic detection unit 2150 can be used to capture the acoustic signals of the various objects around the robot for determining the orientation, position, and / or direction of movement of these objects, especially when the objects have different acoustic characteristics. This helps to efficiently determine the distance, shape, and / or contour of the various objects by some other units (such as the E - skin unit 2110, the LiDAR unit 2120, or the proximity sensing unit 2140).

[0042] In some examples, the motion sensing unit 2160 may include an accelerometer, a gyroscope, a magnetometer, etc., or a combination thereof, to provide comprehensive motion sensing capabilities for obtaining the motion state of the robot. By integrating the measurement data from the motion sensors included in the motion sensing unit 2160, the perception information acquisition module 2100 can be capable of obtaining key motion information in real time, such as the acceleration, angular velocity, attitude, and motion direction of the robot, thereby helping to support tasks including robot navigation, positioning, path planning, control, etc.

[0043] In some examples, the joint motor encoder 2170 can be used to obtain the rotation data of the robotic arm joints, encode the obtained rotation data of the robotic arm joints, and provide it (e.g., in real time or at regular intervals) to, for example, the three-dimensional (3D) virtual environment construction module 2500 described below, so as to realize the display of the robotic arm (e.g., display based on modeling). According to some embodiments, the 3D virtual environment described herein is not limited to the first-person perspective graphical representation, but can also be represented (e.g., displayed) from the third-person perspective according to the specific application and / or use of the robot.

[0044] These units included in the perception information acquisition module 2100 can operate in coordination with each other to effectively and comprehensively obtain the perception information about the environment around the robot and the objects present therein. It should be understood that the above units are provided only for illustrative purposes. Those of ordinary skill in the art can easily understand that the perception information acquisition module 2100 may include more, fewer, or different units than Figure 2 shown. For example, in some cases, the perception information acquisition module 2100 may include a temperature sensing unit. The temperature sensing unit can be used to determine the temperature distribution of various objects and / or regions around the robot. This provides additional information about the impact of these objects on the safety of the robot, especially when the robot is operating in a hazardous terrain or in a high-risk industrial production environment. In some cases, the perception information acquisition module 2100 may include a gas sensing unit to detect the presence of combustible / hazardous gas leakage in the environment where the robot operates, which is particularly useful in cases involving robot operation in an unknown environment. The present disclosure does not limit the specific type and / or number of sensing units included in the perception information acquisition module 2100.

[0045] Then, based on the information obtained by the perception information acquisition module 2100, the types of objects around the robot can be identified, thereby enabling the construction and differentiation of three-dimensional models of these objects. This helps to obtain an enhanced modeled representation of the three-dimensional environment in which the robot interacts with the objects included therein to perform operations associated with a given task in the environment.

[0046] In some embodiments of the present disclosure, system 200 may include a signal processor module 2200. In some examples, the signal processor module may be a general-purpose signal processor (such as a digital signal processor (DSP) or a system-on-chip (SoC), etc.), a dedicated signal processor (such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an image processor, an audio processor, or a video processor, etc.), or a combination thereof. The signal processor module 2200 may be responsible for processing various types of perception information acquired by the perception information acquisition module 2100. For example, when the perception information is point cloud data, the processing may include point cloud segmentation, point cloud completion, point cloud detection, point cloud registration, etc. When the perception information is in the form of an image signal, the processing performed by the signal processor module 2200 may include image processing operations, such as scaling, rotation, segmentation, enhancement, compression, etc. For sound signals, the processing may include audio signal processing techniques, such as time-frequency domain transformation, audio quality enhancement, noise reduction, etc.

[0047] Based on the received perception information (e.g., raw or processed perception information), different types of objects present in the space where the robot is located are identified, so as to distinguish the three-dimensional models of the objects to be manipulated or avoided by the robot. In some examples, different object types may include preset objects and non-preset objects. By way of example and not limitation, the preset objects may be explicitly or implicitly specified by a user (e.g., the operator or developer of the robot or system 200), or may be pre-stored objects imported into the three-dimensional virtual environment, such as a robotic arm or a part thereof, a workbench, a tool, a workpiece to be processed, etc. In addition, the non-preset objects may refer to objects not specified by the user, or may be objects not imported into the three-dimensional virtual environment in advance, such as moving objects, people, or other obstacles randomly appearing in the operating environment of the robot, etc. In some cases, the workpiece to be processed may be a preset object, which may be explicitly or implicitly specified by the user, or may be pre-imported into the three-dimensional virtual environment via a modeling file.

[0048] The object type determination module 2300 may receive the processed perception information from the signal processor module 2200 and / or may directly receive the raw (e.g., unprocessed) perception information from the perception information acquisition module 2100. As Figure 2 shown, the object type determination module 2300 may include a comparison module 2310 and a preset template object database 2320. During the actual operation of the robot, after the perception information acquisition module 2100 acquires the perception information about the objects in the environment where the robot is located, the object type determination module 2300 may determine whether there are any preset objects in the environment.

[0049] In some examples, based on the perception information obtained by the robot, when the position data and / or surface shape in the perception information match the position data and / or surface shape of a preset template object, the matched position data and / or surface shape are recognized as belonging to the preset object. As used herein, for the E-skin unit 211, the position data in the perception information may refer to the coordinates of an object detected in the operating environment of the robot, which can be understood as the coordinates of some discrete key feature points on the surface of the object, or as a set of coordinate points constituting the surface of the object. For the position data of the pre-stored preset template object, it can be understood as the coordinates of discrete points such as the key feature points on the surface of the object, or as a set of all points constituting its 3D model. As used herein, on the one hand, for the perception information collected by the E-skin unit 2110 of the perception information acquisition module 2100, for example, the surface shape refers to the surface of an object (or a part of the object) within the detection range of the E-skin unit 211 facing the E-skin unit 211. This surface shape can be calculated based on the spatial coordinates of each electrode unit that generates a capacitance change signal and the detected distance of the object, and is represented in the form of point cloud data, stereo mesh data, single position data, or a local shape description based on features, or a smooth surface representation obtained through fitting processing; on the other hand, for the preset template object, the surface shape refers to information such as the shape, boundary, and / or position of the preset template object obtained from, for example, a modeling file imported by the user or size parameters manually input, which can describe the surface shape of the object. The representation form of this surface shape is preferably consistent with the representation form of the surface shape of the object obtained by perception for similarity comparison. If a match is found in terms of position data and / or surface shape, the perception information can be attributed to the preset template object, and the object corresponding to the perception information can be classified as a preset object. It can be understood that the match mentioned herein means that if the position data detected by the E-skin unit 2110 has a spatial overlap with the position data of any one or more objects in the pre-stored preset template object, the position data of the overlapping part is recognized as belonging to the corresponding preset template object; and / or if the similarity of the sensed surface shape to the surface shape of any one or more of the pre-stored preset template objects is greater than a preset threshold, the surface shape of the part with a similarity greater than the preset threshold is recognized as belonging to the corresponding preset template object. This shape matching can be achieved by comparing features such as the contours, edges, or key points of the objects being compared. The threshold for matching can be adjusted based on the requirements of a specific application to achieve the desired accuracy. Additionally, the position and surface shape of the object can be mutually verified to ensure the accuracy of recognition.

[0050] In some examples, data of the preset template object (e.g., location data and / or surface shape of the preset template object, etc.) may be stored in the preset template object database 2320 in the form of entries. In some examples, the preset template object database 2320 may be co-located with the comparison module 2310 in the object type determination module 2300. In some examples, the preset template object database 2320 may be separated from the object type determination module 2300 (e.g., located remotely). In some examples, the storage 2400 may be deployed on, for example, a cloud server. Optionally, the preset template object database 2320 may receive new entries or updated entries about the preset template object from the storage 2400. In some examples, existing entries in the preset template object database 2320 may be loaded from the storage 2400.

[0051] Based on the perception information acquired by the robot, when the position data and / or surface shape in the perception information does not match the position data and / or surface shape of all objects in the preset template object, the position data and / or surface shape that does not constitute a match is identified as belonging to a non-preset object. It should be noted that the perception information may include position data (and / or surface shape) belonging to the preset template object and position data (and / or surface shape) belonging to the non-preset object, and the position data and / or surface shape that does not constitute a match may be only a part of the perception information. As an example and not a limitation, it can be understood that when the sensed position data does not constitute a spatial overlap with any preset template object, and / or the similarity between the sensed surface shape and the surface shape of all preset template objects is lower than a preset threshold, it is considered that it does not constitute a match. In some cases, an object whose object type is a non-preset object can be stationary or non-stationary.

[0052] In some cases, the matching mentioned in this article may also refer to that when the robot imports a preset template object model, the perception information acquisition module is taught to only acquire perception information that excludes the position and surface shape of the preset template object, and the perceived object corresponding to this perception information is directly identified as a non-preset object.

[0053] In some examples, a modeling file of the robot's operating environment may be first imported in order to obtain information about objects in the default physical environment; then, the robot's surrounding environment and objects therein are perceived through the various units included in the perception information acquisition module (such as the E-skin unit 2110, etc.); then, the information of objects in the default physical environment is compared / matched with the perceived object information to identify which are imported objects, followed by determining the object type (for example, preset object type versus non-preset object type) and / or modeling of non-imported objects.

[0054] like Figure 2As shown, one or both of the perception information acquisition module 2100 and the object type determination module 2300 can be communicatively connected to the three-dimensional (3D) virtual environment construction module 2500, thereby allowing the 3D virtual environment construction module 2500 to individually model one or more objects in the robot environment based on the acquired perception information and the determined object type. In some examples, the signal processor module 2200 can also be communicatively connected to the 3D virtual environment construction module 2500 to provide processed perception information, although this connection is not shown in the figure.

[0055] In some examples, the 3D virtual environment construction module 2500 can include a proximity rendering unit 2510, a user selection unit 2520, an object display unit 2530, a safe operation area determination unit 2540, and a motion trajectory display unit 2550.

[0056] By way of example and not limitation, objects in the three-dimensional virtual environment can be annotated with their distance from the robot's corresponding e-skin (e.g., the closest e-skin) and / or can be distinguished by color to indicate proximity. For example, objects can be divided into N distance ranges, and each range is associated with a different color for the three-dimensional modeling of objects falling within that distance range. The color depth, chromaticity, transparency, or other optical / visual characteristics of the object model can be configured to be related to the distance, where closer objects appear darker. In some cases, when an object is large in size, it can span multiple distance ranges, so that the color of the object is rendered with different depths, chromaticities, or transparencies, etc., according to each distance range spanned by the object. This can also provide an intuitive hint for safe operation. It should be understood that any other suitable graphical means for indicating the proximity between the object and the robot can be employed, which is not limited herein.

[0057] By way of example and not limitation, if the detected perception information is identified as belonging to a preset object, the module can display the three-dimensional model of the object according to a preset rule. In general, after the modeling file of the environment where the robot is located is imported, all preset template objects are preferably directly displayed in the 3D virtual environment. Therefore, when the object type determination module 2300 determines that a certain object type is a preset object based on the acquired induction information, it is default not to perform repeated modeling and display of the object. In some cases, once an object is determined to be a preset object, the 3D model of the object is displayed in the 3D virtual environment (e.g., according to a preset rule).

[0058] In some examples, the proximity rendering unit 2510 may be configured to render the color depth, chromaticity, transparency, or a combination thereof of the models of the sensed objects based on the distance between each of the sensed objects and the respective closest component of the robot. Additionally or alternatively, the proximity rendering unit 2510 may indicate the distance between the sensed objects and the respective closest component of the robot.

[0059] By way of example and not limitation, the preferred way to display preset objects in a 3D virtual environment is to display all of these objects. However, options may be provided in the display interface to allow the user to select which preset objects to display and which not to display. For non-preset objects, options may also be presented on the display interface to allow the user to select whether to display the 3D model or representation of a non-preset object and which 3D model or representation of the non-preset object to display.

[0060] In some examples, the user selection unit 2520 may be configured to provide options on the display interface for selecting any object identified as a preset object and / or any object identified as a non-preset object that exists in the physical environment in which the robot is located for display in the corresponding 3D virtual environment.

[0061] In some examples, one of the strategies for displaying a three-dimensional model of an object may include determining whether it belongs to a robotic arm or a part thereof based on (e.g., raw and / or processed) perception information. In some examples, determining whether the perception information belongs to a robotic arm or a part thereof may include comparing, for example, position data sensed by an electronic skin and / or surface shape with robotic arm pose information (e.g., relative positions and angles of each robotic arm component) and a kinematic model, and so on. In some examples, in the case where it is determined that the perception information belongs to a robotic arm or a part thereof, it may not be repeatedly displayed.

[0062] In some examples, one of the strategies for displaying a three-dimensional model of an object may also include displaying the object only when it is sensed (e.g., determined) by the electronic skin as a moving object.

[0063] In some examples, one of the strategies for displaying a three-dimensional model of an object may include continuously displaying the object when it is confirmed to be a stationary object after perception. The display strategy for stationary objects may include displaying the object relative to a (e.g., fixed) reference point in three-dimensional space, such as the case where a certain tool is fixed under some working conditions. In some cases, when planning the motion trajectory of the robot, stationary objects may be considered as obstacles.

[0064] In some examples, one strategy for displaying a three-dimensional model of an object may include associating the attributes of the 3D model of the object being displayed (e.g., size, integrity of details, clarity of contours, etc.) with the distance between the object and the robot (e.g., the e-skin).

[0065] In some examples, one strategy for displaying a three-dimensional model of an object may include directly inputting the 3D model of the object into the system by a control program or a user, retrieving the model representation (e.g., implicitly) from a cloud memory (e.g., storage 2400), or calculating a fitted model of the object based on the sensed information of the object collected from sensors (e.g., e-skin, camera, proximity sensor, etc.).

[0066] By way of example and not limitation, whether the sensed information belongs to the identification of a preset tool may be based on the position information of the object sensed by, for example, the e-skin, the position information of the robotic arm collected by, for example, internally integrated sensors, and the determination of the relative position between the object and the robotic arm. The preset tool may include an end effector or a pipeline (e.g., a trailing cable) that follows the movement of the robotic arm, etc., which is not limited herein. The trailing cable connected to the robot may be visible in the 3D virtual environment. In some cases, the trailing cable may be arranged on the body of the robot to provide power (e.g., for backup power) or provide backup control and data communication (e.g., as a device for wired communication). In some situations, the trailing cable may communicate with the mechanical gripper. It should be understood that the functionality of the trailing cable may be adapted to a specific application and / or operating environment, and the present disclosure does not impose any restrictions thereon. During the operation of the robot in a physical environment, the representation of the trailing cable may remain within the field of view in the first-person / third-person perspective and be substantially stationary relative to at least a part of the robot (e.g., relative to the end portion of the robotic arm on which the mechanical gripper is mounted). In some examples, the trailing cable may be a pre-imported template object. In some examples, due to different requirements of the operation scenario and / or task of the robot, the trailing cable may be a non-preset object. In other words, the graphical representation of the trailing cable in the 3D virtual environment may not be based on any imported modeling-related files, but an approximation (e.g., a fitted representation) obtained from processing the sensed information acquired by one or more sensors of the robot. It should be understood that the follower object may include any other suitable objects that may move with the robot or a part thereof and / or be constrained to the robot body. For example, the follower object may include, but is not limited to, wheeled, tracked, or articulated drive mechanisms, which are not limited herein.

[0067] In some examples, it is possible to determine whether the position data in the perception information belongs to a preset tool based on the position data in the perception information, the pose of the robot, and the preset position relationship between the preset tool and the body model of the robot. As used herein, the body model of the robot refers to a model representing the main structure or body part of the robot system, which may include components such as robotic arms, joints, chassis, actuators, etc. that make up the robot's body. The body model can define the geometric shape, degrees of freedom of movement, motion posture, etc. of the robot. By representing the robot in the modeling environment with the body model, the robot in the physical environment can be manipulated to complete various tasks, such as moving, grasping, operating objects, etc. In the example, when it is determined that the perception information belongs to a preset tool, the object can be displayed in the 3D virtual environment in a manner that is stationary relative to at least a part of the robot. Thereby, more intuitive environment rendering, real-time updating of the object state, and functions such as supporting robot operation and path planning can be provided.

[0068] In the case where there are non-preset objects in the robot environment, compared with the case where only preset objects exist, the path planning for the robot to avoid obstacles becomes more complex. This complexity stems from the need to consider these non-preset objects and other preset objects simultaneously. To solve this problem, a method of determining a safe operation area based on the positions and shapes of the preset objects can be followed. When a non-preset object enters (e.g., appears in) the environment (e.g., a 3D virtual environment, or more specifically, the determined safe operation area), an obstacle avoidance path can then be planned based on the non-preset object to ensure that the planned path remains within the safe operation area.

[0069] In some examples, the safe operation area determination unit 2540 can be configured to determine the safe operation area of the 3D virtual environment based on the preset template objects in the 3D virtual environment.

[0070] In addition, the safe operation area determination unit 2540 can include a display subunit 2542 for preparing to display the determined safe operation area in the 3D virtual environment, e.g., formulating a strategy for displaying the safe operation area (e.g., color, line shape, shadow, etc. or a combination thereof).

[0071] In some examples, displaying the planned motion trajectory (e.g., for a robotic arm) can include: (a) deviating the displayed motion trajectory to bypass objects (including preset and non-preset objects), which ensures that the planned motion avoids colliding with any obstacles in the path; (b) making the displayed motion trajectory closer to certain objects, such as the object to be grasped, which allows precise positioning and interaction with specific objects of interest; (c) displaying path points along the motion trajectory, including the starting point, ending point, and intermediate points, which provides a visual representation of the planned trajectory and helps understand the expected motion of the robot or allows the user to modify the planned trajectory, etc.

[0072] In some examples, the motion trajectory planning module 2600 can be deployed in the system 2000. The motion trajectory planning module 2600 can be configured to plan the motion trajectory of the robot. In some cases, the motion trajectory can be determined to be within the safe operation area while avoiding non-preset objects. In some examples, the planned trajectory can always remain within the safe operation area, thus ensuring the safety of robot operation.

[0073] In some examples, the motion trajectory display unit 2550 can be configured to receive one or more planned motion trajectories of the robot for displaying them together with the determined safe operation area on the display interface.

[0074] In some examples, the 3D environment display module 2800 can be configured to display the 3D model of the constructed object, the 3D virtual environment corresponding to the physical environment where the robot is located, the determined safe operation area, and / or the planned motion trajectory of the robot, etc. In some examples, the 3D environment display module 2800 can include a graphical user interface (GUI) 2802 to provide display functionality.

[0075] In some examples, documents related to modeling can be imported into the control unit of the robot (such as a controller, a microprocessor, or the system 200, etc.) to construct a 3D operation environment. These documents can be, for example, CAD documents or other relevant files. The CAD documents can describe the 3D models of the preset objects, including their shapes and position coordinates.

[0076] In some examples, the CAD documents can be stored in the storage 2400.

[0077] In some examples, the real-time robot pose module 2700 can be deployed in the system 200 and configured to receive the CAD documents from the storage 2400 for real-time display when the detected object is recognized as the robot or a part thereof. In some cases, the robot body or a part thereof can be displayed in real time after importing the modeling-related files of the robot.

[0078] By way of example and not limitation, the robot (such as its robotic arm) can also autonomously detect the objects around it. Due to the swing range of the robotic arm (e.g., the angular range of the arm), and the ability of the robot to carry corresponding vision tools (such as depth cameras, TOF cameras, or other stereo vision tools, etc.), electronic skin, LiDAR, or to use externally attached vision tools or radars to detect the real-time shapes and positions of the objects in the environment, the robot itself can perform modeling operations on the physical operation environment (including the objects therein).

[0079] By way of example and not limitation, modeling-related documents can be imported, mainly including the shape and position information of preset objects. After detecting / identifying the shape and coordinates of an object through an e-skin or other visual assistive device, the identified shape and position can be matched with the corresponding shape and position information in the modeling-related documents. A successful match allows the use of the shape and position information of the corresponding object from the modeling-related documents. In the case of an unsuccessful match, the shape and position of the object detected by sensing can be verified and adjusted (possibly including correction). Accordingly, the resulting shape and position can be used to display the object.

[0080] It should be understood that the purpose of constructing a 3D model of the operating environment is to display the operating environment in a 3D virtual form, obtain knowledge about the working environment of the robotic arm (including the working environment of different types of objects therein), plan the movement trajectory of the robot (or its robotic arm) to avoid objects therein, and determine a safe operating area for the real-time simulation and / or remote manipulation of the robot.

[0081] By way of example and not limitation, when planning the movement trajectory of a robotic arm, an appropriate expansion factor, also known as a safety distance, can be set based on the conditions of the working environment in which the robot operates (e.g., whether the environment is crowded, etc.). For example, since preset objects are usually stationary, the robotic arm can maintain a relatively small but safe distance from these identified preset objects to bypass them (e.g., avoid colliding with them), thereby maximizing the safe movement range of the robot as much as possible. For example, when there are multiple preset objects close to each other, setting a relatively small safety distance for the preset objects can prevent the planned trajectory from being unable to pass between adjacent preset objects, resulting in a situation where the operation trajectory of the robot cannot be planned, which is beneficial to improving the mobility of the robot while ensuring safe operation.

[0082] Figure 3 The figure shows a schematic flowchart of a method for displaying the operating environment of a robot according to some embodiments of the present disclosure. As Figure 3As shown, method 3000 begins at 3002, where the object type of each of one or more objects in the operating environment of the robot can be determined based on the perception information acquired by the robot. Then, at 3004, a three-dimensional (3D) virtual environment of the operating environment can be constructed at least in part based on the object type of each of the one or more objects. Thereafter, at 3006, the 3D virtual environment of the operating environment can be displayed via a display interface. In this way, this allows for the display of three-dimensional models of the robot or a part thereof (e.g., the robotic arm) and preset objects, as well as three-dimensional models of non-preset objects detected around the robot. This facilitates path planning of the robot and effectively prevents collisions. By visualizing these models, the operator or researcher can understand the position and orientation of the objects of interest, thereby enabling safe navigation of the robot. The robot can also adjust its motion trajectory based on the detected objects and their sensed characteristics, thus ensuring efficient and conflict-free operation.

[0083] Figures 4 to 8 Depict exemplary three-dimensional virtual (e.g., simulated) scenes in accordance with some exemplary embodiments disclosed herein. Like reference numerals in these figures may represent the same or similar elements. For example, reference numerals with the same number starting from the right side may represent the same or similar elements throughout the various figures.

[0084] Figure 4 Illustrate a schematic diagram of a robot control system in accordance with some embodiments of the present disclosure.

[0085] As Figure 4As shown, an exemplary robot control system may include a robot 4000b, a controller 4910, and a display 4900. The robot 4000b is located in a real physical environment 4920 and may include a mechanical gripper 4010b for grasping a workpiece 4100b to be processed placed on a workbench 4200b in the physical environment 4920. Correspondingly, in the display 4900, a 3D virtual representation 4000a corresponding to the robot 4000b (including a 3D virtual representation 4010a corresponding to the mechanical gripper 4010b, etc.), a 3D virtual representation 4200a corresponding to the workbench 4200b, and a 3D virtual representation 4100a of the workpiece 4100b to be processed, etc. are displayed. As an example, the robot 4000b (or a part thereof, such as the mechanical gripper 4010b), the workbench 4200b, and the workpiece 4100b to be processed may be preset objects. For example, they are modeled as template objects and pre-stored in a modeling file for displaying a pre-modeled 3D environment representation on the display 4900 before the robot senses the surrounding environment and manipulates the objects therein. In some examples, the controller 4910 facilitates communication between the display 4900 and the robot 4000b (e.g., via a wired line or a wireless communication link), thereby facilitating user guidance of the on-site operation of the robot 4000b. As shown in the figure, when a non-preset object (e.g., the hand 4400b of a human 4400) appears in the physical environment 4920, a 3D virtual representation 4400a of the contour of the non-preset object 4400b sensed by a sensor (e.g., an electronic skin) of the robot 4000b is correspondingly shown in the display 4900. In the example, the 3D virtual representation 4400a may only show a part of the non-preset object (e.g., the part corresponding to the hand 4400b of the human 4400), rather than showing the entire non-preset object. Additionally, the 3D virtual representation 4400a may reflect the proximity between the non-preset object (or a part thereof) and the 3D virtual representation of the robot. For example, the smaller the transparency of the part of the non-preset object closer to the robot, the closer it is to the robot.

[0086] Figure 5 The figure shows a schematic diagram of a 3D virtual environment according to some embodiments of the present disclosure in which the proximity between a non-preset object and a robot is depicted.

[0087] In Figure 5 the 3D virtual environment, a 3D virtual representation 5000a corresponding to the robot in the real physical environment (including a 3D virtual representation 5010a of the mechanical gripper), a 3D virtual representation 5200a of the workbench, and a 3D virtual representation 5100a of the workpiece to be processed placed thereon are included.

[0088] As Figure 5As shown, the proximity between the 3D virtual representation 5400a of the non-preset object and the 3D virtual representation 5000a of the robot can be represented with different transparencies according to the distance magnitude between the two. That is, when the hand in the physical environment gets closer to the robot, the hand part is rendered with a smaller transparency in the 3D virtual environment to remind the user in front of the display 4900 to manipulate the robot (e.g., the mechanical gripper corresponding to the 3D virtual representation 5010a) in the real physical environment to avoid obstacles from the human hand, so as to prevent potential safety accidents. The transparency can be set as needed. For example, the transparency of the part exceeding the maximum predetermined distance L3 is 100%, that is, it indicates that this part is not displayed at all. The distance between the hand, especially the end point closest to the robot, and the 3D virtual representation of the robot is defined as the minimum distance L1 between the two. Therefore, a certain distance L2 between L1 and L3 is used as the threshold with a transparency of 50%. It can be understood that there are other graphical representations for indicating the distance between the non-preset object and the robot, and the present disclosure does not impose any restrictions on this.

[0089] Figure 6 The figure shows a schematic diagram of a 3D virtual environment according to some embodiments of the present disclosure in which a window for a user to select items to be displayed is presented.

[0090] As shown in the figure, the user can select (e.g., select more than one item) the item(s) to be displayed in the 3D virtual environment through the displayed window. Additionally, the user can add (e.g., customize) the types of items to be displayed through the interface and then select them for display, and so on.

[0091] Figure 7 The figure shows a schematic diagram of a 3D virtual environment according to some embodiments of the present disclosure in which the process of the robot avoiding a non-preset object is shown.

[0092] As shown in the figure, three graphics showing the 3D virtual representation of the robot are presented, namely 7000a, 7000b, and 7000c. The graphic 7000a with a solid outline corresponds to the robot at the current moment, while the transparent graphics 7000b and 7000c represent the robot at future moments. When the robot senses an obstacle (such as a human hand) in the physical environment, a 3D virtual representation 7400 of the human hand is displayed in the 3D virtual environment representation. As shown in the figure, the graphic 7000c represents the virtual representation of the robot in the operation trajectory planned by the robot controller or the user in the absence of non-predefined objects. When a non-predefined object appears near the robotic arm in the real physical environment, for example, in the 3D virtual environment representation, the sensed contour 7400 of the non-predefined object is displayed at this moment, while the graphic 7000b represents that the robot has avoided the obstacle (i.e., the 3D contour 7400 of the human hand) in this case. This allows the user to manipulate the robot to avoid the 3D virtual representation 7400 of the human hand according to the (e.g., real-time) modeled representation on the display, and thus ultimately successfully grasp the workpiece to be processed (e.g., shown as the 3D virtual representation 7100) located on the 3D virtual representation 7200 of the workbench.

[0093] Figure 8 The figure shows a schematic diagram of a real physical scenario and a corresponding virtual 3D virtual environment according to some embodiments of the present disclosure; and Figure 9 The figure shows another schematic diagram of a real physical scenario and a corresponding virtual 3D virtual environment according to some embodiments of the present disclosure.

[0094] As Figure 8 shown, when the robotic arm (as a pre-stored predefined template object) approaches the workbench, on the display interface, for demonstration purposes, the part sensed by the robot's sensor (such as e-skin) (such as a corner of the workbench) can be displayed in a blurred manner. Similarly, as Figure 9 shown, when the robotic arm (as a pre-stored predefined template object) approaches its own body, on the display interface, for demonstration purposes, the sensed body part can be displayed in a blurred manner. This is particularly beneficial for prompting the user standing in front of the display to pay attention to avoiding nearby objects (such as fixed objects or parts of the robot itself) during robot manipulation to avoid damaging the (multiple) components of the robot and / or objects in the physical environment.

[0095] The provided embodiments are for illustrative purposes only and are intended to highlight the different features of the claims. The features shown in a particular example are not limited to that particular example and can be combined with other examples. It should be understood that the claims are not limited to any specific example. The description of the methods and process flowcharts is provided as an illustrative example and does not imply a particular order of execution of the blocks. The order of the blocks can be changed in any order, and terms such as "after", "then", or "next" are purely indicative and do not impose any limitation on the order. The articles "a", "an", or "the" used to refer to claim elements in the singular should not be construed as limiting the element to a single instance.

[0096] The illustrative logic, logic blocks, modules, circuits, and algorithmic blocks described can be implemented using electronic hardware, computer software, or a combination of both. The choice between hardware and software implementations depends on the particular application and design constraints. A person of ordinary skill in the art can adopt different approaches to implement the described functionality for each particular application without departing from the scope of the present disclosure. The hardware for implementing these components can include a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or various combinations designed to perform the functions disclosed. The general-purpose processor can be a microprocessor or any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, or a hybrid of a microprocessor and a DSP core. Alternatively, specific blocks or methods can be performed by circuitry specifically designed for a particular function.

[0097] The foregoing description enables others to make or use the disclosed subject matter. Modifications to the embodiments are apparent, and the basic principles can be applied to other embodiments without departing from the spirit or scope of the foregoing description. Accordingly, the foregoing description should not be construed as strictly limited to the embodiments shown, but rather should be interpreted in the broadest scope consistent with the disclosed principles and novel features. In this document, references to elements in the singular form do not imply exclusivity, but rather include "one or more", unless explicitly stated otherwise. Similarly, unless otherwise explicitly stated, the term "some" refers to one or more instances. All structural and functional equivalents of the elements described in the foregoing description are expressly incorporated by reference and are intended to be covered by the claims. Furthermore, the disclosure herein should not be construed as dedicating the disclosed subject matter to the public, regardless of whether such disclosure is explicitly recited in the claims. The claims should not be construed as means-plus-function, unless the phrase "means for" is used to expressly recite the element. It should be understood that the specific order or hierarchy of the blocks in the disclosed processes is for illustrative purposes only. The order or hierarchy of the blocks in the process can be rearranged according to design preferences, while remaining within the scope of the previously described. The appended method claims present the elements of the various blocks in a sample order, but they are not limited to that specific order or hierarchy.

[0098] If the disclosed subject matter is implemented as a product, it can be stored in a computer-readable storage medium. This understanding allows the technical methods presented in this disclosure to be partially or fully implemented as a software product. Alternatively, the software product can implement a part of the technical methods that benefit from traditional technologies. The software product can be stored in storage media (non-volatile and volatile), including but not limited to USB drives, portable hard drives, ROM, RAM, floppy disks, EPROM, EEPROM, optical disc memories, magnetic disc memories, or any other medium capable of storing program code in the form of instructions or data structures and accessible by a computer or machine. The term "disc" as used herein includes various types of storage media, such as compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs. Discs typically rely on magnetic technology for data reproduction. In contrast, the term "diskette" also includes these aforementioned storage media, but specifically denotes its optical data reproduction ability by using a laser.

[0099] Other aspects of the present disclosure are described in the following enumerated Exemplary Embodiments (EEE).

[0100] EEE 1. A method for displaying an operating environment of a robot, comprising: determining an object type of each of one or more objects in the operating environment of the robot based on sensing information acquired by the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least partially based on the object type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface.

[0101] EEE 2. The method according to EEE 1, wherein one or more electronic skins are provided on the robot, and the sensing information includes signals generated by the one or more electronic skins in response to the one or more objects approaching the robot.

[0102] EEE 3. The method according to EEE 1 or EEE 2, wherein the object type includes a preset object and / or a non-preset object.

[0103] EEE 4. The method according to any one of EEE 1 to EEE 3, wherein determining the object type of each of the one or more objects includes: based on the sensing information acquired by the robot, when the position data and / or surface shape in the sensing information match the position data and / or surface shape of a preset template object, the matching position data and / or surface shape are recognized as belonging to a preset object.

[0104] EEE 5. The method according to any one of EEE 1 to EEE 4, wherein determining the object type of each of the one or more objects includes: based on the sensing information acquired by the robot, when the position data and / or surface shape in the sensing information do not match the position data and / or surface shape of all objects in the preset template object, the non-matching position data and / or surface shape are recognized as belonging to a non-preset object.

[0105] EEE 6. The method according to any one of EEE 1 to EEE 5, wherein constructing the 3D virtual environment of the operating environment includes: modeling the non-preset object based on the sensing information acquired by the robot, wherein the model color depth, chromaticity, transparency or a combination thereof of the non-preset object varies with the distance between the non-preset object and the nearest component of the robot, and / or the distance between the non-preset object and the nearest component of the robot is indicated.

[0106] EEE 7. The method according to any one of EEE 1 to EEE 6, wherein displaying the 3D virtual environment of the operating environment via the display interface includes: providing, via the display interface, an option for selecting any one of the objects of the object type of the preset object and / or any one of the objects of the object type of the non-preset object among the one or more objects for display on the display interface.

[0107] EEE 8. The method according to any one of EEE 1 to EEE 7, wherein the preset template object includes a preset tool, and wherein the method further includes: determining whether the position data belongs to the preset tool based on the position data in the perception information, the pose of the robot, and a preset positional relationship between the preset tool and the body model of the robot.

[0108] EEE 9. The method according to any one of EEE 1 to EEE 8, wherein the preset template object includes a part of the robot, and wherein the method further includes: determining whether the perception information belongs to the part of the robot based on the position data and / or the surface shape in the perception information and the pose of the robot.

[0109] EEE 10. The method according to any one of EEE 1 to EEE 9, wherein the perception information includes the pose information of the robot, and wherein the method further includes: displaying at least a part of the robot in real time via the display interface based on the imported model file of the robot and the pose information of the robot.

[0110] EEE 11. The method according to any one of EEE 1 to EEE 10, wherein the preset template object includes a preset fixed object, and wherein the method further includes: displaying the preset fixed object in the 3D virtual environment in a manner stationary relative to a fixed reference point, wherein the fixed reference point includes the base of the robotic arm of the robot.

[0111] EEE 12. The method according to any one of EEE 1 to EEE 11, wherein constructing the 3D virtual environment of the operating environment includes: determining and / or displaying a safe operation area of the 3D virtual environment based on the preset template object in the 3D virtual environment.

[0112] EEE 13. The method according to any one of EEE 1 to EEE 12, wherein the method further includes: planning a motion trajectory of the robot, the motion trajectory being determined to be within the safe operation area while avoiding the non-preset objects.

[0113] EEE 14. The method according to any one of EEE 1 to EEE 13, wherein displaying the 3D virtual environment of the operating environment via the display interface includes: displaying, via the display interface, one or more planned motion trajectories of the robot in the 3D virtual environment.

[0114] EEE 15. The method according to any one of EEE 1 to EEE 13, wherein the preset template object is modeled by at least one of the following: computer-aided design (CAD) documents, machine vision, lidar, and / or electronic skin.

[0115] EEE 16. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: determining, based on perception information acquired by a robot, the object type of each of one or more objects in the operating environment of the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least partially based on the object type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface.

[0116] EEE 17. The non-transitory computer-readable storage medium according to EEE 16, wherein one or more electronic skins are provided on the robot, and the perception information includes signals generated by the one or more electronic skins in response to the one or more objects approaching the robot.

[0117] EEE 18. The non-transitory computer-readable storage medium according to EEE 16 or EEE 17, wherein the object type includes a preset object and / or a non-preset object.

[0118] EEE 19. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 18, wherein determining the object type of each of the one or more objects includes: based on the perception information acquired by the robot, when the position data and / or surface shape in the perception information match the position data and / or surface shape of a preset template object, the matching position data and / or surface shape are recognized as belonging to a preset object.

[0119] EEE 20. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 19, wherein determining the object type of each of the one or more objects includes: based on the perception information acquired by the robot, when the position data and / or surface shape in the perception information do not match the position data and / or surface shape of all objects in the preset template object, the position data and / or surface shape that do not match are identified as belonging to a non-preset object.

[0120] EEE 21. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 20, wherein constructing the 3D virtual environment of the operating environment includes: modeling the non-preset object based on the perception information acquired by the robot, wherein the model color depth, chromaticity, transparency or a combination thereof of the non-preset object varies with the distance between the non-preset object and the nearest component of the robot, and / or the distance between the non-preset object and the nearest component of the robot is indicated.

[0121] EEE 22. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 21, wherein displaying the 3D virtual environment of the operating environment via the display interface includes: providing, via the display interface, an option for selecting any one of the objects of the preset object type and / or any one of the objects of the non-preset object type among the one or more objects for display on the display interface.

[0122] EEE 23. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 22, wherein the preset template object includes a preset tool, and wherein the instructions, when executed by the one or more processors, cause the one or more processors to further perform an operation, the operation including: determining whether the position data belongs to the preset tool based on the position data in the perception information, the pose of the robot, and a preset position relationship between the preset tool and the body model of the robot.

[0123] EEE 24. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 23, wherein the preset template object includes a part of the robot, and wherein the instructions, when executed by the one or more processors, cause the one or more processors to further perform an operation, the operation including: determining whether the perception information belongs to the part of the robot based on the position data and / or the surface shape in the perception information and the pose of the robot.

[0124] EEE 25. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 24, wherein the sensed information includes the pose information of the robot, and wherein the instructions, when executed by the one or more processors, cause the one or more processors to further perform operations, the operations including: based on the imported model file of the robot and the pose information of the robot, displaying at least a part of the robot in real time via the display interface.

[0125] EEE 26. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 25, wherein the preset template object includes a preset fixed object, and wherein the instructions, when executed by the one or more processors, cause the one or more processors to further perform the following operations: displaying the preset fixed object in the 3D virtual environment in a manner stationary relative to a fixed reference point, wherein the fixed reference point includes the base of the robotic arm of the robot.

[0126] EEE 27. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 26, wherein constructing the 3D virtual environment of the operating environment includes: determining and / or displaying a safe operation area of the 3D virtual environment based on the preset template object in the 3D virtual environment.

[0127] EEE 28. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 27, wherein the instructions, when executed by the one or more processors, cause the one or more processors to further perform operations, the operations including: planning a motion trajectory of the robot, the motion trajectory being determined to be within the safe operation area while avoiding the non-preset objects.

[0128] EEE 29. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 28, wherein displaying the 3D virtual environment of the operating environment via the display interface includes: displaying one or more planned motion trajectories of the robot in the 3D virtual environment via the display interface.

[0129] EEE 30. The non-transitory computer-readable storage medium according to any one of EEE 16 to EEE 29, wherein the preset template object is modeled by at least one of the following: computer-aided design (CAD) documents, machine vision, lidar, and / or electronic skin.

[0130] EEE 31. A system for displaying an operating environment of a robot, the system comprising one or more sensors; one or more processors; and one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: determining an object type of each of one or more objects in the operating environment of the robot based on sensing information acquired by the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least in part based on the object type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface.

[0131] EEE 32. A robot comprising a robotic arm; one or more sensors including an electronic skin; a controller; a display; one or more storage devices storing instructions that, when executed, cause the controller to perform operations including: determining an object type of each of one or more objects in the operating environment of the robot based on sensing information acquired by the robot; constructing a three-dimensional (3D) virtual environment of the operating environment at least in part based on the object type of each of the one or more objects; and displaying the 3D virtual environment of the operating environment via a display interface.

[0132] Although the present disclosure has been described in connection with actual and preferred embodiments, it should be understood that the present disclosure is not limited to the disclosed embodiments, but is intended to cover various arrangements without departing from the broadest interpretation of the appended claims.

Claims

1. A method for displaying an operating environment of a robot, wherein the robot is provided with one or more electronic skins, wherein the one or more electronic skins include one or more electrode arrays for sensing position data and surface shape of objects present in the operating environment of the robot without contacting the objects, the method comprising: Obtaining capacitance values ​​or capacitance value changes of electrodes of the one or more electrode arrays of the one or more electronic skins; acquiring sensory information including signals generated by the one or more electronic skins in response to detecting the presence of one or more objects within a detection range of the one or more electrode arrays based on the capacitance values ​​of the electrodes, wherein the sensory information indicates position data and surface shapes of the one or more objects; determining an object type of each of the one or more objects in the operating environment of the robot based on one or both of the position data and the surface shape indicated by the perception information acquired by the robot via the one or more electronic skins; constructing a 3D virtual environment of the operating environment based at least in part on the sensory information and the object type of each of the one or more objects, the 3D virtual environment including a 3D model of at least a portion of the one or more objects present in the operating environment; and The 3D virtual environment of the operating environment is displayed via a display interface.

2. The method according to claim 1, wherein: Determining the object type includes determining, based on the perception information acquired by the robot via the one or more electronic skins, that the object type of each of the one or more objects is one of a preset object and a non-preset object.

3. The method according to claim 2, wherein: Determining that the object type of each of the one or more objects is one of a preset object and a non-preset object comprises: Based on the perception information acquired by the robot, determining that one or both of the position data and the surface shape in the perception information matches one or both of the position data and the surface shape of one of a plurality of preset template objects, and One or both of the position data in the perception information and the surface shape that match each other are identified as belonging to the preset object.

4. The method according to claim 3, wherein: Determining that the object type of each of the one or more objects is one of a preset object and a non-preset object comprises: Based on the perception information acquired by the robot, determining that one or both of the position data and the surface shape in the perception information do not match the position data and the surface shape of all preset template objects in the preset template objects, and The one or both of the position data and the surface shape that do not constitute a match are identified as belonging to the non-preset object.

5. The method according to claim 2, wherein: The 3D virtual environment of the operating environment is constructed as follows: Modeling the non-preset object based on the perception information acquired by the robot, wherein the modeling of the non-preset object includes one or both of the following: varying the color depth, hue, transparency, or a combination thereof of the model of the non-preset object as the non-preset object is spaced from the nearest neighboring component of the robot, and A distance between the non-preset object and a nearest neighboring component of the robot is indicated.

6. The method according to claim 2, wherein: Displaying the 3D virtual environment of the operating environment via the display interface includes: An option is provided via the display interface for selecting one or both of the following among the one or more objects for display on the display interface: The object type of the one or more objects is any one of the preset objects, and The object type of the one or more objects is any one of the non-preset objects.

7. The method according to claim 3, wherein: The preset template object includes a preset tool, and wherein the method further includes: Based on the position data in the perception information, the posture of the robot, and the preset position relationship between the preset tool and the body model of the robot, it is determined whether the position data belongs to the preset tool.

8. The method according to claim 3, wherein: The preset template object comprises a part of the robot, and wherein the method further comprises: Based on the one or both of the position data and the surface shape in the perception information and the posture information of the robot, it is determined whether the perception information belongs to the part of the robot.

9. The method according to claim 1, wherein: The perception information includes posture information of the robot, and wherein the method further comprises: Based on the imported model file of the robot and the posture information of the robot, at least a part of the robot is displayed in real time via the display interface.

10. The method according to claim 3, wherein: The preset template object includes a preset fixed object, and wherein the method further includes: The preset fixed object is displayed in the 3D virtual environment in a stationary manner relative to a fixed reference point, wherein the fixed reference point includes a base of the robot.

11. The method according to claim 3, wherein: Constructing the 3D virtual environment of the operating environment includes determining a safe operating area of ​​the 3D virtual environment based on the preset template object in the 3D virtual environment, and displaying the 3D virtual environment of the operating environment includes displaying the safe operating area of ​​the 3D virtual environment via the display interface.

12. The method according to claim 11, wherein: The method further comprises: A motion trajectory of the robot is planned, where the motion trajectory is determined to be within the safe operation area while avoiding the non-preset objects.

13. The method according to claim 1, wherein: Displaying the 3D virtual environment of the operating environment via the display interface includes: displaying one or more planned motion trajectories of the robot in the 3D virtual environment via the display interface.

14. The method according to claim 1, wherein: Determining the object type includes: Based on the capacitance values ​​of the electrodes of the one or more electrode arrays and the coordinate information corresponding to the electrodes of the one or more electrode arrays, the position data and one or both of the surface shapes of the one or more objects in the operating environment of the robot are respectively derived.

15. The method according to claim 1, wherein: The 3D virtual environment of the operating environment is constructed as follows: In response to determining that the object type of a first object among the one or more objects is a preset object based on the perception information, a first 3D model representing first position data and a first surface shape of a preset template object corresponding to the first object is imported, and the first 3D model of the preset template object is rendered in the 3D virtual environment of the operating environment of the robot.

16. The method according to claim 1, wherein: The 3D virtual environment of the operating environment is constructed as follows: In response to determining that the object type of a second object among the one or more objects is a non-preset object based on the perception information, second position data and a second 3D model of a second surface shape representing at least a portion of the second object are dynamically generated, and the second 3D model of the second object is rendered in real time in the 3D virtual environment of the operating environment of the robot.

17. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform operations for displaying an operating environment of a robot, the robot being provided with one or more electronic skins, the one or more electronic skins comprising one or more electrode arrays for sensing position data and surface shape of an object present in the operating environment of the robot without contacting the object, the operations comprising: Obtaining capacitance values ​​or capacitance value changes of electrodes of the one or more electrode arrays of the one or more electronic skins; acquiring sensory information including signals generated by the one or more electronic skins in response to detecting the presence of one or more objects within a detection range of the one or more electrode arrays based on the capacitance values ​​of the electrodes, wherein the sensory information indicates position data and surface shapes of the one or more objects; determining an object type of each of the one or more objects in the operating environment of the robot based on one or both of the position data and the surface shape indicated by the perception information acquired by the robot via the one or more electronic skins; constructing a 3D virtual environment of the operating environment based at least in part on the sensory information and the object type of each of the one or more objects, the 3D virtual environment including a 3D model of at least a portion of the one or more objects present in the operating environment; and The 3D virtual environment of the operating environment is displayed via a display interface.

18. The non-transitory computer-readable storage medium of claim 17, wherein: Determining the object type includes determining, based on the perception information acquired by the robot, that the object type of each of the one or more objects is one of a preset object and a non-preset object.

19. The non-transitory computer-readable storage medium of claim 18, wherein: Determining that the object type of each of the one or more objects is one of a preset object and a non-preset object comprises: Based on the perception information acquired by the robot, determining that one or both of the position data and the surface shape in the perception information matches one or both of the position data and the surface shape of one of a plurality of preset template objects, and One or both of the position data in the perception information and the surface shape that match each other are identified as belonging to the preset object.

20. The non-transitory computer-readable storage medium of claim 19, wherein: Determining that the object type of each of the one or more objects is one of a preset object and a non-preset object comprises: Based on the perception information acquired by the robot, determining that one or both of the position data and the surface shape in the perception information do not match the position data and the surface shape of all preset template objects in the preset template objects, and The one or both of the position data and the surface shape that do not constitute a match are identified as belonging to the non-preset object.

21. A system for displaying a robot operating environment, the system comprising: one or more sensors, including one or more electronic skins; one or more processors; and One or more storage devices storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations for displaying an operating environment of a robot, wherein the robot is provided with one or more electronic skins, the one or more electronic skins comprising one or more electrode arrays for sensing position data and surface shape of an object present in the operating environment of the robot without contacting the object, the operations comprising: Obtaining capacitance values ​​or capacitance value changes of electrodes of the one or more electrode arrays of the one or more electronic skins; acquiring sensory information including signals generated by the one or more electronic skins in response to detecting the presence of one or more objects within a detection range of the one or more electrode arrays based on the capacitance values ​​of the electrodes, wherein the sensory information indicates position data and surface shapes of the one or more objects; determining an object type of each of the one or more objects in the operating environment of the robot based on one or both of the position data and the surface shape indicated by the perception information acquired by the robot via the one or more electronic skins; constructing a 3D virtual environment of the operating environment based at least in part on the sensory information and the object type of each of the one or more objects, the 3D virtual environment including a 3D model of at least a portion of the one or more objects present in the operating environment; and The 3D virtual environment of the operating environment is displayed via a display interface.

22. A robot comprising: Robot body; one or more sensors, the one or more sensors comprising one or more electronic skins; Controller; monitor; and One or more storage devices storing instructions, which, when executed, cause the controller to perform operations for displaying an operating environment of a robot, wherein the robot is provided with one or more electronic skins, the one or more electronic skins comprising one or more electrode arrays for sensing position data and surface shape of an object present in the operating environment of the robot without contacting the object, the operations comprising: Obtaining capacitance values ​​or capacitance value changes of electrodes of the one or more electrode arrays of the one or more electronic skins; acquiring sensory information including signals generated by the one or more electronic skins in response to detecting the presence of one or more objects within a detection range of the one or more electrode arrays based on the capacitance values ​​of the electrodes, wherein the sensory information indicates position data and surface shapes of the one or more objects; determining an object type of each of the one or more objects in the operating environment of the robot based on one or both of the position data and the surface shape indicated by the perception information acquired by the robot via the one or more electronic skins; constructing a 3D virtual environment of the operating environment based at least in part on the sensory information and the object type of each of the one or more objects, the 3D virtual environment including a 3D model of at least a portion of the one or more objects present in the operating environment; and The 3D virtual environment of the operating environment is displayed via a display interface.

Images