3D environment modeling methods and equipment, computer storage media, and industrial robot operating platforms

Abstract

This invention provides a three-dimensional environment modeling method, comprising: scanning the surrounding environment to obtain a point cloud flow; detecting one or more augmented reality codes in the surrounding environment; and refining and optimizing the point cloud flow based at least on information carried by the one or more augmented reality codes and the positions of the one or more augmented reality codes, in order to model the surrounding environment. This invention also relates to a three-dimensional environment modeling device, a computer storage medium, and an industrial robot operating platform.

Description

[Technical Field]

[0001] This invention relates to a three-dimensional environment modeling method and equipment, a computer storage medium, and an industrial robot operating platform. [Background Technology]

[0002] In certain industrial applications, cameras (such as RGB-D cameras) are used to scan the environment and reconstruct it using the scanned and recorded data. However, subsequent environment reconstruction often involves a significant amount of manual work, especially during post-processing. Users must specify the boundaries of the work area in the x, y, or z planes, delete temporary 3D objects that existed at the time of recording but did not belong to the scene, or fill in objects that were only partially scanned.

[0003] In addition, the depth sensor of an RGB-D camera encounters problems when capturing reflective surfaces such as glass; for example, parts that were originally solid surfaces may be missing or contain noise in the reconstructed 3D model.

[0004] Therefore, an improved 3D environment modeling scheme is desired. [Summary of the Invention]

[0005] One aspect of the present invention proposes a three-dimensional environment modeling method, the method comprising: scanning the surrounding environment to obtain a point cloud flow; detecting one or more augmented reality codes in the surrounding environment; and manipulating the point cloud flow based at least on information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes, in order to model the surrounding environment (thereby achieving accelerated modeling and automatic optimization and improvement of point cloud flow operations).

[0006] Preferably, in the above-described three-dimensional environment modeling method, scanning the surrounding environment includes: using an RGB-D camera to scan the surrounding environment from multiple angles.

[0007] Preferably, the aforementioned three-dimensional environment modeling method may further include: recording the point cloud flow.

[0008] Preferably, in the above-described 3D environment modeling method, detecting one or more augmented reality codes in the surrounding environment includes: using an augmented reality tracking library to detect one or more augmented reality codes in the surrounding environment and their degrees of freedom pose relative to the RGB-D camera while scanning the surrounding environment; and recording the degrees of freedom pose and registering it to a point cloud coordinate system.

[0009] Preferably, in the above-described three-dimensional environment modeling method, the one or more augmented reality codes are placed on the surface of the target object.

[0010] Preferably, in the above-described three-dimensional environment modeling method, operating on the point cloud flow based at least on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes includes: constructing a geometric structure at the location of the one or more augmented reality codes according to the information carried by the one or more augmented reality codes, wherein the orientation of the geometric structure relative to the one or more augmented reality codes and the content of the geometric structure are encoded in the information.

[0011] Preferably, in the above-described three-dimensional environment modeling method, the geometric structure is a polygonal boundary / wall, a cuboid, or a cylinder, etc.

[0012] Preferably, in the above-described three-dimensional environment modeling method, operating the point cloud stream based at least on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes includes: removing obstacles or boundaries in the point cloud stream at or near the location of the one or more augmented reality codes according to the information carried by the one or more augmented reality codes, wherein the extent and orientation of the region of the points to be removed are encoded in the information.

[0013] Preferably, in the above-described 3D environment modeling method, operating on the point cloud stream based at least on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes includes: performing region operations on the point cloud stream, wherein the region operations include region segmentation, filtering, and / or clustering of the target object, and the parameters of the operations are defined in the information carried by the one or more augmented reality codes.

[0014] Another aspect of the present invention provides a three-dimensional environment modeling method, the method comprising: scanning the environment surrounding the industrial robot to obtain a point cloud flow; detecting one or more augmented reality codes in the surrounding environment; manipulating the point cloud flow based at least on information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes; and processing the manipulated point cloud flow into a surface mesh so as to model the surrounding environment (thereby achieving accelerated modeling and automatic optimization and improvement of point cloud flow operations).

[0015] Another aspect of the present invention provides a three-dimensional environment modeling device, the device comprising: a scanning device for scanning the surrounding environment to obtain a point cloud stream; a detection device for detecting one or more augmented reality codes in the surrounding environment; and an execution device for manipulating the point cloud stream based at least on information carried by the one or more augmented reality codes and the position of the one or more augmented reality codes, in order to model the surrounding environment.

[0016] Preferably, in the above-mentioned three-dimensional environment modeling device, the scanning device is configured to scan the surrounding environment from multiple angles using an RGB-D camera.

[0017] Preferably, the above-mentioned three-dimensional environment modeling equipment may further include: a recording device for recording the point cloud flow.

[0018] Preferably, in the above-mentioned three-dimensional environment modeling device, the detection device includes: a first detection unit, used to detect one or more augmented reality codes in the surrounding environment and their degrees of freedom pose relative to the RGB-D camera using an augmented reality tracking library when scanning the surrounding environment; and a first recording unit, used to record the degrees of freedom pose and register it to a point cloud coordinate system.

[0019] Preferably, in the above-described 3D environment modeling device, the one or more augmented reality codes are placed on the surface of the target object.

[0020] Preferably, in the above-described three-dimensional environment modeling device, the execution device is configured to construct a geometric structure at the location of the one or more augmented reality codes based on information carried by the one or more augmented reality codes, wherein the orientation of the geometric structure relative to the one or more augmented reality codes and the content of the geometric structure are encoded in the information.

[0021] Preferably, in the above-mentioned three-dimensional environment modeling equipment, the geometric structure is a polygonal boundary / wall, a cuboid, or a cylinder, etc.

[0022] Preferably, in the above-described three-dimensional environment modeling device, the execution device is further configured to remove obstacles or boundaries at or near the location of the one or more augmented reality codes in the point cloud stream based on information carried by the one or more augmented reality codes, wherein the extent and orientation of the region of the points to be removed are encoded in the information.

[0023] Preferably, in the above-mentioned three-dimensional environment modeling device, the execution device is configured to perform region operations on the point cloud stream, wherein the region operations include region segmentation, filtering and / or clustering of the target object, and the parameters of the operations are defined in the information carried by the one or more augmented reality codes.

[0024] Another aspect of the present invention provides a three-dimensional environment modeling device for an industrial robot, the device comprising: a scanning device for scanning the surrounding environment of the industrial robot to obtain a point cloud flow; a detection device for detecting one or more augmented reality codes in the surrounding environment; an execution device for manipulating the point cloud flow based at least on information carried by the one or more augmented reality codes and the position of the one or more augmented reality codes; and a processing device for processing the manipulated point cloud flow into a surface mesh for modeling the surrounding environment.

[0025] Another aspect of the present invention provides a computer storage medium including instructions that, when executed, perform the three-dimensional environment modeling method as described above.

[0026] Another aspect of the present invention provides an industrial robot operating platform, which includes the three-dimensional environment modeling device as described above.

[0027] Compared to existing 3D environment modeling solutions, the aforementioned solution utilizes one or more augmented reality codes within the environment, avoiding or reducing manual operations involved in data processing after environmental scanning, making it more convenient and efficient. The information carried by the augmented reality codes can be customized as needed to modify or perform other operations on the point cloud or mesh to be reconstructed. [Attached Image Description]

[0028] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings:

[0029] Figure 1 A three-dimensional environment modeling method according to an embodiment of the present invention is shown;

[0030] Figure 2 A method for three-dimensional environment modeling of industrial robots according to an embodiment of the present invention is shown; and

[0031] Figure 3 A three-dimensional environment modeling apparatus according to an embodiment of the present invention is shown.

Detailed Implementation Methods

[0032] The following description illustrates specific embodiments of the invention to teach those skilled in the art how to make and use the invention in the best manner. Some conventional aspects have been simplified or omitted for the purpose of teaching the principles of the invention. Those skilled in the art should understand that variations derived from these embodiments will fall within the scope of the invention. Those skilled in the art should understand that the following features can be combined in various ways to form multiple variations of the invention. Therefore, the invention is not limited to the specific embodiments described below, but is defined only by the claims and their equivalents.

[0033] refer to Figure 1 , Figure 1 A three-dimensional environment modeling method 1000 according to an embodiment of the present invention is shown.

[0034] In step S110, the surrounding environment is scanned to obtain point cloud flow.

[0035] In step S120, one or more augmented reality codes in the surrounding environment are detected.

[0036] In step S130, the point cloud stream is manipulated at least based on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes in order to model the surrounding environment.

[0037] In the context of this invention, the term "augmented reality code" refers to a marker used to perform specific operations on a point cloud stream obtained after scanning. Augmented reality codes can take the form of QR codes, data matrices, or Maxicode, and this invention is not limited to these forms. Operations performed on the point cloud stream can be encoded or defined in the augmented reality code. In practical use, one or more augmented reality codes can be placed within a working area, such as on a plane and on the object to be segmented. As an example, these augmented reality codes may carry information such as whether adjacent surfaces should be included in or removed from the reconstruction model, or whether certain point cloud operations (e.g., segmentation) should be performed. Furthermore, parameters for these operations can be stored in the augmented reality code. As yet another example, augmented reality codes can be used to mark surfaces that cannot be detected by a 3D scanner (e.g., an RGB-D camera). For example, augmented reality codes can define a glass wall, including its dimensions, or an artificial wall can be added to the reconstruction model at the location where the augmented reality code is placed.

[0038] The above-mentioned 3D environment modeling scheme utilizes one or more augmented reality codes in the environment, avoiding or reducing the manual operations involved in the data processing after environmental scanning, making it more convenient and efficient.

[0039] In one embodiment, step S110 includes scanning the surrounding environment from multiple angles using an RGB-D camera. Currently, RGB-D depth cameras include Microsoft Kinect, ASUS Xtion, Orbbec, Intel RealSense, etc., and this invention does not limit this to any particular type. In some embodiments, multiple RGB-D cameras can be used to scan the surrounding environment from multiple angles, and the multiple RGB-D cameras can be located at different positions. For example, a first RGB-D camera is located at a first position, and a second RGB-D camera is located at a second position different from the first position.

[0040] although Figure 1 As not shown in the diagram, in one embodiment, the 3D environment modeling method 1000 may further include: recording the point cloud flow. This recording step may be located between steps S110 and S120.

[0041] In one embodiment, step S120 includes: while scanning the surrounding environment, using an augmented reality tracking library to detect one or more augmented reality codes in the surrounding environment and their degrees of freedom poses relative to the RGB-D camera; and recording the degrees of freedom poses and registering them to a point cloud coordinate system. For example, in step S120, augmented reality codes (e.g., QR codes) and their six degrees of freedom poses (and / or other stored information in the augmented reality tracking library) are detected and jointly registered or calibrated to the coordinate system of the point cloud stream.

[0042] In one embodiment, the one or more augmented reality codes are stationary relative to a camera (e.g., an RGB-D depth camera) and are placed on the surface of a target object. For example, one or more augmented reality codes could be placed on a glass wall to help the RGB-D depth camera recognize the wall. The one or more augmented reality codes also encode the dimensions of the wall. As another example, one or more augmented reality codes could be placed at a planar boundary to fill a predetermined-size boundary in the point cloud stream at the location of the one or more augmented reality codes. Yet another example is that one or more augmented reality codes could be placed on a target object to remove the target object or a portion thereof from the point cloud stream.

[0043] In some embodiments, the location and pattern of the augmented reality code may include information relating to customized operations during post-processing of the recorded point cloud stream. In one embodiment, step S130 includes: constructing a geometry at the location of the one or more augmented reality codes based on information carried by the one or more augmented reality codes, wherein the orientation of the geometry relative to the one or more augmented reality codes and the content of the geometry are encoded in the information. The geometry may be a polygonal boundary / wall, a cuboid, or a cylinder, etc., and is not limited thereto. In another embodiment, step S130 includes: removing obstacles or boundaries at or near the location of the one or more augmented reality codes from the point cloud stream based on information carried by the one or more augmented reality codes, wherein the extent and orientation of the region of points to be removed are encoded in the information. In yet another embodiment, step S130 includes: performing a region operation on the point cloud stream, wherein the region operation includes region segmentation, filtering, and / or clustering of the target objects, the parameters of which are defined in the information carried by the one or more augmented reality codes. Those skilled in the art will understand that the different embodiments of step S130 described above can be arranged and combined as needed. For example, step S130 may delete the pattern of points and replace them with other points, i.e., simultaneously perform (1) removing obstacles or boundaries in the point cloud flow at or near the location of the one or more augmented reality codes based on the information carried by the one or more augmented reality codes, wherein the extent and orientation of the region of the points to be removed are encoded in the information; and (2) constructing a geometry at the location of the one or more augmented reality codes based on the information carried by the one or more augmented reality codes, wherein the orientation of the geometry relative to the one or more augmented reality codes and the content of the geometry are encoded in the information.

[0044] Figure 2 A three-dimensional environment modeling method 2000 for industrial robots according to an embodiment of the present invention is shown.

[0045] In step S210, the surrounding environment of the industrial robot is scanned to obtain point cloud flow.

[0046] In step S220, one or more augmented reality codes in the surrounding environment are detected.

[0047] In step S230, the point cloud stream is operated on at least based on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes.

[0048] In step S240, the point cloud flow after the operation is processed into a surface mesh so that the surrounding environment can be modeled.

[0049] In one embodiment, the point cloud stream is acquired and recorded by a handheld RGB-D camera driven by a portable microcomputer. As the area to be recorded and reconstructed is scanned by the RGB-D camera, augmented reality code is detected by the RGB sensor in the camera. This can be achieved, for example, by including a tracking library (such as ARuCO). The library outputs the 6-DOF pose of the augmented reality code relative to the camera. All augmented reality code poses are recorded and collectively registered to the recorded point cloud coordinate system.

[0050] In one embodiment, after the recording session is completed, the recorded point cloud stream is transferred to a dedicated PC for offline post-processing, or post-processing is performed on a scanning PC. Specifically, the point cloud stream is fed into the latest reconstruction algorithm. In one embodiment, noise is first removed from the point cloud stream using a point cloud library (PCL). Then, the point cloud data is registered and stitched together using the PCL and the Iterative Nearest Point (ICP) algorithm to reconstruct the entire environment. Those skilled in the art will understand that various Iterative Nearest Point (ICP) methods can be employed, including but not limited to the precise registration method for searching nearest points using point-to-plane search proposed by Chen and Medioni and Bergevin et al., the fast registration method for searching nearest points using point-to-projection search proposed by Rusinkiewicz and Levoy, and the registration method for searching nearest points using Contractive-projection-point search proposed by Soon-Yong and Murali.

[0051] Then, using custom software code (included in the augmented reality code), the location of each detected augmented reality code and information about each code are imported into the processing sequence. Based on the stored information, portions of the point cloud stream are deleted, defined regions are added to the point cloud, or specific operations are applied to the point cloud stream. After performing predefined or custom actions or operations, the final augmented point cloud stream is further processed into a surface mesh.

[0052] The aforementioned 3D environment modeling method 2000 achieves automatic construction of boundaries or shapes in the 3D model by simply adding augmented reality code to the environment. It also automatically removes unwanted surfaces, defines reflective surfaces, eliminates noise, and provides descriptions of 3D starting points (performing operations such as filtering and segmentation on the point cloud).

[0053] Figure 3 A three-dimensional environment modeling device 3000 according to an embodiment of the present invention is shown.

[0054] like Figure 3As shown, the 3D environment modeling device 3000 includes a scanning device 310, a detection device 320, and an execution device 330. The scanning device 310 scans the surrounding environment to obtain a point cloud stream. The detection device 320 detects one or more augmented reality codes in the surrounding environment. The execution device 330 manipulates the point cloud stream based at least on information carried by the one or more augmented reality codes and their positions to model the surrounding environment.

[0055] In the context of this invention, the term "augmented reality code" refers to a marker used to perform specific operations on a point cloud stream obtained after scanning. Augmented reality codes can take the form of QR codes, data matrices, or Maxicode, and this invention is not limited to these forms. Operations performed on the point cloud stream can be encoded or defined in the augmented reality code. In practical use, one or more augmented reality codes can be placed within a working area, such as on a plane and on the object to be segmented. As an example, these augmented reality codes may carry information such as whether adjacent surfaces should be included in or removed from the reconstruction model, or whether certain point cloud operations (e.g., segmentation) should be performed. Furthermore, parameters for these operations can be stored in the augmented reality code. As yet another example, augmented reality codes can be used to mark surfaces that cannot be detected by a 3D scanner (e.g., an RGB-D camera). For example, augmented reality codes can define a glass wall, including its dimensions, or an artificial wall can be added to the reconstruction model at the location where the augmented reality code is placed.

[0056] The aforementioned 3D environment modeling device 3000 utilizes one or more augmented reality codes in the environment, avoiding or reducing manual operations involved in data processing after environmental scanning, making it more convenient and efficient.

[0057] In one embodiment, the scanning device 310 is configured to scan the surrounding environment from multiple angles using RGB-D cameras. Currently, RGB-D cameras include Microsoft Kinect, ASUS Xtion, Orbbec, Intel RealSense, etc., and this invention does not limit this to any particular type. In some embodiments, the scanning device 310 may use multiple RGB-D cameras to scan the surrounding environment from multiple angles, and the multiple RGB-D cameras may be located in different positions; for example, a first RGB-D camera may be located in a first position, and a second RGB-D camera may be located in a second position different from the first position.

[0058] although Figure 3 As not shown in the diagram, the aforementioned three-dimensional environment modeling device 3000 may further include: a recording device for recording the point cloud flow.

[0059] In one embodiment, the detection device 320 may include a first detection unit and a first recording unit. The first detection unit is used to detect one or more augmented reality codes in the surrounding environment and their degrees of freedom poses relative to the RGB-D camera using an augmented reality tracking library while scanning the surrounding environment. The first recording unit is used to record the degrees of freedom poses and register them to a point cloud coordinate system. In some embodiments, the detection device 320 may be configured to detect augmented reality codes (e.g., QR codes) and their six degrees of freedom poses (and / or other stored information in the augmented reality tracking library), and jointly register or calibrate them to the coordinate system of the point cloud stream.

[0060] In one embodiment, the one or more augmented reality codes are stationary relative to a camera (e.g., an RGB-D depth camera) and are placed on the surface of a target object. For example, one or more augmented reality codes could be placed on a glass wall to help the RGB-D depth camera recognize the wall. The one or more augmented reality codes also encode the dimensions of the wall. As another example, one or more augmented reality codes could be placed at a planar boundary to fill a predetermined-size boundary in the point cloud stream at the location of the one or more augmented reality codes. Yet another example is that one or more augmented reality codes could be placed on a target object to remove the target object or a portion thereof from the point cloud stream.

[0061] In some embodiments, the location and pattern of the augmented reality code may include information relating to customized operations during post-processing of the recorded point cloud stream.

[0062] In one embodiment, the execution device 330 is configured to construct a geometric structure at the location of the one or more augmented reality codes based on information carried by the one or more augmented reality codes, wherein the orientation of the geometric structure relative to the one or more augmented reality codes and the content of the geometric structure are encoded in the information. The geometric structure may be a polygonal boundary / wall, a cuboid, or a cylinder, etc., and the present invention is not limited thereto.

[0063] In another embodiment, the execution device 330 is further configured to remove obstacles or boundaries at or near the location of the one or more augmented reality codes in the point cloud stream based on information carried by the one or more augmented reality codes, wherein the extent and orientation of the region of the points to be removed are encoded in the information.

[0064] In yet another embodiment, the execution device 330 is configured to perform region operations on the point cloud stream, wherein the region operations include region segmentation, filtering, and / or clustering of the target object, and the parameters of the operations are defined in information carried by the one or more augmented reality codes.

[0065] Those skilled in the art will understand that different embodiments of the execution device 330 described above can be arranged and combined as needed. For example, the execution device 330 can be configured to simultaneously perform the following two actions: (1) removing obstacles or boundaries at or near the location of the one or more augmented reality codes in the point cloud based on information carried by the one or more augmented reality codes, wherein the extent and orientation of the region of the point to be removed are encoded in the information; and (2) constructing a geometric structure at the location of the one or more augmented reality codes based on information carried by the one or more augmented reality codes, wherein the orientation of the geometric structure relative to the one or more augmented reality codes and the content of the geometric structure are encoded in the information.

[0066] In some embodiments, such as in the scenario of 3D environment modeling applied to industrial robots, the 3D environment modeling device may include a scanning device, a detection device, an execution device, and a processing device. Specifically, the scanning device is used to scan the environment surrounding the industrial robot to obtain a point cloud stream. The detection device is used to detect one or more augmented reality codes in the surrounding environment. The execution device is used to manipulate the point cloud stream based at least on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes. The processing device is used to process the manipulated point cloud stream into a surface mesh so that the surrounding environment can be modeled.

[0067] Those skilled in the art will readily understand that the three-dimensional environment modeling method provided in one or more embodiments of the present invention can be implemented by a computer program. For example, when a computer storage medium (e.g., a USB flash drive) containing the computer program is connected to a computer, running the computer program will execute the three-dimensional environment modeling method of the embodiments of the present invention.

[0068] In summary, several embodiments of the present invention provide a three-dimensional environment modeling scheme that uses augmented reality code to accelerate environment reconstruction and automatically improves and optimizes the point cloud flow containing the augmented reality code. Although only some specific embodiments of the invention have been described, those skilled in the art will understand that the invention can be implemented in many other forms without departing from its spirit and scope, such as on an industrial robot operating platform. Therefore, the examples and embodiments shown are considered illustrative rather than restrictive, and the invention may encompass various modifications and substitutions without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A three-dimensional environment modeling method, the method comprising: The surrounding environment is scanned to obtain point cloud flow; Detect one or more augmented reality codes in the surrounding environment, wherein the augmented reality code refers to a marker that performs a specific operation on the point cloud stream obtained after scanning, and wherein the augmented reality code is placed in the surrounding environment; as well as The point cloud stream is manipulated based at least on information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes in order to model the surrounding environment. The operation on the point cloud stream, based at least on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes, includes: Based on information carried by the one or more augmented reality codes, a geometric structure is constructed at the location of the one or more augmented reality codes, wherein the orientation of the geometric structure relative to the one or more augmented reality codes and the content of the geometric structure are encoded in the information; or Based on information carried by the one or more augmented reality codes, obstacles or boundaries at or near the location of the one or more augmented reality codes are removed from the point cloud stream, wherein the extent and orientation of the region of the points to be removed are encoded in the information; or Perform region operations on the point cloud stream, wherein the region operations include region segmentation, filtering and / or clustering of target objects, and the parameters of the operations are defined in the information carried by the one or more augmented reality codes.

2. The three-dimensional environment modeling method as described in claim 1, wherein, Scanning the surrounding environment includes: The surrounding environment is scanned from multiple angles using an RGB-D camera.

3. The three-dimensional environment modeling method as described in claim 1 further includes: Record the point cloud flow.

4. The three-dimensional environment modeling method as described in claim 2, wherein, Detecting one or more augmented reality codes in the surrounding environment includes: While scanning the surrounding environment, an augmented reality tracking library is used to detect one or more augmented reality codes in the surrounding environment and their degrees of freedom pose relative to the RGB-D camera; and Record the pose of the stated degrees of freedom and register it to the point cloud coordinate system.

5. The three-dimensional environment modeling method as described in claim 1, wherein, The one or more augmented reality codes are placed on the surface of the target object.

6. The three-dimensional environment modeling method as described in claim 1, wherein, The geometry is a polygonal boundary, a wall, a cuboid, or a cylinder.

7. A three-dimensional environment modeling method, the method comprising: The environment surrounding the industrial robot is scanned to obtain point cloud flow; Detect one or more augmented reality codes in the surrounding environment, wherein the augmented reality code refers to a marker that performs a specific operation on the point cloud stream obtained after scanning, and wherein the augmented reality code is placed in the surrounding environment; The point cloud stream is manipulated based at least on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes; as well as The processed point cloud flow is converted into a surface mesh for modeling the surrounding environment. The operation on the point cloud stream, based at least on the information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes, includes: Based on information carried by the one or more augmented reality codes, a geometric structure is constructed at the location of the one or more augmented reality codes, wherein the orientation of the geometric structure relative to the one or more augmented reality codes and the content of the geometric structure are encoded in the information; or Based on information carried by the one or more augmented reality codes, obstacles or boundaries at or near the location of the one or more augmented reality codes are removed from the point cloud stream, wherein the extent and orientation of the region of the points to be removed are encoded in the information; or Perform region operations on the point cloud stream, wherein the region operations include region segmentation, filtering and / or clustering of target objects, and the parameters of the operations are defined in the information carried by the one or more augmented reality codes.

8. A three-dimensional environment modeling device, the device comprising: A scanning device used to scan the surrounding environment in order to obtain point cloud flow; A detection device for detecting one or more augmented reality codes in the surrounding environment, wherein the augmented reality code refers to a marker that performs a specific operation on a point cloud stream obtained after scanning, and wherein the augmented reality code is placed in the surrounding environment; as well as An execution device is configured to manipulate the point cloud stream based at least on information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes, in order to model the surrounding environment. The actuator is configured as follows: Based on information carried by the one or more augmented reality codes, a geometric structure is constructed at the location of the one or more augmented reality codes, wherein the orientation of the geometric structure relative to the one or more augmented reality codes and the content of the geometric structure are encoded in the information; or Based on information carried by the one or more augmented reality codes, obstacles or boundaries at or near the location of the one or more augmented reality codes are removed from the point cloud stream, wherein the extent and orientation of the region of the points to be removed are encoded in the information; or Perform region operations on the point cloud stream, wherein the region operations include region segmentation, filtering and / or clustering of target objects, and the parameters of the operations are defined in the information carried by the one or more augmented reality codes.

9. The three-dimensional environment modeling device as described in claim 8, wherein, The scanning device is configured to scan the surrounding environment from multiple angles using an RGB-D camera.

10. The three-dimensional environment modeling device as described in claim 8, further comprising: A recording device for recording the point cloud flow.

11. The three-dimensional environment modeling device as described in claim 9, wherein, The detection device includes: The first detection unit is configured to, while scanning the surrounding environment, utilize an augmented reality tracking library to detect one or more augmented reality codes in the surrounding environment and their degrees of freedom pose relative to the RGB-D camera; and The first recording unit is used to record the pose of the degrees of freedom and register it to the point cloud coordinate system.

12. The three-dimensional environment modeling device as described in claim 8, wherein, The one or more augmented reality codes are placed on the surface of the target object.

13. The three-dimensional environment modeling device as described in claim 8, wherein, The geometry is a polygonal boundary, a wall, a cuboid, or a cylinder.

14. A three-dimensional environment modeling device for industrial robots, the device comprising: A scanning device is used to scan the environment surrounding the industrial robot in order to obtain point cloud flow; A detection device for detecting one or more augmented reality codes in the surrounding environment, wherein the augmented reality code refers to a marker that performs a specific operation on a point cloud stream obtained after scanning, and wherein the augmented reality code is placed in the surrounding environment; An execution device for operating on the point cloud stream based at least on information carried by the one or more augmented reality codes and the location of the one or more augmented reality codes; as well as A processing device is used to process the manipulated point cloud flow into a surface mesh for modeling the surrounding environment. The actuator is configured as follows: Based on information carried by the one or more augmented reality codes, a geometric structure is constructed at the location of the one or more augmented reality codes, wherein the orientation of the geometric structure relative to the one or more augmented reality codes and the content of the geometric structure are encoded in the information; or Based on information carried by the one or more augmented reality codes, obstacles or boundaries at or near the location of the one or more augmented reality codes are removed from the point cloud stream, wherein the extent and orientation of the region of the points to be removed are encoded in the information; or Perform region operations on the point cloud stream, wherein the region operations include region segmentation, filtering and / or clustering of target objects, and the parameters of the operations are defined in the information carried by the one or more augmented reality codes.

15. A computer storage medium comprising instructions that, when executed, perform a three-dimensional environment modeling method as described in any one of claims 1 to 7.

16. An industrial robot operating platform, comprising a three-dimensional environment modeling device as claimed in any one of claims 8 to 14.

Images