Three-dimensional reconstruction data processing method and device and computer equipment

Abstract

The invention relates to a three-dimensional reconstruction data processing method, and the method comprises the steps: controlling an image module of a robot to rotate in at least one direction, and collecting the image data of an environment at different visual angles in the rotation process; acquiring actual offset information of the image module; according to the actual offset information, obtaining a pose conversion relation of an image module pose relative to a robot coordinate system; obtaining multi-frame three-dimensional point clouds of the environment at different visual angles of a world coordinate system according to the image data, the pose conversion relation of the pose of the image module relative to the coordinate system of the robot and the relative relation between the coordinate systems; and fusing the multi-frame three-dimensional point clouds of the environment in the world coordinate system to obtain a three-dimensional reconstruction result of the environment. According to the method, the visual angle range of robot image acquisition is enlarged, multi-frame three-dimensional point clouds of different visual angles are superposed to obtain a more accurate and detailed three-dimensional reconstruction result with a larger visual field angle in a world coordinate system, and the three-dimensional reconstruction quality is improved.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a method, device and computer equipment for processing three-dimensional reconstruction data. Background Art

[0002] With the development of robot technology, robots have greatly brought convenience to people's lives. Especially with the development of cleaning robots and AI technology, the development of the cleaning robot industry has ushered in a new booming period. Cleaning robots are becoming an important component of homes and other entertainment venues, playing an increasingly important role in cleaning services.

[0003] For robots, 3D reconstruction is mainly used to build scene information and various maps, so that the robot can know its position in the world environment and can perceive the location information of surrounding obstacles. Through path planning, the robot can achieve positioning, navigation, obstacle avoidance and path planning. Therefore, accurate 3D reconstruction is very important for improving the performance of robots. Traditional 3D reconstruction methods have the problem of poor reconstruction effect. Summary of the invention

[0004] Based on this, it is necessary to provide a method for processing machine three-dimensional reconstruction data, a robot, a computer-readable storage medium and a computer program product that can improve the reconstruction effect and enhance the robot performance in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for processing three-dimensional reconstruction data, the method comprising:

[0006] Controlling the image module of the robot to rotate in at least one direction and collecting image data of the environment at different viewing angles during the rotation process;

[0007] Acquiring actual offset information of the image module;

[0008] According to the actual offset information, a posture transformation relationship of the image module posture relative to the robot coordinate system is obtained;

[0009] Obtaining a multi-frame three-dimensional point cloud of the environment in the world coordinate system according to the image data, the posture conversion relationship of the image module posture relative to the robot coordinate system, and the relative relationship between the coordinate systems;

[0010] The three-dimensional point clouds of multiple frames of the environment in the world coordinate system are fused to obtain a three-dimensional reconstruction result of the environment.

[0011] The above-mentioned method for processing 3D reconstruction data utilizes an image module that is movable in at least one direction, acquires image data during movement, obtains actual offset information of the image module, determines the position conversion relationship of the image module posture relative to the robot coordinate system, and then utilizes the position conversion relationship of the image module posture relative to the robot coordinate system and the relative relationship between the coordinate systems to obtain a multi-frame 3D point cloud of the environment in the world coordinate system. Since the image module is movable in at least one direction, for the robot, the viewing angle range of the robot image acquisition is increased, and a multi-frame 3D point cloud of the environment in different viewing angles in the world coordinate system is obtained. The multi-frame 3D point cloud of different viewing angles is superimposed and matched to the world coordinate system, and a 3D reconstruction result with a larger viewing angle, more accuracy, and more details in the world coordinate system is obtained, thereby improving the quality of 3D reconstruction.

[0012] In a second aspect, the present application provides a method for processing three-dimensional reconstruction data, the method comprising:

[0013] Get the robot's location information;

[0014] Taking the position information of the robot as a reference, obtaining a multi-frame target three-dimensional point cloud within the target viewing angle in the world coordinate system from the three-dimensional reconstruction result; the three-dimensional reconstruction result is constructed based on image data collected by an image module of the robot, and the image module rotates in at least one direction and collects image data of the environment at different viewing angles during the rotation process;

[0015] Processing is performed based on the multi-frame target three-dimensional point cloud.

[0016] The above-mentioned method for processing 3D reconstruction data obtains a multi-frame target 3D point cloud of the environment within the target range in the world coordinate system from the 3D reconstruction result by taking the robot's position information as a reference. Since the 3D reconstruction result includes the 3D point cloud of the environment collected at different perspectives by the image module with an adjustable angular viewing angle range, the multi-frame target 3D point cloud extracted from the 3D reconstruction result has multiple perspectives. Processing based on the multi-frame target 3D point cloud can utilize rich 3D point clouds from multiple perspectives to improve robot performance.

[0017] In a third aspect, the present application provides a robot comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the steps of the method described in the above embodiments of the claims when executing the computer program.

[0018] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the methods of the above-mentioned embodiments are implemented.

[0019] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the methods of the above embodiments when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of the structure of a robot in one embodiment;

[0021] Figure 2 A schematic diagram of signal flow between a sensing part and a control part of a robot in one embodiment;

[0022] Figure 3 is a flow chart of a method for processing three-dimensional reconstruction data in one embodiment;

[0023] Figure 4 is a schematic diagram of the relationship between various coordinate systems in an embodiment;

[0024] Figure 5 A flowchart of the steps of obtaining a multi-frame three-dimensional point cloud of an environment at different perspectives in a world coordinate system according to the image data, the posture conversion relationship of the image module posture relative to the robot coordinate system, and the relative relationship between the coordinate systems in one embodiment;

[0025] Figure 6 A schematic flow chart of the steps of obtaining the posture transformation relationship of the image module posture relative to the robot coordinate system according to the actual offset information in one embodiment;

[0026] Figure 7 is a schematic diagram of an adjustment shaft according to an embodiment;

[0027] Figure 8 is a flow chart of a method for processing three-dimensional reconstruction data in another embodiment;

[0028] Fig. 9 A schematic diagram of a process flow of a method for processing three-dimensional reconstruction data in an implementation example;

[0029] Fig.10 A schematic flow chart of a method for processing three-dimensional reconstruction data in yet another embodiment;

[0030] Fig.11 is a structural block diagram of a device for processing three-dimensional reconstruction data in one embodiment;

[0031] Fig.12 is a structural block diagram of a device for processing three-dimensional reconstruction data in yet another embodiment;

[0032] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] The present application embodiment provides a robot, such as Figure 1 As shown, the robot comprises a robot body 101 , a module adjustment device 102 and an image module 103 . The module adjustment device 102 is mounted on the robot body 101 , and the image module 103 is mounted on the module adjustment device 102 .

[0035] Specifically, the image module 103 may include a digital image acquisition device and a depth image acquisition device.

[0036] Specifically, the module adjustment device 102 is a pan-tilt structure including at least one axis, which can drive the image module 103 to rotate in at least one direction.

[0037] The module adjustment device 102 can be a single-axis gimbal structure, including any one of a course axis, a pitch axis or a roll axis, which can control the image module 102 to move horizontally or vertically relative to the robot body 101, or control the image module 102 to rotate left and right relative to the robot body 101.

[0038] The module adjustment device 102 can also be a dual-axis gimbal structure. In one embodiment, the module adjustment device 102 includes a yaw axis and a pitch axis, which can control the image module 102 to move in the horizontal direction and / or vertical direction relative to the robot body 101. In one embodiment, the module adjustment device 102 includes a pitch axis and a roll axis, which can control the image module 102 to move in the vertical direction relative to the robot body 101, and / or, control the image module 102 to rotate relative to the robot body 101. In one embodiment, the module adjustment device 102 includes a yaw axis and a roll axis, which can control the image module 102 to move in the horizontal direction relative to the robot body 101, and / or, control the image module 102 to rotate relative to the robot body 101.

[0039] The module adjustment device 102 may also be a three-axis gimbal structure. In one embodiment, the module adjustment device 102 includes a course axis (controlling the horizontal direction), a pitch axis (controlling the vertical direction) and a roll axis (controlling rotation), which can control the image module 102 to move in the vertical direction and / or horizontal direction relative to the robot body 101, and / or control the image module 102 to rotate relative to the robot body 101.

[0040] like Figure 2 As shown, the sensing and control part of the robot includes:

[0041] Controller 201, driver 202 and multiple types of sensors 203, multiple types of sensors may include image module, IMU (Inertial Measurement Unit), laser radar, infrared sensor, collision sensor, ultrasonic sensor, wall sensor, cliff sensor and anti-fall sensor, etc. Among them, each sensor in the multiple types of sensors 203 and the driver 202 are connected to the controller 201 by signal, and the controller 201 processes the sensor data collected by the multiple types of sensors, makes a decision, generates a control instruction based on the decision result, and sends the control instruction to the driver 202. The driver 202 may include a driver for the robot driving wheel, a driver for the module adjustment device, etc. The decision may include navigation, obstacle avoidance and control of various components of the robot.

[0042] The robot refers to a self-moving device equipped with a sensor device that can move autonomously to perform tasks. In one embodiment, the robot includes but is not limited to a cleaning robot, a food delivery robot, and a logistics robot. The cleaning robot includes robots used for cleaning home and non-home scenes, such as sweepers, mops, automatic floor scrubbers, entertainment venues (such as swimming pool cleaners), and other cleaning robots.

[0043] In one embodiment, Figure 3 As shown, a method for processing three-dimensional reconstruction data is provided, and the method is applied to Figure 2 The controller in the example is used to illustrate the following steps:

[0044] Step 302 , controlling the image module of the robot to rotate in at least one direction, and collecting image data of the environment at different viewing angles during the rotation process.

[0045] The image module refers to a sensor device and combination for acquiring image data. Image data may include digital images and depth images. In one embodiment, the image module includes a digital image acquisition device and a depth image acquisition device. The image acquisition device is not limited to conventional cameras and cameras, and includes various sensors for semantic recognition / navigation and obstacle avoidance after acquiring data. The depth image acquisition device is not limited to various sensors such as TOF / 3DTOF / binocular cameras that can directly or indirectly acquire depth images. The image module may also include an RGBD camera. Functionally, the RGBD camera adds a depth measurement function to the function of an ordinary RGB image acquisition device, and can collect digital images and depth images.

[0046] The field of view of the image module is fixed. The field of view refers to the angle between the outer contour of the module where the sensor can obtain data and the center of the module. It is an angle range. The field of view includes horizontal and vertical field of view, which is similar to the range of viewing angles that the human eye can see when the eyeball is not moving.

[0047] Among them, the image module can be rotated in at least one direction through the module angle adjustment device. At least one direction may include at least one of the horizontal direction, the vertical direction and the depth direction. Compared with the solution in which the image module is fixed, the image module of this embodiment can be rotated in at least one direction, the field of view of the image module remains unchanged, and the direction of the image module changes, thereby changing the viewing angle of the image module. For example, the image module is movable in the vertical direction, such as moving upward, or moving upward. Compared with the solution in which the image module is fixed, the viewing angle of the image module in the height direction can be adjusted, such as having a higher or lower viewing angle. For another example, the image module is movable in the horizontal direction, and compared with the solution in which the image module is fixed, the viewing angle of the image module in the horizontal direction can be increased, such as having a left or right viewing angle. For another example, the image module can be rotated 360 degrees horizontally, and the image module can be controlled to rotate clockwise or counterclockwise. Compared with the direction in which the image module is fixed, the circumferential angle of the image module can be increased, and the viewing angle in the circumferential direction can be increased.

[0048] For example, if there is a sofa in the environment, the solution with a fixed image module requires controlling the robot to move in order to see the sofa from different angles. However, in the solution of the movable image module of this embodiment, only the module adjustment device of the robot needs to be controlled, and different parts of the sofa can be seen from the top, bottom, left or right angles, so that the whole picture of the sofa can be obtained.

[0049] Compared with the application scenario where the image module has a fixed angle, the image module in this embodiment rotates in at least one direction and captures multiple frames of digital images and depth images during the rotation process. Therefore, the viewing angle range that the image module can capture in the corresponding direction is increased, and splicing images of different viewing angles in this direction can obtain a larger viewing angle range in this direction.

[0050] For example, the image module is movable in at least one of the horizontal and vertical directions, and multiple frames of digital images and depth images are collected during the rotation process. Since the image module can move in the vertical direction, the viewing angle range in the vertical direction that the image module can collect is variable. For example, a larger vertical viewing angle range can be obtained by splicing the image after the vertical field of view is expanded. Since the image module can rotate in the horizontal direction, the viewing angle range in the horizontal direction that the image module can collect is variable. For example, a larger horizontal viewing angle range can be obtained by splicing the image after the horizontal field of view is expanded without the need for a mobile robot. Since the image module can rotate 360 ​​degrees horizontally, the rotational viewing angle range that the image module can collect is variable. For example, a larger circumferential viewing angle range can be obtained by splicing the image after the rotational field of view is expanded without the need for a mobile robot.

[0051] As mentioned above, the module adjustment device 102 can be a single-axis gimbal structure, a dual-axis gimbal structure or a three-axis gimbal structure, and can rotate in the direction corresponding to the corresponding axis. The module angle adjustment device adopts a two-axis gimbal structure, which includes a yaw axis and a pitch axis. The yaw axis adjusts the horizontal angle, and the pitch axis adjusts the vertical angle. The yaw axis of the module angle adjustment device is used to adjust the horizontal direction of the module image. The information collected by the module during the rotation process can be spliced ​​to obtain an image with an expanded horizontal field of view or its form data. The pitch axis of the module angle adjustment device is used to adjust the vertical direction of the module. The information collected by the module during the rotation process can be spliced ​​to obtain an image with an expanded vertical field of view or its form data. The roll axis of the module angle adjustment device is used to adjust the rotation angle of the module image. The information collected by the module during the rotation process can be spliced ​​to obtain an image with an expanded circumferential viewing angle or its form data.

[0052] In response to the 3D reconstruction instruction, the robot's image module can be controlled to rotate in at least one direction to perform 3D reconstruction. The 3D reconstruction instruction can be set to be triggered based on specific conditions, can be set to be triggered by the robot in real time, or can be set to be triggered in response to user settings.

[0053] In one embodiment, the controller controls at least one adjustment axis of the robot module angle adjustment device to rotate in response to the three-dimensional reconstruction instruction, such as controlling at least one of the heading axis, the pitch axis, and the roll axis to rotate, thereby driving the image module to rotate in the horizontal direction or the vertical direction, or driving the image module to rotate. The image module collects multiple frames of digital images and depth images of the environment during the rotation process.

[0054] Among them, the digital image (RGB) can be collected by an image acquisition device (such as a camera) in the image module, the depth image (D) can be collected by a depth sensor, and the digital image and depth image can also be collected by an RGBD camera.

[0055] In the traditional way, the position and angle of the robot's image module are fixed. Once designed and produced, the angles for obtaining information in the horizontal and vertical directions are fixed. The viewing angle range in the vertical direction that the image module can collect is fixed, while the viewing angle range in the horizontal direction needs to be adjusted by moving the robot. In this embodiment, the image module of the device is movable, thereby expanding the collection viewing angle range of the image module.

[0056] Step 304: Acquire actual offset information of the image module.

[0057] The actual offset information of the image module represents the offset degree of the rotated image module relative to the image module in the initial state.

[0058] In order to facilitate algorithm calibration and attitude solution, the image module is installed on the module adjustment device, and the center of the image module can be aligned with the center of the pitch axis of the module adjustment device, so that the image module coordinate system and the origin of the pitch axis coordinate system of the module adjustment device are aligned. The center of the image module can also be aligned with the center of the heading axis of the module adjustment device, so that the image module coordinate system and the origin of the horizontal coordinate system of the module adjustment device are aligned. In one embodiment, the relationship between the coordinate systems is as follows: Figure 4 shown.

[0059] Among them, the adjustment angle of the module angle adjustment device is controlled by a motor, and a code disk is provided on each adjustment axis. The rotation angle of each adjustment axis can be obtained through the code disk, and then the actual offset information of the image module can be obtained. Taking the module adjustment device including the heading axis and the pitch axis as an example, if the module adjustment device only rotates on the heading axis, the actual offset information of the image module can be determined according to the rotation angle on the heading axis. If the module adjustment device only rotates on the pitch axis, the actual offset information of the image module can be determined according to the rotation angle on the pitch axis. If the module adjustment device rotates on the heading axis and the pitch axis, the actual offset information of the image module can be determined according to the rotation angles on the pitch axis and the heading axis.

[0060] Step 306: Obtain a posture transformation relationship of the image module posture relative to the robot coordinate system according to the actual offset information.

[0061] The robot coordinate system refers to a coordinate system constructed with a certain point of the robot as the origin. In one implementation, the robot coordinate system can be constructed with the center of the robot as the origin.

[0062] Among them, the positional relationship between the image module and the robot body is obtained in advance through calibration, and then the position conversion relationship of the image module posture relative to the robot coordinate system is obtained according to the actual offset information and the positional relationship between the image module and the robot body. According to the position conversion relationship of the image module posture relative to the robot coordinate system, the coordinate data in the image module coordinate system collected during rotation can be converted to obtain the coordinate data in the robot coordinate system. For example, for the three-dimensional point cloud corresponding to a certain pixel point in the digital image during rotation, according to the conversion relationship between the image coordinate system and the image module coordinate system, it is converted to the coordinate information in the image module coordinate system, and then according to the position conversion relationship of the image module posture relative to the robot coordinate system, the coordinate information in the image module coordinate system is converted to the robot coordinate system, and the coordinate information in the three-dimensional point cloud corresponding to a certain pixel point in the digital image collected during rotation in the robot coordinate system is obtained.

[0063] Step 308, obtaining a multi-frame three-dimensional point cloud of the environment at different perspectives in the world coordinate system based on the image data, the posture transformation relationship of the image module posture relative to the robot coordinate system, and the relative relationship between coordinate systems.

[0064] In order to obtain the correspondence between the pixels in the digital image and the pixels in the depth image, the digital image and the depth image can be aligned. Specifically, alignment refers to determining the correspondence between the pixels in the digital image and the pixels in the depth image. On this basis, based on the correspondence between the pixels in the digital image and the pixels in the depth image, the digital image and the depth image are fused to obtain multi-frame 3D point cloud information in the module coordinate system.

[0065] Specifically, for each point cloud in the multi-frame three-dimensional point cloud of the module coordinate system, the three-dimensional coordinate information of each point cloud in the image module coordinate system is converted to the robot coordinate system according to the posture transformation relationship of the image module relative to the robot coordinate system and the relative relationship between the image module coordinate system and the robot coordinate system. Then, based on the transformation relationship between the robot coordinate system and the world coordinate system obtained by a predetermined calibration, the coordinate coefficients in the robot coordinate system are converted to the world coordinate system to obtain the three-dimensional coordinate information of the multi-frame three-dimensional point cloud information of the module coordinate system in the robot coordinate system.

[0066] Step 310 , fusing multiple frames of three-dimensional point clouds of the environment in the world coordinate system to obtain a three-dimensional reconstruction result of the environment.

[0067] Specifically, feature matching and registration are performed on multiple frames of three-dimensional point clouds in the world coordinate system. Since the image module of the robot can rotate in at least one direction, the viewing angle range of the image module can be expanded in the horizontal direction, and / or the vertical direction, and / or the circumferential direction, so that digital images and depth images with a larger viewing angle range in the horizontal direction, and / or the vertical direction, and / or the circumferential direction can be collected, and more three-dimensional point cloud data of different viewing angles can be obtained. Therefore, in the process of registering the three-dimensional point cloud, the three-dimensional point cloud data of the same object at different viewing angles can be obtained through feature configuration and registration, and then the multiple frames of three-dimensional point clouds at different times and different viewing angles can be superimposed and matched to the world coordinate system, so as to obtain a three-dimensional map with a larger viewing angle, more accuracy, and more details in the world coordinate system.

[0068] The above-mentioned method for processing 3D reconstruction data utilizes an image module that can rotate in at least one direction, collects image data during the rotation process, obtains the actual offset information of the image module, determines the posture conversion relationship of the image module posture relative to the robot coordinate system, and then utilizes the posture conversion relationship of the image module posture relative to the robot coordinate system and the relative relationship between the coordinate systems to obtain a multi-frame 3D point cloud of the environment in the world coordinate system. Since the image module can rotate in at least one direction, for the robot, the viewing angle range of the robot image acquisition is increased, and a multi-frame 3D point cloud of the environment in different viewing angles in the world coordinate system is obtained. The multi-frame 3D point cloud of different viewing angles is superimposed and matched to the world coordinate system, and a larger viewing angle, more accurate, and more detailed 3D reconstruction result in the world coordinate system is obtained, which improves the quality of 3D reconstruction.

[0069] In another embodiment, the image data includes a digital image and a depth image. Figure 5 As shown, according to the image data, the posture transformation relationship of the image module posture relative to the robot coordinate system, and the relative relationship between the coordinate systems, a multi-frame three-dimensional point cloud of the environment at different perspectives in the world coordinate system is obtained, including:

[0070] Step 502 : Acquire alignment information of the digital image and the depth image, and fuse the digital image and the depth image based on the alignment information to obtain a multi-frame three-dimensional point cloud of the environment in the module coordinate system.

[0071] The digital image is an image represented in the form of a two-dimensional array, and its digital unit is a pixel; the gray value of each pixel is stored in the corresponding two-dimensional matrix. The digital image can be a grayscale image or a color image. The color image can be an RGB image.

[0072] A depth image, also called a range image, refers to an image that uses the distance (depth) values ​​from the image collector to each point in the scene as pixel values.

[0073] For the scheme of collecting digital images and depth images respectively by an image acquisition device and a depth acquisition device, the spatial coordinate systems of the digital image and the depth image are different. The coordinate system of the digital image is an image coordinate system established with the origin of the image acquisition device as the center, while the coordinate system of the depth image is a depth coordinate system established with the origin of the depth acquisition device as the center. Therefore, the digital image and the depth image are independent of each other.

[0074] In order to obtain the correspondence between the pixels in the digital image and the pixels in the depth image, the digital image and the depth image may be aligned. Specifically, alignment refers to determining the correspondence between the pixels in the digital image and the pixels in the depth image. The specific alignment method may use a classical algorithm or a trained model to achieve alignment of the image and depth data.

[0075] One alignment method can be: obtain the intrinsic parameter matrix of the image acquisition device, the intrinsic parameter matrix of the depth acquisition device, and the extrinsic parameter rotation matrix and offset matrix of the image acquisition device and the depth acquisition device, traverse the coordinates of each pixel point in the digital image, and calculate the coordinates of the pixel points in the depth image corresponding to the pixel points in the digital image through the alignment formula, so as to obtain the correspondence between the pixel points in the digital image and the pixel points in the depth image.

[0076] On this basis, based on the correspondence between the pixels in the digital image and the pixels in the depth image, the digital image and the depth image are fused to obtain a three-dimensional point cloud.

[0077] Correspondingly, according to the correspondence between the pixels in the digital image and the pixels in the depth image, the depth value of each pixel in the digital image (that is, the distance from a point in the real scene corresponding to the pixel to the camera) can be obtained, and then each pixel in the digital image can be converted to the module coordinate system to obtain multi-frame three-dimensional point cloud information in the module coordinate system.

[0078] Step 504, obtaining a multi-frame three-dimensional point cloud of the environment in the world coordinate system based on the posture transformation relationship of the image module posture relative to the robot coordinate system, the multi-frame three-dimensional point cloud of the environment in the module coordinate system, and the relative relationship between the coordinate systems.

[0079] Specifically, for each point cloud in the multi-frame three-dimensional point cloud of the module coordinate system, the three-dimensional coordinate information of each point cloud in the image module coordinate system is converted to the robot coordinate system according to the posture transformation relationship of the image module relative to the robot coordinate system, and then based on the predetermined calibrated transformation relationship between the robot coordinate system and the world coordinate system, the coordinate coefficients in the robot coordinate system are converted to the world coordinate system to obtain the three-dimensional coordinate information of the multi-frame three-dimensional point cloud information of the module coordinate system in the robot coordinate system.

[0080] Then, feature matching and registration are performed on multiple frames of 3D point clouds in the world coordinate system. Since the robot's image module can rotate in at least one direction, the viewing angle of the image module is expanded, so that image data with a larger viewing angle range can be collected, such as digital images and depth images with a larger viewing angle range in the horizontal direction and / or vertical direction, and more 3D point cloud data with different viewing angles can be obtained. Thus, in the process of 3D point cloud registration, 3D point cloud data of the same object at different viewing angles can be obtained through feature configuration and registration, and then multiple frames of 3D point clouds at different times and angles can be superimposed and matched to the world coordinate system, so as to obtain a 3D reconstruction result with a larger viewing angle, more accuracy, and more details in the world coordinate system.

[0081] In another embodiment, the fusing of multiple frames of three-dimensional point clouds of the environment in the world coordinate system to obtain a three-dimensional reconstruction result of the environment may include: preprocessing the multiple frames of three-dimensional point cloud information in the module coordinate system, inputting the preprocessed multiple frames of three-dimensional point clouds of the environment in the world coordinate system into a pre-trained fusion model, and outputting the three-dimensional reconstruction result from the fusion model to obtain a three-dimensional map of the environment in the world coordinate system.

[0082] Specifically, the preprocessing may include processing multiple frames of three-dimensional point cloud information in the module coordinate system into a format acceptable to the fusion model, such as a point cloud data format.

[0083] The fusion model is pre-trained based on the collected multi-frame 3D point clouds and corresponding 3D reconstruction results in the world coordinate system. The training process of the fusion model includes:

[0084] Data preparation stage: In the data preparation stage, multi-frame 3D point cloud information of the world coordinate system and the corresponding 3D reconstruction results are collected and divided into a training set and a test set;

[0085] In the model building phase, a model is selected and the corresponding network structure is built based on the selected model. The model can use PointNet, PointCNN, etc. The performance and usage scenarios of different models are slightly different, so it is necessary to select a suitable model based on the actual situation.

[0086] In the model training phase, the model is trained using the training set and the model parameters are adjusted so that the model can accurately predict the 3D reconstruction results.

[0087] Model evaluation: In the model evaluation phase, the test set is used to evaluate the model and calculate the model's accuracy, recall and other indicators;

[0088] Model optimization: During the model optimization phase, the model is optimized based on the evaluation results, such as adjusting the network structure, modifying the loss function, etc.

[0089] In this embodiment, a neural network model is used to fuse multiple frames of three-dimensional point clouds of the environment in the world coordinate system to obtain a three-dimensional reconstruction result of the environment.

[0090] In another embodiment, feature detection and feature registration can also be used to reconstruct a three-dimensional map of the environment in the world coordinate system based on the posture transformation relationship between the image module posture and the robot coordinate system and the multi-frame three-dimensional point cloud of the environment in the module coordinate system.

[0091] In another embodiment, the image module is mounted on a module adjustment device, the module adjustment device has at least one degree of freedom, and the module adjustment device has an adjustment axis for realizing each degree of freedom; the actual offset information includes a rotation angle of the adjustment axis.

[0092] According to the actual offset information, the posture transformation relationship of the image module posture relative to the robot coordinate system is obtained, such as Figure 6 As shown, including:

[0093] Step 602: Obtain a rotation matrix of the image module relative to the robot according to the rotation angle of the adjustment axis.

[0094] Specifically, the rotation angle of the adjustment axis of the module adjustment device can be controlled by a motor, and the rotation angle of the adjustment axis can be obtained by a code disk. The code disk angle and the rotation angle are in a linear relationship, and the rotation angle of the adjustment axis is obtained according to the code disk angle and the linear relationship. Then, the rotation matrix of the image module relative to the robot is derived according to the rotation angle of the adjustment axis.

[0095] Step 604, based on the initial position relationship between the image module and the module adjustment device, and the rotation matrix of the image module relative to the robot, obtain the posture transformation relationship of the image module posture relative to the robot coordinate system.

[0096] Among them, the initial position relationship between the image module and the module adjustment device is related to the installation position. To facilitate algorithm calibration and attitude solution, the image module is installed on the module adjustment device, and the center of the image module and the pitch axis center of the module adjustment device can be aligned, so that the image module coordinate system and the origin of the pitch axis coordinate system of the module adjustment device are aligned. It is also possible to align the center of the image module and the heading axis center of the module adjustment device, so that the image module coordinate system and the origin of the horizontal coordinate system of the module adjustment device are aligned.

[0097] After installation, the initial position of the image module relative to the robot coordinate system is R_0. Since there is no rotation in the initial state, the initial position relationship between the image module and the module adjustment device can be considered to have only one translation vector T.

[0098] Therefore, the posture transformation relationship of the image module posture relative to the robot coordinate system can be obtained based on the rotation matrix R and the translation vector T.

[0099]

[0100] Among them, (x_{robot}, y_{robot}, z_{robot}) are the coordinates in the robot coordinate system, (x_{mod}, y_{mod}, z_{mod}) are the coordinates in the module coordinate system, (t_x, t_y, t_z) are the translation vector T, T is the known initial value, and R is the real-time rotation matrix.

[0101] In this embodiment, the image module is installed on the module adjustment device, and the module adjustment device has an adjustment axis for realizing the degree of freedom. The rotation angle of the adjustment axis can be easily detected to obtain the actual offset information of the image module, and then the rotation angle of the adjustment axis is used to obtain the rotation matrix of the image module relative to the robot, and then the position transformation relationship of the image module posture relative to the robot coordinate system is obtained according to the initial position relationship between the image module and the module adjustment device, and the rotation matrix. This method simplifies the calculation amount of the rotation matrix and improves the calculation efficiency of the position transformation relationship of the image module posture relative to the robot coordinate system.

[0102] In one embodiment, the adjustment axis comprises a single direction adjustment axis. Figure 7 As shown, the module adjustment device is adjustable in the X-axis, Y-axis or Z-axis direction, that is, the module adjustment device can set an adjustment axis on the X-axis, Y-axis or Z-axis, and the corresponding adjustment axes are the heading axis, pitch axis and roll axis respectively.

[0103] The rotation matrix of the image module relative to the robot is obtained according to the rotation angle of the adjustment axis, including: according to the rotation angle of the adjustment axis in the single direction, the rotation matrix of the adjustment axis in the single direction is obtained; the rotation matrix of the image module relative to the robot includes the rotation matrix of the adjustment axis in the single direction.

[0104] Specifically, according to the rotation angle of the adjustment axis in a single direction, the rotation matrix of the adjustment axis in a single direction is obtained. Taking the adjustment axis in a single direction as an example, it can adjust the horizontal direction. Assuming that the rotation angle is α, the rotation matrix in this direction is

[0105]

[0106] Take the pitch axis as an example, which can adjust the vertical direction. Assuming the rotation angle is β, the rotation matrix in this direction is:

[0107]

[0108] Take the roll axis as an example, which can adjust the rotation direction. Assuming the rotation angle is γ, the rotation matrix in this direction is:

[0109]

[0110] For a solution in which the module adjustment device only includes an adjustment axis in a single direction, the rotation matrix of the image module relative to the robot is the rotation matrix of the adjustment axis in the single direction.

[0111] In this embodiment, the module adjustment device can be in a single direction, and the single-direction adjustment axis is used to enable the image module to rotate in a single direction, and then the posture transformation of the image module relative to the robot coordinate system is calculated based on the rotation angle of the rotation axis. This method simplifies the calculation of the rotation matrix and improves the calculation efficiency of the posture transformation relationship of the image module posture relative to the robot coordinate system.

[0112] In another embodiment, the adjustment axis includes adjustment axes in at least two directions. Figure 7 As shown, the module adjustment device is adjustable in at least two directions of the X-axis, Y-axis or Z-axis directions, that is, the module adjustment device can be provided with adjustment axes in at least two directions of the X-axis, Y-axis or Z-axis.

[0113] According to the rotation angle of the adjustment axis, the rotation matrix of the image module relative to the robot is obtained, including: according to the rotation angle of the adjustment axis in at least two directions, the rotation matrix of the adjustment axis in each direction is obtained; according to the product of the rotation matrices of the adjustment axis in each direction, the rotation matrix of the image module relative to the robot is obtained.

[0114] Specifically, first, according to the rotation angle of the adjustment axis in each direction, the rotation matrix of the adjustment axis in that direction is obtained. For example, if the module adjustment device includes a yaw axis and a pitch axis, assuming that the rotation angle of the yaw axis is α, then the rotation matrix in that direction is R x (α), the rotation angle of the pitch axis is β, then the rotation matrix in this direction is R y (β), then the rotation matrix of the image module relative to the robot is the product of the two, which is:

[0115] R=R x (α)R y (β)

[0116] For example, if the module adjustment device includes a yaw axis and a roll axis, assuming that the rotation angle of the yaw axis is α, then the rotation matrix in this direction is R x (α), the rotation angle of the roll axis is γ, then the rotation matrix in this direction is R z(γ), the rotation matrix of the image module relative to the robot is the product of the two, which is:

[0117] R=R x (α)R z (γ)

[0118] For example, if the module adjustment device includes a pitch axis and a roll axis, assuming that the rotation angle of the pitch axis is β, then the rotation matrix in this direction is R y (β), the rotation angle of the roll axis is γ, then the rotation matrix in this direction is R z (γ), the rotation matrix of the image module relative to the robot is the product of the two, which is:

[0119] R=R y (β)R z (γ)

[0120] For example, if the module adjustment device includes a yaw axis, a pitch axis, and a roll axis, and the rotation angle of the yaw axis is α, then the rotation matrix in this direction is R x (α), the rotation angle of the pitch axis is β, then the rotation matrix in this direction is R y (β), the rotation angle of the roll axis is γ, then the rotation matrix in this direction is R z (γ), the rotation matrix of the image module relative to the robot is the product of the two, which is:

[0121] R=R x (α)R y (β)R z (γ)

[0122] In this embodiment, by setting the adjustment axes in at least two directions of the module adjustment device, the image module can be adjusted in at least two directions, and the rotation matrix of the image module relative to the robot can be calculated according to the rotation angles in each direction. Taking the adjustment axis including the horizontal heading axis and the vertical pitch axis as an example, the rotation matrix of the adjustment axis in each direction is obtained according to the rotation angles of the adjustment axis in at least two directions, including: obtaining the rotation matrix of the heading axis according to the rotation angle of the heading axis; obtaining the rotation matrix of the pitch axis according to the rotation angle of the pitch axis.

[0123] Specifically, assuming that the angles of the two motor encoders are theta_1 and theta_2, respectively, according to the linear relationship between the encoder angle and the rotation angle, the rotation angles of the pitch axis and the yaw axis can be obtained as α and β, respectively. Then the real-time rotation matrix R can be expressed as:

[0124] R=R x (α)R y (β)

[0125]

[0126]

[0127] In this embodiment, the pitch axis and yaw axis of the module adjustment device are set so that the image module can be adjusted in the vertical and horizontal directions. According to the rotation angles of the pitch axis and yaw axis, the rotation matrix of the image module relative to the robot can be calculated.

[0128] In one embodiment, the robot can trigger the 3D reconstruction command in real time to obtain the 3D reconstruction result of the environment in real time. The disadvantage of this method is that it occupies the computing power of the robot when it is working.

[0129] In another embodiment, the three-dimensional reconstruction instruction may be triggered when at least one of the robot's operating state, recognition result, task type, and task processing result meets the three-dimensional reconstruction condition.

[0130] The three-dimensional map reconstruction condition may include: the robot is in an idle state. That is, when the robot is in an idle state, the three-dimensional reconstruction instruction is triggered. This method can use the time when the robot is in an idle state to trigger the robot to perform three-dimensional map reconstruction.

[0131] The conditions for 3D map reconstruction may also include: the robot's recognition results include the robot recognizing a specific object, the robot being unable to recognize an object, and the robot not being able to recognize an object. When the robot recognizes a specific object, the robot being unable to recognize an object, and the robot not being able to recognize an object, 3D reconstruction is triggered. This method can trigger 3D reconstruction when the recognition effect is not good, and obtain a 3D reconstruction result with a larger viewing angle, more accuracy, and more detail in the world coordinate system. This can improve the quality of image recognition.

[0132] In one embodiment, when the task type is an obstacle avoidance task, a navigation task, or a path rule task, three-dimensional reconstruction is triggered, thereby utilizing a larger viewing angle, more accurate, and more detailed three-dimensional reconstruction result to improve the obstacle avoidance effect, navigation effect, and path planning effect.

[0133] In one embodiment, when the recognition result is that a specific type of obstacle is detected, or when the detection result is at least one of that the robot touches the obstacle and that the robot is trapped.

[0134] When the robot detects a specific type of obstacle, it triggers a 3D reconstruction command to reconstruct a 3D map of the environment in the world coordinate system to facilitate better obstacle avoidance planning. Specific types of obstacles may include fragile obstacles, door obstacles, feces obstacles, etc.

[0135] When it is detected that the robot touches an obstacle, a 3D reconstruction instruction is triggered to reconstruct a 3D map of the environment in the world coordinate system in order to identify the obstacle and develop an obstacle avoidance path to avoid the obstacle.

[0136] When the robot is detected to be trapped, a 3D reconstruction command is triggered.

[0137] In this embodiment, by setting the trigger conditions, it is possible to utilize idle resources, or trigger the three-dimensional reconstruction when needed, so as to avoid the three-dimensional reconstruction occupying resources and affecting the robot's task execution.

[0138] In one embodiment, controlling the image module of the robot to rotate in at least one direction includes: determining target offset information of the image module; and controlling the image module of the robot to rotate according to the target offset information.

[0139] Specifically, when 3D reconstruction needs to be triggered, the target offset information of the image module can be determined according to at least one of the object information in the environment and the task type currently performed by the robot. Then, the image module of the robot is controlled to rotate according to the target offset information to capture the image of the target perspective.

[0140] In one embodiment, when the task type performed by the robot is object recognition, the target offset information of the image module can be determined based on the object information. For example, if the entire object is not captured, the target offset information of the image module can be determined based on the position of the object to increase the viewing angle of the image module so that the entire object can be captured.

[0141] In one embodiment, when the task type performed by the robot is object recognition, the semantic recognition function requires the sensor to capture as much overall information of the object as possible to improve the recognition accuracy. Generally, a large vertical viewing angle is required. At this time, the vertical viewing angle of the image module can be increased to capture the entire object.

[0142] In one embodiment, when the robot performs a path planning task or an obstacle avoidance task, the obstacle avoidance function requires the sensor to capture as much information as possible around the body. Generally, the vertical field of view height is required to be from the ground to an angle slightly higher than the body, but the horizontal field of view angle should be as large as possible, so that the machine can obtain more surrounding information without rotating its own angle, which can improve the obstacle avoidance capability. At this time, the horizontal viewing angle of the image module can be increased to obtain sufficiently rich information around the robot body.

[0143] In this embodiment, when three-dimensional reconstruction is triggered, the robot image module is controlled to rotate according to the target offset information, and the viewing angle of the image module can be adjusted in real time and flexibly according to the situation in the task and environment, so as to capture images from the target perspective during the three-dimensional reconstruction process.

[0144] In another embodiment, determining the target offset information of the image module includes: when the recognition result of the robot meets the adjustment condition of the image module, determining the viewing angle range of the image module in the vertical direction; and determining the target offset information of the image module according to the viewing angle range in the vertical direction.

[0145] Among them, the robot's recognition task is a semantic recognition process. The semantic recognition function requires the sensor to capture as much overall information of the object as possible to improve the recognition accuracy. Generally, a large vertical viewing angle is required. For example, the probability of identifying the type of pet is higher when the whole body of the pet is seen than when only the feet are seen.

[0146] Based on this, when the robot recognizes a specific type of object, or when object recognition fails, the viewing angle range of the image module in at least one direction is determined, and the at least one direction includes a vertical direction.

[0147] In one implementation, only the viewing angle range in the vertical direction may be determined.

[0148] In one implementation, the horizontal and vertical viewing angles can be determined based on the characteristics of semantic recognition and the information of the object. For example, while expanding the vertical viewing angle, the corresponding horizontal viewing angle can be expanded based on the position information of the object and the missing information of the object, thereby determining the target viewing angle of the image module in the horizontal and vertical directions.

[0149] For example, when pet feces is found, in order to avoid touching and dragging it everywhere, the viewing angle needs to be expanded to determine the location information of the pet feces. For example, when furniture and shoes are identified, the viewing angle can be expanded to determine the location information of the furniture and shoes for app display and obstacle avoidance during cleaning. For another example, when fragile items are identified, the viewing angle can be expanded to determine the location of the fragile items to adjust the safe distance and travel speed of the machine. For another example, when objects such as doors are identified, the viewing angle can be expanded, and the category and whether they are currently closed can be used to determine whether to close the door. For another example, when a threshold is identified, the location of the threshold is determined to facilitate the determination of the action strategy, assisting the machine to pass while reducing the possibility of being trapped.

[0150] In this embodiment, based on the requirements of robot semantics, when three-dimensional reconstruction is needed, the robot is actively controlled to adjust the viewing angle range in the vertical direction, which is consistent with the requirement of semantic recognition to capture the overall information of the object to improve recognition accuracy. Therefore, the accuracy of semantic recognition can be improved.

[0151] In another embodiment, determining the target offset information of the image module includes: when the robot performs the target task, or the processing result of performing the target task based on the two-dimensional map meets the adjustment condition of the image module, determining the viewing angle range of the image module in at least one direction, wherein the at least one direction includes the horizontal direction; determining the target offset information of the image module according to the viewing angle range of the image module in at least one direction. Among them, the robot has the need for path planning and the need for obstacle avoidance. Path planning and obstacle avoidance require the sensor to capture as much information as possible around the fuselage. Generally, the vertical field of view height is required to be from the ground to an angle slightly higher than the fuselage, but the horizontal field of view angle should be as large as possible, so that the machine can obtain more surrounding information without rotating its own angle, which can improve the obstacle avoidance capability.

[0152] Specifically, the adjustment condition of the image module may include that when the processing result of executing the task based on the two-dimensional map is a task failure, the target task may include an obstacle avoidance task, a navigation task or a path planning task.

[0153] Therefore, when the robot needs to perform an obstacle avoidance task, a path planning task, or a navigation task, or when the processing result of the robot performing the obstacle avoidance task, path planning task, or navigation task based on the two-dimensional map satisfies the adjustment condition of the module, the viewing angle range of the image module in at least one direction is determined. The path planning result of the robot satisfies the adjustment condition of the module, which may be that the robot path planning fails. At this time, the viewing angle range of the image module in at least one direction may be determined, and the at least one direction includes the horizontal direction.

[0154] In one implementation, only the viewing angle range in the horizontal direction may be determined.

[0155] In one implementation, the horizontal and vertical viewing angles can be determined based on the characteristics of the path planning task and the information of the object. For example, while expanding the horizontal viewing angle, the vertical viewing angle can be expanded based on the information of the obstacle, thereby determining the target viewing angles of the image module in the horizontal and vertical directions.

[0156] In this embodiment, based on the needs of path planning, when three-dimensional reconstruction is required, the robot is actively controlled to adjust the viewing angle range including the horizontal direction, which is consistent with the requirement of path planning to capture as much information around the fuselage as possible. Therefore, the accuracy of obstacle avoidance in path planning can be improved.

[0157] Through the above analysis, it can be determined that the robot has different viewing angle requirements for semantic recognition and path planning. The semantic recognition function requires the sensor to capture as much overall information of the object as possible to improve the recognition accuracy. Generally, a large vertical viewing angle is required. For example, the probability of identifying the type of pet by seeing the whole body of a pet is greater than that of only seeing its feet. In this way, when the horizontal field of view is sufficient, the vertical field of view does not need to be guaranteed.

[0158] The obstacle avoidance function requires the sensor to capture as much information as possible around the fuselage. Generally, the vertical field of view height is required to be from the ground to slightly higher than the fuselage, but the horizontal field of view angle should be as large as possible. In this way, the machine can obtain more surrounding information without rotating its own angle, thereby improving the obstacle avoidance capability.

[0159] If the viewing angle of the image module is fixed, it cannot meet the requirements of semantics and obstacle avoidance at the same time. In this embodiment, by setting the image module to be rotatable in at least one direction, the viewing angle of the image module is adjustable, and further, it can be adjusted in the horizontal and vertical directions to solve the contradiction between the field of view of semantic recognition and obstacle avoidance. The overall solution is to meet the field of view in both horizontal and vertical directions by adjusting the horizontal and vertical angles of the cleaning robot module.

[0160] In one embodiment, a data processing method for three-dimensional reconstruction, such as Figure 8 As shown, it also includes:

[0161] Step 801, obtaining the position information of the robot.

[0162] Among them, the robot's position information can be detected based on the IMU sensor.

[0163] Step 804 , using the position information of the robot as a reference, obtaining a multi-frame target three-dimensional point cloud within the target viewing range in the world coordinate system from the three-dimensional reconstruction result.

[0164] As mentioned earlier, the robot's image module can rotate in at least one direction and collect multiple frames of digital images and depth images of the environment during the rotation process. Therefore, the 3D reconstruction result includes the 3D point cloud of the environment collected by the image module at different perspectives. It can obtain multiple frames of 3D point clouds at different times and angles by superimposing and matching them in the world coordinate system to obtain a larger perspective, more accurate and more detailed 3D reconstruction result in the world coordinate system.

[0165] On the basis of the three-dimensional map, taking the position information of the robot as a reference, a multi-frame three-dimensional point cloud of the target within the target viewing range in the world coordinate system is obtained from the three-dimensional reconstruction result.

[0166] The target viewing angle range may be preset or determined based on the current task of the robot.

[0167] Since the 3D reconstruction result has a larger viewing angle, the extracted multi-frame target 3D point cloud also has a larger viewing angle, and a rich multi-frame target 3D point cloud can be extracted from the 3D reconstruction result for subsequent processing.

[0168] Step 806: Processing based on the multi-frame target three-dimensional point cloud.

[0169] Specifically, the processing of the multi-frame target three-dimensional point cloud can be determined according to the task currently performed by the robot. If the task currently performed is a recognition task, recognition is performed based on the multi-frame target three-dimensional point cloud. If the task currently performed is a path planning task, path planning is performed based on the multi-frame target three-dimensional point cloud. If the task currently performed is obstacle avoidance, obstacle avoidance processing is performed based on the multi-frame target three-dimensional point cloud.

[0170] The method of this embodiment obtains multi-frame target three-dimensional point clouds within the target viewing angle in the world coordinate system from the three-dimensional reconstruction result by taking the robot's position information as a reference. Since the three-dimensional reconstruction result includes the three-dimensional point cloud of the environment collected at different viewing angles by the image module with an adjustable angle viewing angle, the multi-frame target three-dimensional point clouds extracted from the three-dimensional reconstruction result have multiple different viewing angles. Processing based on the multi-frame target three-dimensional point clouds can utilize rich three-dimensional point clouds from multiple viewing angles to improve robot performance.

[0171] In one embodiment, when the recognition result of the robot meets the adjustment conditions of the image module, the position information of the robot is used as a reference to obtain multiple frames of target three-dimensional point clouds within the target viewing range in the world coordinate system from the three-dimensional reconstruction result, including: obtaining the current viewing range of the robot; using the position information of the robot as a parameter, determining the three-dimensional point cloud of the current viewing range and the three-dimensional point cloud of the viewing range adjacent to the current viewing range from the three-dimensional reconstruction result.

[0172] Among them, the adjustment conditions of the image module include the robot recognizing a specific object, the robot being unable to recognize an object, the robot not being able to recognize an object, etc.

[0173] When it is detected that the robot's recognition result meets the adjustment conditions of the image module, the three-dimensional point cloud of the current viewing range is first determined in the three-dimensional map, and then the three-dimensional point cloud of the viewing range adjacent to the current viewing range is determined, such as the three-dimensional point cloud of the viewing range adjacent to the current viewing range in the vertical direction, so as to extract multi-frame three-dimensional point clouds with an expanded viewing angle.

[0174] For example, when pet feces is found, in order to avoid touching and dragging it everywhere, the viewing angle needs to be expanded to determine the location information of the pet feces. For example, when furniture and shoes are identified, the viewing angle can be expanded to determine the location information of the furniture and shoes for app display and obstacle avoidance during cleaning. For another example, when fragile items are identified, the viewing angle can be expanded to determine the location of the fragile items to adjust the safe distance and travel speed of the machine. For another example, when objects such as doors are identified, the viewing angle can be expanded, and the category and whether they are currently closed can be used to determine whether to close the door. For another example, when a threshold is identified, the location of the threshold is determined to facilitate the determination of the action strategy, assisting the machine to pass while reducing the possibility of being trapped.

[0175] Among them, after obtaining multiple frames of target three-dimensional point clouds, the three-dimensional point clouds are preprocessed, and the preprocessing includes point cloud filtering, downsampling, normal estimation and other operations to improve the accuracy of subsequent recognition.

[0176] Then, the three-dimensional point cloud of the current viewing range and the three-dimensional point cloud of the viewing range adjacent to the current viewing range are spliced ​​to obtain a spliced ​​point cloud; recognition is performed based on the spliced ​​point cloud to determine at least one of the object category, object position and object posture of the object included in the spliced ​​point cloud.

[0177] Specifically, the recognition process includes: using an object segmentation algorithm based on deep learning to segment the preprocessed point cloud and segment each object in the point cloud. For each segmented object, an object recognition algorithm based on deep learning can be used to identify each object as a corresponding category through a pre-trained model. For each identified object, a posture estimation algorithm based on deep learning can be used to estimate the position and posture of the object.

[0178] Finally, the object category, position and posture estimation results of the recognition results are output to the robot control system so that the robot can perform subsequent statistical records, navigation strategies and update and optimize the three-dimensional map.

[0179] The object is identified by stitching point clouds, which is different from the previous identification through one frame of data. If the frame of data only has an image (RGB), it is a two-dimensional identification; if it contains depth information, but there is only one frame, because the viewing angle is limited, it is only the RGBD seen in a certain direction. In this embodiment, the RGBD information of multiple perspectives is integrated to form a complete three-dimensional point cloud with color information, which is very different. For example, the traditional way is to look at people through photos, but in this embodiment, you can directly confirm the person 360*2=720 degrees without blind spots. Therefore, this method can improve the accuracy of recognition.

[0180] In another embodiment, when the identified obstacle is not in the center of the field of view, the processing based on the multi-frame target three-dimensional point cloud includes: stitching the three-dimensional point cloud of the current viewing range and the three-dimensional point cloud of the viewing range adjacent to the current viewing range to obtain a stitched point cloud; identifying at least one of the target cleaning area and obstacle information based on the stitched point cloud; determining the target adjustment angle of the image module based on at least one of the target cleaning area and obstacle information; controlling the rotation of the image parameters according to the target adjustment angle, and collecting image data of the environment at different viewing angles during the rotation process.

[0181] Specifically, when the identified obstacle is not in the center of the field of view, the viewing angle needs to be adjusted. Specifically, obtain the three-dimensional information of the environment, that is, obtain the reconstructed three-dimensional map, and process the point cloud data, such as using filtering, segmentation and registration algorithms, to obtain the three-dimensional data of the environment near the machine. Analyze and identify the three-dimensional model, such as obtaining object recognition and posture estimation, and obtain information such as the area to be cleared and the location and size of the obstacle. According to the area to be cleared and the location and size of the obstacle, calculate the angle that the gimbal needs to be adjusted so that as much of the area to be cleared as possible can be seen through multiple perspectives, and obstacles can be avoided. Then control the gimbal to adjust to the above-calculated angle to perform navigation planning, path planning and obstacle avoidance.

[0182] In another embodiment, when the processing result of the robot performing a task based on a two-dimensional map meets the adjustment conditions of the image module, the position information of the robot is used as a reference to obtain a multi-frame target three-dimensional point cloud of the environment within the target range in the world coordinate system from the three-dimensional reconstruction result, including: obtaining the current position of the robot; taking the current position of the robot as the center, obtaining a three-dimensional point cloud around the current position from the three-dimensional reconstruction result, and the surrounding three-dimensional point cloud at least includes the surrounding three-dimensional point cloud in the horizontal direction.

[0183] Specifically, the robot's path planning results meet the adjustment conditions of the image module, including failure to perform related tasks based on the two-dimensional map. When the robot performs obstacle avoidance tasks, navigation tasks, or path planning fails based on the two-dimensional map, the robot's current position and posture are obtained from the lidar. The robot needs to load the previously established three-dimensional point cloud within a certain range near the robot according to the current position. The point cloud within a certain range includes at least the surrounding three-dimensional point cloud in the horizontal direction, so as to perform navigation, obstacle avoidance and path rule processing.

[0184] Specifically, the loaded 3D point cloud map is preprocessed, including point cloud filtering, downsampling, normal estimation and other operations, to improve the accuracy of subsequent navigation and obstacle avoidance.

[0185] Then, the posture information of the robot is obtained; a local map is constructed based on the three-dimensional point cloud around the current position; and path planning, obstacle avoidance or navigation is performed according to the local map and the posture information of the robot.

[0186] Specifically, a three-dimensional map is constructed using three-dimensional point cloud data and recognition results, such as using open source libraries such as Octomap. Then, the three-dimensional map can be used for navigation planning. This method takes advantage of the rich information in the three-dimensional map, and there is no situation where multiple layers are projected onto one layer. The three-dimensional size of the machine is taken into account during planning, which can effectively avoid being trapped and colliding, and can also combine semantic information to find a more suitable navigation and path. Alternatively, the robot needs to use a navigation planning algorithm to plan a path on the three-dimensional point cloud map based on the current position and target position. Alternatively, the robot needs to use the established three-dimensional map information, combined with the number of sensors obtained in real time, to implement an obstacle avoidance algorithm, such as based on the recognition results of previous deep learning, combined with real-time laser radar data, to determine the category and space occupancy of objects on the current path, and then optimize the planned path, such as fine-tuning part of the path according to the obstacle category to control the safety distance. Finally, the robot needs to convert the control instructions output by the path tracking algorithm into the robot's rotation control instructions, such as the robot's speed and angular velocity.

[0187] In one embodiment, the method for processing three-dimensional reconstruction data of the present application is as follows: Fig. 9 As shown,

[0188] The camera and depth sensor are installed on the module adjustment device. The camera collects RGB information, and the depth sensor collects depth information. The depth information and RGB information are aligned using an alignment algorithm, and the posture information is obtained from other modules of the robot (such as IMU). Then, three-dimensional reconstruction is performed based on the posture information, aligned depth information and RGB information.

[0189] The specific process of three-dimensional reconstruction has been described in detail in the previous embodiments, and will not be repeated here. The present application utilizes a method of adjustable image module, which can adaptively open and adjust the angle, thereby improving the intelligence of the robot.

[0190] In another embodiment, a method for processing three-dimensional reconstruction data can be implemented based on the constructed three-dimensional map, such as Fig.10 As shown, the method includes:

[0191] Step 1002, obtain the position information of the robot.

[0192] Step 1004, with reference to the position information of the robot, obtain a multi-frame target three-dimensional point cloud within the target viewing angle in the world coordinate system from the three-dimensional reconstruction result; the three-dimensional reconstruction result is constructed based on the image data collected by the image module of the robot, and the image module rotates in at least one direction and collects image data of the environment at different viewing angles during the rotation process.

[0193] Step 1006: Processing based on the multi-frame target three-dimensional point cloud.

[0194] The processing method of the 3D reconstruction data of this embodiment is not limited to the method used to construct the 3D reconstruction image. Since the 3D reconstruction result includes the 3D point cloud of the environment collected by the image module with adjustable angle viewing range at different viewing angles, the multi-frame target 3D point cloud extracted from the 3D reconstruction result has multiple viewing angles, and processing based on the multi-frame target 3D point cloud can utilize the rich 3D point clouds of multiple viewing angles to improve the robot performance.

[0195] In another embodiment, when the recognition result of the robot satisfies the adjustment condition of the image module, the method of obtaining a multi-frame target three-dimensional point cloud within the target viewing angle in the world coordinate system from the three-dimensional reconstruction result with reference to the position information of the robot includes:

[0196] Get the robot's current viewing range;

[0197] Taking the position information of the robot as a parameter, the three-dimensional point cloud of the current viewing range and the three-dimensional point cloud of the viewing range adjacent to the current viewing range are determined from the three-dimensional reconstruction result.

[0198] In another embodiment, the processing based on the multi-frame target three-dimensional point cloud includes:

[0199] Splicing the three-dimensional point cloud of the current viewing angle range with the three-dimensional point cloud of the viewing angle range adjacent to the current viewing angle range to obtain a spliced ​​point cloud; the adjacent viewing angle range at least includes the adjacent viewing angle range in the vertical direction;

[0200] Recognition is performed based on the spliced ​​point cloud to determine at least one of an object category, an object position, and an object posture of an object included in the spliced ​​point cloud.

[0201] In another embodiment, when the identified obstacle is not at the center of the field of view, the processing based on the multi-frame target three-dimensional point cloud includes: splicing the three-dimensional point cloud of the current viewing angle range and the three-dimensional point cloud of the viewing angle range adjacent to the current viewing angle range to obtain a spliced ​​point cloud;

[0202] Identifying at least one of information of a target clearing area and obstacles according to the stitched point cloud;

[0203] Determining a target adjustment angle of the image module according to at least one of the target clearing area and the obstacle information;

[0204] The image parameters are controlled to rotate according to the target adjustment angle, and image data of the environment at different viewing angles are collected during the rotation process.

[0205] In another embodiment, when the processing result of the robot executing the task based on the two-dimensional map satisfies the adjustment condition of the image module, the three-dimensional point cloud of multiple frames of the environment in the target range in the world coordinate system is obtained from the three-dimensional reconstruction result with reference to the position information of the robot, including:

[0206] Get the current position of the robot;

[0207] Taking the current position of the robot as the center, a three-dimensional point cloud around the current position is obtained from the three-dimensional reconstruction result; the surrounding three-dimensional point cloud at least includes the surrounding three-dimensional point cloud in the horizontal direction.

[0208] In another embodiment, the processing based on the multi-frame target three-dimensional point cloud includes:

[0209] Acquiring posture information of the robot;

[0210] Build a local map based on the 3D point cloud around the current location;

[0211] Path planning, obstacle avoidance or navigation is performed based on the local map and the posture information of the robot.

[0212] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0213] Based on the same inventive concept, the embodiment of the present application also provides a 3D reconstruction data processing device for implementing the 3D reconstruction data processing method involved above. The implementation solutions provided by each device to solve the problem are similar to the implementation solutions recorded in the above method, so the specific limitations in the embodiments of each device provided below can refer to the limitations of the method above, and will not be repeated here.

[0214] In one embodiment, a 3D reconstruction data processing device, such as Fig.11 As shown, including:

[0215] A rotation control module 1102 is used to control the image module of the robot to rotate in at least one direction and collect image data of the environment at different viewing angles during the rotation process;

[0216] The offset acquisition module 1104 is used to acquire the actual offset information of the image module;

[0217] A relationship acquisition module 1106 is used to obtain a posture transformation relationship of the image module posture relative to the robot coordinate system according to the actual offset information;

[0218] A conversion module 1108, configured to obtain a multi-frame three-dimensional point cloud of the environment at different perspectives in a world coordinate system according to the image data, the position conversion relationship of the image module posture relative to the robot coordinate system, and the relative relationship between the coordinate systems;

[0219] The fusion module 1110 is used to fuse multiple frames of three-dimensional point clouds of the environment in the world coordinate system to obtain a three-dimensional reconstruction result of the environment.

[0220] In another embodiment, the image data includes a digital image and a depth image;

[0221] The offset acquisition module is used to obtain the alignment information of the digital image and the depth image, and fuse the digital image and the depth image based on the alignment information to obtain a multi-frame three-dimensional point cloud of the environment in the module coordinate system; according to the posture transformation relationship of the image module posture relative to the robot coordinate system, the multi-frame three-dimensional point cloud of the environment in the module coordinate system, and the relative relationship between the coordinate systems, the multi-frame three-dimensional point cloud of the environment in the world coordinate system is obtained.

[0222] In another embodiment, the image module is mounted on a module adjustment device, the module adjustment device has at least one degree of freedom, and the module adjustment device has an adjustment axis for realizing each degree of freedom; the actual offset information includes a rotation angle of the adjustment axis;

[0223] The offset acquisition module is used to obtain the rotation matrix of the image module relative to the robot according to the rotation angle of the adjustment axis; based on the initial position relationship between the image module and the module adjustment device, and the rotation matrix of the image module relative to the robot, obtain the posture transformation relationship of the image module posture relative to the robot coordinate system.

[0224] In another embodiment, when the adjustment axis includes an adjustment axis in a single direction, the relationship acquisition module is used to obtain the rotation matrix of the adjustment axis in the single direction according to the rotation angle of the adjustment axis in the single direction; the rotation matrix of the image module relative to the robot includes the rotation matrix of the adjustment axis in the single direction.

[0225] In another embodiment, when the adjustment axis includes adjustment axes in at least two directions, the relationship acquisition module is used to obtain the rotation matrix of the adjustment axis in each direction according to the rotation angle of the adjustment axis in the at least two directions; and obtain the rotation matrix of the image module relative to the robot according to the product of the rotation matrices of the adjustment axis in each direction.

[0226] In another embodiment, it also includes a trigger module, which is used to trigger the image module controlling the robot to rotate in at least one direction when at least one of the robot's operating state, recognition result, task type and task processing result meets the three-dimensional reconstruction condition.

[0227] In another embodiment, the rotation control module includes:

[0228] The offset determination module is used to determine the target offset information of the image module.

[0229] The rotation module is used to control the rotation of the image module of the robot according to the target offset information.

[0230] In another embodiment, the offset determination module is used to determine the target viewing angle range of the image module in at least one direction when the recognition result of the robot meets the adjustment condition of the image module, and the at least one direction includes a vertical direction; and determine the target offset information of the image module according to the target viewing angle range of the image module in the at least one direction.

[0231] In another embodiment, the offset determination module is used to determine the target viewing angle range of the image module in at least one direction when the robot performs the target task, or the processing result of performing the target task based on the two-dimensional map meets the adjustment condition of the image module, and the at least one direction includes a horizontal direction; according to the target viewing angle range of the image module in at least one direction, determine the target offset information of the image module.

[0232] In another embodiment, a processing device for three-dimensional reconstruction data, such as Fig.12 As shown, including:

[0233] The information acquisition module 1202 is used to obtain the position information of the robot;

[0234] The extraction module 1204 is used to obtain a multi-frame target three-dimensional point cloud within the target viewing angle in the world coordinate system from the three-dimensional reconstruction result with reference to the position information of the robot; the three-dimensional reconstruction result is constructed based on image data collected by an image module of the robot, and the image module rotates in at least one direction and collects image data of the environment at different viewing angles during the rotation process;

[0235] The processing module 1206 is used to perform processing based on the multi-frame target three-dimensional point cloud.

[0236] In another embodiment, the extraction module is used to obtain the current viewing range of the robot; using the robot's position information as a parameter, determine the three-dimensional point cloud of the current viewing range and the three-dimensional point cloud of the viewing range adjacent to the current viewing range from the three-dimensional reconstruction result.

[0237] In another embodiment, a processing module is used to stitch the three-dimensional point cloud of a current viewing range and the three-dimensional point cloud of a viewing range adjacent to the current viewing range to obtain a stitched point cloud; the adjacent viewing range at least includes an adjacent viewing range in a vertical direction; and identify based on the stitched point cloud to determine at least one of the object category, object position and object posture of the object included in the stitched point cloud.

[0238] In another embodiment, a processing module is used to identify at least one of the target cleaning area and obstacle information based on the stitched point cloud; determine the target adjustment angle of the image module based on at least one of the target cleaning area and obstacle information; control the rotation of the image parameters according to the target adjustment angle, and collect image data of the environment at different perspectives during the rotation process.

[0239] In another embodiment, when the processing result of the robot performing a task based on a two-dimensional map meets the adjustment conditions of the image module, the extraction module is used to obtain the current position of the robot; with the current position of the robot as the center, the three-dimensional point cloud around the current position is obtained from the three-dimensional reconstruction result; the surrounding three-dimensional point cloud at least includes the surrounding three-dimensional point cloud in the horizontal direction.

[0240] In another embodiment, a processing module is used to obtain the posture information of the robot; construct a local map based on a three-dimensional point cloud around the current position; and perform path planning, obstacle avoidance, or navigation processing according to the local map and the posture information of the robot.

[0241] Each module in the above-mentioned 3D reconstruction data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0242] In one embodiment, a robot is provided, whose internal structure diagram can be shown as follows: Fig.13 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a threshold recognition method, a door frame orientation recognition method and a robot threshold control method are implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing.

[0243] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0244] In one embodiment, a robot is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the methods of the above embodiments when executing the computer program.

[0245] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods of the above embodiments are implemented.

[0246] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the methods of the above embodiments when executed by a processor.

[0247] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0248] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0249] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing three-dimensional reconstruction data, characterized in that: The method comprises: Controlling the image module of the robot to rotate in at least one direction and collecting image data of the environment from different perspectives during the rotation process; Acquiring actual offset information of the image module; According to the actual offset information, a posture transformation relationship of the image module posture relative to the robot coordinate system is obtained; Obtaining a multi-frame three-dimensional point cloud of the environment at different perspectives in a world coordinate system according to the image data, the posture transformation relationship of the image module posture relative to the robot coordinate system, and the relative relationship between the coordinate systems; The multiple frames of three-dimensional point clouds of the environment in the world coordinate system are fused to obtain a three-dimensional reconstruction result of the environment.

2. The method according to claim 1, characterized in that The image data includes a digital image and a depth image; The method of obtaining a multi-frame three-dimensional point cloud of the environment at different perspectives in the world coordinate system according to the image data, the posture conversion relationship of the image module posture relative to the robot coordinate system, and the relative relationship between the coordinate systems includes: Acquire alignment information of the digital image and the depth image, and fuse the digital image and the depth image based on the alignment information to obtain a multi-frame three-dimensional point cloud of the environment in a module coordinate system; According to the posture conversion relationship of the image module posture relative to the robot coordinate system, the multi-frame three-dimensional point cloud of the environment in the module coordinate system, and the relative relationship between the coordinate systems, a multi-frame three-dimensional point cloud of the environment in the world coordinate system is obtained.

3. The method according to claim 1 or 2, characterized in that: The image module is mounted on a module adjustment device, the module adjustment device has at least one degree of freedom, and the module adjustment device has an adjustment axis for realizing each degree of freedom; The actual offset information includes the rotation angle of the adjustment axis; The step of obtaining the posture transformation relationship of the image module posture relative to the robot coordinate system according to the actual offset information includes: According to the rotation angle of the adjustment axis, a rotation matrix of the image module relative to the robot is obtained; Based on the initial position relationship between the image module and the module adjustment device, and the rotation matrix of the image module relative to the robot, the posture transformation relationship of the image module posture relative to the robot coordinate system is obtained.

4. The method according to claim 3, characterized in that: When the adjustment axis includes an adjustment axis in a single direction, obtaining a rotation matrix of the image module relative to the robot according to the rotation angle of the adjustment axis includes: According to the rotation angle of the adjustment axis in the single direction, the rotation matrix of the adjustment axis in the single direction is obtained; the rotation matrix of the image module relative to the robot includes the rotation matrix of the adjustment axis in the single direction.

5. The method according to claim 3, characterized in that: When the adjustment axis includes adjustment axes in at least two directions, obtaining the rotation matrix of the image module relative to the robot according to the rotation angle of the adjustment axis includes: Obtaining a rotation matrix of the adjustment axis in each direction according to the rotation angle of the adjustment axis in the at least two directions; The rotation matrix of the image module relative to the robot is obtained according to the product of the rotation matrices of the adjustment axes in various directions.

6. The method according to claim 1, characterized in that The method further comprises: When at least one of the robot's operating state, recognition result, task type, and task processing result meets the three-dimensional reconstruction condition, the image module controlling the robot is triggered to rotate in at least one direction.

7. The method according to claim 1, characterized in that The image module of the robot is controlled to rotate in at least one direction, including: Determine target offset information of the image module; The image module of the robot is controlled to rotate according to the target offset information.

8. The method according to claim 7, characterized in that Determine the target offset information of the image module, including: When the recognition result of the robot meets the adjustment condition of the image module, determining the target viewing angle range of the image module in at least one direction, wherein the at least one direction includes a vertical direction; The target offset information of the image module is determined according to the target viewing angle range of the image module in the at least one direction.

9. The method according to claim 7, characterized in that: Determine the target offset information of the image module, including: When the robot performs the target task, or the processing result of performing the target task based on the two-dimensional map satisfies the adjustment condition of the image module, determining the target viewing angle range of the image module in at least one direction, wherein the at least one direction includes a horizontal direction; Target offset information of the image module is determined according to a target viewing angle range of the image module in at least one direction.

10. A method for processing three-dimensional reconstruction data, characterized in that: The method further comprises: Get the robot's location information; Taking the position information of the robot as a reference, obtaining a multi-frame target three-dimensional point cloud within the target viewing angle in the world coordinate system from the three-dimensional reconstruction result; the three-dimensional reconstruction result is constructed based on image data collected by an image module of the robot, and the image module rotates in at least one direction and collects image data of the environment at different viewing angles during the rotation process; Processing is performed based on the multi-frame target three-dimensional point cloud.

11. The method according to claim 10, characterized in that When the recognition result of the robot meets the adjustment condition of the image module, the position information of the robot is used as a reference to obtain a multi-frame target three-dimensional point cloud within the target viewing angle in the world coordinate system from the three-dimensional reconstruction result, including: Get the robot's current viewing range; Taking the position information of the robot as a parameter, the three-dimensional point cloud of the current viewing range and the three-dimensional point cloud of the viewing range adjacent to the current viewing range are determined from the three-dimensional reconstruction result.

12. The method according to claim 11, characterized in that The processing based on the multi-frame target three-dimensional point cloud includes: Splicing the three-dimensional point cloud of the current viewing angle range with the three-dimensional point cloud of the viewing angle range adjacent to the current viewing angle range to obtain a spliced ​​point cloud; the adjacent viewing angle range at least includes the adjacent viewing angle range in the vertical direction; Recognition is performed based on the spliced ​​point cloud to determine at least one of an object category, an object position, and an object posture of an object included in the spliced ​​point cloud.

13. The method according to claim 11, characterized in that When the identified obstacle is not at the center of the field of view, the processing based on the multi-frame target three-dimensional point cloud includes: splicing the three-dimensional point cloud of the current viewing angle range and the three-dimensional point cloud of the viewing angle range adjacent to the current viewing angle range to obtain a spliced ​​point cloud; Identifying at least one of information of a target clearing area and obstacles according to the stitched point cloud; Determining a target adjustment angle of the image module according to at least one of the target clearing area and the obstacle information; The image parameters are controlled to rotate according to the target adjustment angle, and image data of the environment at different viewing angles are collected during the rotation process.

14. The method according to claim 10, characterized in that When the processing result of the robot executing the task based on the two-dimensional map meets the adjustment condition of the image module, the three-dimensional point cloud of multiple frames of the target within the target viewing angle in the world coordinate system is obtained from the three-dimensional reconstruction result with reference to the position information of the robot, including: Get the current position of the robot; Taking the current position of the robot as the center, a three-dimensional point cloud around the current position is obtained from the three-dimensional reconstruction result; the surrounding three-dimensional point cloud at least includes the surrounding three-dimensional point cloud in the horizontal direction.

15. The method according to claim 14, characterized in that The processing based on the multi-frame target three-dimensional point cloud includes: Acquiring posture information of the robot; Build a local map based on the 3D point cloud around the current location; Path planning, obstacle avoidance or navigation is performed based on the local map and the posture information of the robot.

16. A three-dimensional reconstruction data processing device, characterized in that: The device comprises: A rotation control module, used to control the image module of the robot to rotate in at least one direction and collect image data of the environment at different viewing angles during the rotation process; An offset acquisition module, used to acquire actual offset information of the image module; A relationship acquisition module, used to obtain a posture transformation relationship of the image module posture relative to the robot coordinate system according to the actual offset information; A conversion module, used to obtain a multi-frame three-dimensional point cloud of the environment at different perspectives in a world coordinate system according to the image data, the posture conversion relationship of the image module posture relative to the robot coordinate system, and the relative relationship between the coordinate systems; A fusion module is used to fuse the multiple frames of three-dimensional point clouds of the environment in the world coordinate system to obtain a three-dimensional reconstruction result of the environment.

17. A three-dimensional reconstruction data processing device, characterized in that: The device comprises: An information acquisition module is used to obtain the position information of the robot; An extraction module is used to obtain a multi-frame target three-dimensional point cloud within the target viewing angle in the world coordinate system from the three-dimensional reconstruction result with reference to the position information of the robot; the three-dimensional reconstruction result is constructed based on image data collected by an image module of the robot, and the image module rotates in at least one direction and collects image data of the environment at different viewing angles during the rotation process; A processing module is used to perform processing based on the multi-frame target three-dimensional point cloud.

18. A robot comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

Images