Three-dimensional reconstruction data processing method and device and computer equipment

Abstract

The application relates to a three-dimensional reconstruction data processing method, which comprises the following steps: controlling an image module of a robot to rotate in at least one direction and collecting image data of an environment under different visual angles during the rotation; acquiring actual offset information of the image module; obtaining a pose conversion relationship of an image module posture relative to a robot coordinate system according to the actual offset information; obtaining multiple frames of three-dimensional point clouds of the environment under different visual angles in a world coordinate system according to the image data, the pose conversion relationship of the image module posture relative to the robot coordinate system and the relative relationship between the coordinate systems; and fusing the multiple frames of three-dimensional point clouds of the environment in the world coordinate system to obtain a three-dimensional reconstruction result of the environment. The method increases the visual angle range of image collection of the robot, superimposes multiple frames of three-dimensional point clouds under different visual angles to obtain a three-dimensional reconstruction result with a larger field of view angle, higher accuracy and higher detail in the world coordinate system, and improves the three-dimensional reconstruction quality.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a method, apparatus, and computer device for processing three-dimensional reconstruction data. Background Technology

[0002] With the development of robotics technology, robots have brought immense convenience to people's lives. In particular, the development of cleaning robots and AI technology has ushered in a new era of rapid growth for the cleaning robot industry. Cleaning robots are becoming an important component of homes and other entertainment venues, playing an increasingly vital role in cleaning services.

[0003] For robots, 3D reconstruction is primarily used to construct scene information and various maps, enabling the robot to know its pose in the world environment and perceive the location of surrounding obstacles. Through path planning, this allows the robot to achieve localization, navigation, obstacle avoidance, and path planning. Therefore, accurate 3D reconstruction is crucial for improving robot performance. Traditional 3D reconstruction methods suffer from poor reconstruction results. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for processing machine 3D reconstruction data, a robot, a computer-readable storage medium, and a computer program product that can improve reconstruction results and enhance robot performance in response to the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for processing three-dimensional reconstruction data, the method comprising:

[0006] The robot's image module is controlled to rotate in at least one direction, and image data of the environment from different perspectives is acquired during the rotation.

[0007] Obtain the actual offset information of the image module;

[0008] Based on the actual offset information, the pose transformation relationship of the image module relative to the robot coordinate system is obtained;

[0009] Based on the image data, the pose transformation relationship of the image module relative to the robot coordinate system, and the relative relationship between coordinate systems, a multi-frame 3D point cloud of the environment in the world coordinate system is obtained.

[0010] By fusing multiple frames of 3D point clouds of the environment in the world coordinate system, a 3D reconstruction result of the environment is obtained.

[0011] The aforementioned method for processing 3D reconstruction data utilizes an image module that can move in at least one direction. By acquiring image data during its movement, the actual offset information of the image module is obtained, and the pose transformation relationship of the image module relative to the robot coordinate system is determined. Then, using the pose transformation relationship of the image module relative to the robot coordinate system and the relative relationships between coordinate systems, multi-frame 3D point clouds of the environment in the world coordinate system are obtained. Since the image module is movable in at least one direction, the robot's field of view for image acquisition is increased, resulting in multi-frame 3D point clouds of the environment from different perspectives in the world coordinate system. These multi-frame 3D point clouds from different perspectives are then superimposed and matched into the world coordinate system, resulting in a larger field of view, more accurate, and more detailed 3D reconstruction results in the world coordinate system, thus improving the quality of 3D reconstruction.

[0012] Secondly, this application provides a method for processing three-dimensional reconstruction data, the method comprising:

[0013] Obtain the robot's location information;

[0014] Using the robot's position information as a reference, multiple frames of target 3D point cloud within the target's viewpoint range in the world coordinate system are obtained from the 3D reconstruction results; the 3D reconstruction results are constructed based on image data collected by the robot's image module, which rotates in at least one direction and collects image data of the environment from different viewpoints during the rotation.

[0015] Processing is performed based on the multi-frame target 3D point cloud.

[0016] The above-mentioned method for processing 3D reconstruction data obtains multi-frame target 3D point clouds of the environment within the target range in the world coordinate system from the 3D reconstruction results by using the robot's position information as a reference. Since the 3D reconstruction results include 3D point clouds of the environment acquired from different viewpoints by an image module with an adjustable angle range, the multi-frame target 3D point clouds extracted from the 3D reconstruction results have multiple viewpoints. Processing based on the multi-frame target 3D point clouds can utilize the rich 3D point clouds from multiple viewpoints to improve robot performance.

[0017] Thirdly, this application provides a robot including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the methods described in the above embodiments.

[0018] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the methods described in the above embodiments.

[0019] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in the above embodiments. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the robot's structure in one embodiment;

[0021] Figure 2 This is a schematic diagram of the signal flow between the sensing and control parts of a robot in one embodiment;

[0022] Figure 3 This is a flowchart illustrating a method for processing 3D reconstruction data in one embodiment;

[0023] Figure 4 This is a schematic diagram illustrating the relationship between the coordinate systems in one embodiment;

[0024] Figure 5 This is a flowchart illustrating the steps of obtaining multi-frame 3D point clouds of the environment from different perspectives in the world coordinate system based on the image data, the pose transformation relationship of the image module relative to the robot coordinate system, and the relative relationship between coordinate systems.

[0025] Figure 6 This is a flowchart illustrating the steps of obtaining the pose transformation relationship of the image module relative to the robot coordinate system based on the actual offset information in one embodiment.

[0026] Figure 7 This is a schematic diagram of an adjustment shaft according to one embodiment;

[0027] Figure 8 This is a flowchart illustrating a method for processing 3D reconstruction data in another embodiment;

[0028] Figure 9 This is a flowchart illustrating a method for processing 3D reconstruction data in one embodiment.

[0029] Figure 10 This is a flowchart illustrating a method for processing 3D reconstruction data in yet another embodiment.

[0030] Figure 11 This is a structural block diagram of a three-dimensional reconstruction data processing device in one embodiment;

[0031] Figure 12 This is a structural block diagram of the three-dimensional reconstruction data processing device in another embodiment;

[0032] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] This application provides a robot, such as... Figure 1 As shown, the system includes a robot body 101, a module adjustment device 102, and an image module 103. The module adjustment device 102 is mounted on the robot body 101, and the image module 103 is mounted on the module adjustment device 102.

[0035] Specifically, the image module 103 may include a digital image acquisition device and a depth image acquisition device.

[0036] Specifically, the module adjustment device 102 is a gimbal structure including at least one axis, which can drive the image module 103 to rotate in at least one direction.

[0037] The module adjustment device 102 can be a single-axis gimbal structure, including any one of the flight axis, pitch axis or roll axis, which can control the image module 102 to move horizontally or vertically relative to the robot body 101, or control the image module 102 to rotate left and right relative to the robot body 101.

[0038] The module adjustment device 102 can also be a dual-axis gimbal structure. In one embodiment, the module adjustment device 102 includes a yaw axis and a pitch axis, capable of controlling the image module 102 to move relative to the robot body 101 in the horizontal and / or vertical directions. In one embodiment, the module adjustment device 102 includes a pitch axis and a roll axis, capable of controlling the image module 102 to move relative to the robot body 101 in the vertical direction, and / or controlling the image module 102 to rotate relative to the robot body 101. In one embodiment, the module adjustment device 102 includes a yaw axis and a roll axis, capable of controlling the image module 102 to move relative to the robot body 101 in the horizontal direction, and / or controlling the image module 102 to rotate relative to the robot body 101.

[0039] The module adjustment device 102 can also be a three-axis gimbal structure. In one embodiment, the module adjustment device 102 includes a flight axis (controlling the horizontal direction), a pitch axis (controlling the vertical direction), and a roll axis (controlling rotation), which can control the image module 102 to move in the vertical and / or horizontal direction relative to the robot body 101, and / or control the image module 102 to rotate relative to the robot body 101.

[0040] like Figure 2 As shown, the robot's sensing and control components include:

[0041] The system comprises a controller 201, an actuator 202, and multiple sensors 203. These sensors may include an image module, an IMU (Inertial Measurement Unit), LiDAR, infrared sensors, collision sensors, ultrasonic sensors, wall-following sensors, cliff sensors, and fall protection sensors. Each sensor in the multiple sensors 203, as well as the actuator 202, is signal-connected to the controller 201. The controller 201 processes the sensor data collected by the multiple sensors, makes decisions, generates control commands based on the decision results, and sends these commands to the actuator 202. The actuator 202 may include drivers for the robot's drive wheels, drivers for the module adjustment device, etc. Decision-making may include navigation, obstacle avoidance, and control of various robot components.

[0042] Here, "robot" refers to a self-moving device equipped with sensors that can autonomously move and perform tasks. In one embodiment, robots include, but are not limited to, cleaning robots, food delivery robots, and logistics robots. Cleaning robots include robots used for cleaning homes and non-home environments, such as sweeping robots, mopping robots, automatic floor scrubbers, and various cleaning robots used in entertainment venues (e.g., pool cleaning machines).

[0043] In one embodiment, such as Figure 3 As shown, a method for processing three-dimensional reconstruction data is provided, which can be applied to... Figure 2 Taking the controller in the example, the following steps are included:

[0044] Step 302: Control the robot's image module to rotate in at least one direction, and collect image data of the environment from different perspectives during the rotation.

[0045] An image module refers to sensor devices and combinations used to acquire image data. Image data may include digital images and depth images. In one embodiment, the image module includes a digital image acquisition device and a depth image acquisition device. The image acquisition device is not limited to conventional cameras and includes various sensors that acquire data for semantic recognition / navigation and obstacle avoidance. The depth image acquisition device is not limited to various sensors that can directly or indirectly acquire depth images, such as TOF / 3D TOF / binocular cameras. The image module may also include an RGBD camera. Functionally, an RGBD camera adds depth measurement functionality to the functions of a regular RGB image acquisition device, enabling it to acquire both digital and depth images.

[0046] In this module, the field of view (FOP) is fixed. The FOP refers to the angle between the outer contour of the sensor within the module and the center of the module, and is a range of angles. The FOP includes the horizontal FOP and the vertical FOP, analogous to the range of visual angles that the human eye can see when the eyeballs are stationary.

[0047] The image module in this embodiment can rotate in at least one direction via a module angle adjustment device. This at least one direction can include at least one of the horizontal, vertical, and depth directions. Compared to a fixed image module, the image module in this embodiment can rotate in at least one direction while maintaining a constant field of view; only the orientation of the image module changes, thus altering its viewing angle. For example, if the image module is movable in the vertical direction, such as moving upwards, the viewing angle in the height direction can be adjusted compared to a fixed image module, allowing for a higher or lower viewing angle. Similarly, if the image module is movable in the horizontal direction, the viewing angle in the horizontal direction can be increased compared to a fixed image module, allowing for a leftward or rightward viewing angle. Furthermore, if the image module can rotate 360 ​​degrees horizontally, controlling its clockwise or counterclockwise rotation, the circumferential angle of the image module can be increased compared to a fixed orientation, thus increasing the circumferential viewing angle.

[0048] For example, in an environment with a sofa, a fixed image module requires controlling the robot to move in order to see the sofa from different angles. However, in this embodiment, the image module is movable; by controlling the robot's module adjustment device, different parts of the sofa can be seen from above, below, left, or right, thus obtaining the full view of the sofa.

[0049] Compared to application scenarios where the image module has a fixed angle, the image module in this embodiment rotates in at least one direction and acquires multiple frames of digital images and depth images during the rotation. Therefore, the viewing angle range that the image module can acquire in the corresponding direction is increased, and stitching images from different perspectives in that direction can yield a larger viewing angle range in that direction.

[0050] For example, the image module can move in at least one of the horizontal and vertical directions, acquiring multiple frames of digital and depth images during rotation. Because the image module can move vertically, the vertical field of view it can acquire varies; for example, a larger vertical field of view can be obtained by stitching together images with expanded vertical field of view. Similarly, because the image module can rotate horizontally, the horizontal field of view it can acquire varies; for example, a larger horizontal field of view can be obtained by stitching together images with expanded horizontal field of view without moving the robot. Furthermore, because the image module can rotate 360 ​​degrees horizontally, the rotational field of view it can acquire varies; for example, a larger circumferential field of view can be obtained by stitching together images with expanded rotational field of view without moving the robot.

[0051] As mentioned earlier, the module adjustment device 102 can be a single-axis gimbal structure, a dual-axis gimbal structure, or a three-axis gimbal structure, capable of rotation in the direction corresponding to each axis. The module angle adjustment device employs a two-axis gimbal structure, including a yaw axis and a pitch axis. The yaw axis adjusts the horizontal angle, and the pitch axis adjusts the vertical angle. Specifically, the yaw axis of the module angle adjustment device is used to adjust the horizontal direction of the module image; during rotation, the information collected by the module can be stitched together to obtain an image or its format data with an expanded horizontal field of view. The pitch axis of the module angle adjustment device is used to adjust the vertical direction of the module; during rotation, the information collected by the module can be stitched together to obtain an image or its format data with an expanded vertical field of view. Finally, the roll axis of the module angle adjustment device is used to adjust the rotation angle of the module image; during rotation, the information collected by the module can be stitched together to obtain an expanded circumferential field of view image or its format data.

[0052] Specifically, in response to a 3D reconstruction command, the robot's image module can be controlled to rotate in at least one direction to perform 3D reconstruction. The 3D reconstruction command can be set to be triggered based on specific conditions, triggered in real-time by the robot, or triggered in response to user settings.

[0053] In one embodiment, in response to a 3D reconstruction command, the controller controls the rotation of at least one adjustment axis of the robot module's angle adjustment device, such as controlling the rotation of at least one of the yaw, pitch, and roll axes, thereby causing the image module to rotate horizontally or vertically, or to rotate the image module itself. During this rotation, the image module acquires multiple frames of digital images and depth images of the environment.

[0054] Digital images (RGB) can be acquired through an image acquisition device (such as a camera) in the image module, and depth images (D) can be acquired through a depth sensor. Alternatively, digital images and depth images can be acquired through an RGBD camera.

[0055] In traditional methods, the position and angle of a robot's image module are fixed. Once designed and manufactured, the angles at which information is acquired in the horizontal and vertical directions are fixed. The vertical viewing angle range that the image module can acquire is fixed, while the horizontal viewing angle range needs to be adjusted by moving the robot. In this embodiment, the image module is movable, thus expanding the image module's acquisition viewing angle range.

[0056] Step 304: Obtain the actual offset information of the image module.

[0057] The actual offset information of the image module represents the degree of offset of the rotated image module relative to the initial state of the image module.

[0058] To facilitate algorithm calibration and attitude calculation, the image module is mounted on a module adjustment device. The center of the image module can be aligned with the pitch axis center of the module adjustment device, thus aligning the origins of the image module coordinate system and the pitch axis coordinate system of the module adjustment device. Alternatively, the center of the image module can be aligned with the yaw axis center of the module adjustment device, thus aligning the origins of the image module coordinate system and the yaw coordinate system of the module adjustment device. In one embodiment, the relationship between the coordinate systems is as follows: Figure 4 As shown.

[0059] The module angle adjustment device is controlled by a motor, and each adjustment axis is equipped with a encoder. The rotation angle of each adjustment axis can be obtained through the encoder, thereby obtaining the actual offset information of the image module. Taking a module adjustment device that includes a yaw axis and a pitch axis as an example, if the module adjustment device only rotates on the yaw axis, the actual offset information of the image module can be determined based on the rotation angle on the yaw axis. If the module adjustment device only rotates on the pitch axis, the actual offset information of the image module can be determined based on the rotation angle on the pitch axis. If the module adjustment device rotates on both the yaw and pitch axes, the actual offset information of the image module can be determined based on the rotation angles on both the pitch and yaw axes.

[0060] Step 306: Based on the actual offset information, obtain the pose transformation relationship of the image module posture relative to the robot coordinate system.

[0061] A robot coordinate system is a coordinate system constructed with a point on the robot as the origin. In one implementation, the robot coordinate system can be constructed with the robot's center as the origin.

[0062] The process involves pre-calibrating to obtain the positional relationship between the image module and the robot body. Then, based on the actual offset information and the positional relationship between the image module and the robot body, the pose transformation relationship of the image module relative to the robot coordinate system is obtained. Based on this pose transformation relationship, the coordinate data in the image module coordinate system acquired during rotation is transformed into coordinate data in the robot coordinate system. For example, for a 3D point cloud corresponding to a pixel in a digital image during rotation, the coordinates are transformed to the image module coordinate system according to the transformation relationship between the image coordinate system and the image module coordinate system. Then, based on the pose transformation relationship between the image module pose and the robot coordinate system, the coordinates are transformed back to the robot coordinate system, resulting in the coordinate information of the 3D point cloud corresponding to a pixel in the digital image acquired during rotation in the robot coordinate system.

[0063] Step 308: Based on the image data, the pose transformation relationship of the image module relative to the robot coordinate system, and the relative relationship between coordinate systems, obtain multi-frame 3D point clouds of the environment from different perspectives in the world coordinate system.

[0064] To obtain the correspondence between pixels in a digital image and pixels in a depth image, the digital and depth images can be aligned. Specifically, alignment refers to determining the correspondence between pixels in the digital and depth images. Based on this correspondence, the digital and depth images are fused, resulting in multi-frame 3D point cloud information in the module coordinate system.

[0065] Specifically, for each point cloud in the multi-frame 3D point cloud of the module coordinate system, according to the pose transformation relationship of the image module relative to the robot coordinate system and the relative relationship between the image module coordinate system and the robot coordinate system, the 3D coordinate information of each point cloud in the image module coordinate system is transformed to the robot coordinate system. Then, based on the transformation relationship between the robot coordinate system and the world coordinate system obtained by the predetermined calibration, the coordinate coefficients in the robot coordinate system are transformed to the world coordinate system, so as to obtain the 3D coordinate information of the multi-frame 3D point cloud information in the module coordinate system in the robot coordinate system.

[0066] Step 310: Fuse the multi-frame 3D point cloud of the environment in the world coordinate system to obtain the 3D reconstruction result of the environment.

[0067] Specifically, feature matching and registration are performed on multiple frames of 3D point clouds in the world coordinate system. Since the robot's image module is rotatable in at least one direction, its viewing angle can be expanded in the horizontal, and / or vertical, and / or circumferential directions. This allows for the acquisition of digital and depth images with a wider viewing angle in the horizontal, and / or vertical, and / or circumferential directions, resulting in a greater quantity of 3D point cloud data from different perspectives. Therefore, during the 3D point cloud registration process, feature configuration and registration yield 3D point cloud data of the same object from different perspectives. These multiple frames of 3D point clouds from different times and perspectives are then superimposed and matched onto the world coordinate system, resulting in a larger-view, more accurate, and more detailed 3D map in the world coordinate system.

[0068] The aforementioned method for processing 3D reconstruction data utilizes an image module that can rotate in at least one direction. By acquiring image data during rotation, the actual offset information of the image module is obtained, and the pose transformation relationship of the image module relative to the robot coordinate system is determined. Then, using the pose transformation relationship of the image module relative to the robot coordinate system and the relative relationships between coordinate systems, multi-frame 3D point clouds of the environment in the world coordinate system are obtained. Since the image module can rotate in at least one direction, the viewing angle range of the robot's image acquisition is increased, resulting in multi-frame 3D point clouds of the environment from different perspectives in the world coordinate system. These multi-frame 3D point clouds from different perspectives are then superimposed and matched into the world coordinate system, resulting in a larger field of view, more accurate, and more detailed 3D reconstruction results in the world coordinate system, thus improving the quality of 3D reconstruction.

[0069] In another embodiment, the image data includes digital images and depth images. For example... Figure 5 As shown, based on the image data, the pose transformation relationship of the image module relative to the robot coordinate system, and the relative relationships between coordinate systems, multi-frame 3D point clouds of the environment from different perspectives in the world coordinate system are obtained, including:

[0070] Step 502: Obtain the alignment information of the digital image and the depth image, and fuse the digital image and the depth image based on the alignment information to obtain a multi-frame 3D point cloud of the environment in the module coordinate system.

[0071] In this context, a digital image is represented as a two-dimensional array, where the digital units are pixels; the grayscale value of each pixel is stored in a corresponding two-dimensional matrix. A digital image can be a grayscale image or a color image. A color image can be an RGB image.

[0072] A depth image, also called a range image, is an image that uses the distance (depth) from the image acquisition device to each point in the scene as pixel values.

[0073] For schemes that acquire digital images and depth images using image acquisition devices and depth acquisition devices respectively, the spatial coordinate systems of the digital images and depth images are different. The coordinate system of the digital images is established with the origin of the image acquisition device as the center, while the coordinate system of the depth images is established with the origin of the depth acquisition device as the center. Therefore, the digital images and depth images are independent of each other.

[0074] To obtain the correspondence between pixels in a digital image and pixels in a depth image, the digital image and the depth image can be aligned. Alignment specifically refers to determining the correspondence between pixels in the digital image and pixels in the depth image. The alignment can be achieved using classic algorithms or pre-trained models.

[0075] One alignment method is to obtain the intrinsic parameter matrix of the image acquisition device, the intrinsic parameter matrix of the depth acquisition device, and the extrinsic rotation and offset matrices of the image acquisition device and the depth acquisition device, traverse the coordinates of each pixel in the digital image, and calculate the coordinates of the corresponding pixel in the digital image using the alignment formula, thereby obtaining the correspondence between the pixel in the digital image and the pixel in the depth image.

[0076] Based on this, the digital image and the depth image are fused according to the correspondence between pixels in the digital image and pixels in the depth image to obtain a three-dimensional point cloud.

[0077] Correspondingly, based on the correspondence between pixels in the digital image and pixels in the depth image, the depth value of each pixel in the digital image (i.e., the distance from a point in the real scene corresponding to that pixel to the camera) can be obtained. Then, each pixel in the digital image can be transformed into the module coordinate system to obtain multi-frame 3D point cloud information in the module coordinate system.

[0078] Step 504: Based on the pose transformation relationship of the image module posture relative to the robot coordinate system, the multi-frame 3D point cloud of the environment in the module coordinate system, and the relative relationship between coordinate systems, obtain the multi-frame 3D point cloud of the environment in the world coordinate system.

[0079] Specifically, for each point cloud in the multi-frame 3D point cloud of the module coordinate system, according to the pose transformation relationship of the image module relative to the robot coordinate system, the 3D coordinate information of each point cloud in the image module coordinate system is transformed to the robot coordinate system. Then, based on the predetermined transformation relationship between the robot coordinate system and the world coordinate system, the coordinate coefficients in the robot coordinate system are transformed to the world coordinate system, so as to obtain the 3D coordinate information of the multi-frame 3D point cloud information in the module coordinate system in the robot coordinate system.

[0080] Furthermore, feature matching and registration are performed on multiple frames of 3D point clouds in the world coordinate system. Since the robot's image module can rotate in at least one direction, the field of view of the image module is expanded, thereby enabling the acquisition of image data with a larger field of view, such as digital images and depth images with a larger horizontal and / or vertical field of view, resulting in a greater number of 3D point cloud data from different perspectives. During the 3D point cloud registration process, feature configuration and registration are used to obtain 3D point cloud data of the same object from different perspectives. Then, multiple frames of 3D point clouds from different times and angles are superimposed and matched into the world coordinate system, resulting in a larger, more accurate, and more detailed 3D reconstruction result in the world coordinate system.

[0081] In another embodiment, fusing the multi-frame 3D point cloud of the environment in the world coordinate system to obtain the 3D reconstruction result of the environment may include: preprocessing the multi-frame 3D point cloud information in the module coordinate system, inputting the preprocessed multi-frame 3D point cloud of the environment in the world coordinate system into a pre-trained fusion model, and outputting the 3D reconstruction result from the fusion model to obtain a 3D map of the environment in the world coordinate system.

[0082] Specifically, preprocessing may include processing multi-frame 3D point cloud information in the module coordinate system into a format acceptable to the fusion model, such as a point cloud data format.

[0083] The fusion model is pre-trained based on multiple frames of 3D point clouds collected in the world coordinate system and the corresponding 3D reconstruction results. The training process of the fusion model includes:

[0084] In the data preparation phase, multi-frame 3D point cloud information and corresponding 3D reconstruction results in the world coordinate system are collected and divided into training set and test set.

[0085] In the model building phase, a model is selected, and the corresponding network structure is constructed based on the selected model. Models such as PointNet and PointCNN can be used. Different models have slightly different performance and application scenarios, so it is necessary to select the appropriate model based on the actual situation.

[0086] During the model training phase, the model is trained using a training set, and the model parameters are adjusted so that the model can accurately predict the 3D reconstruction results.

[0087] Model evaluation involves using a test set to evaluate the model and calculating metrics such as accuracy and recall.

[0088] Model optimization involves optimizing the model based on the evaluation results, such as adjusting the network structure or modifying the loss function.

[0089] In this embodiment, a neural network model is used to fuse multiple frames of 3D point clouds of the environment in the world coordinate system to obtain the 3D reconstruction result of the environment.

[0090] In another embodiment, feature detection and feature registration can also be used to reconstruct a 3D map of the environment in the world coordinate system based on the pose transformation relationship of the image module posture relative to the robot coordinate system and the multi-frame 3D point cloud of the environment in the module coordinate system.

[0091] In another embodiment, the image module is mounted on a module adjustment device having at least one degree of freedom and an adjustment axis for each degree of freedom; the actual offset information includes the rotation angle of the adjustment axis.

[0092] The pose transformation relationship of the image module relative to the robot coordinate system is obtained based on the actual offset information, such as... Figure 6 As shown, it includes:

[0093] Step 602: Based on the rotation angle of the adjustment axis, obtain the rotation matrix of the image module relative to the robot.

[0094] Specifically, the rotation angle of the adjustment shaft of the module adjustment device can be controlled by a motor, and the rotation angle of the adjustment shaft can be obtained through an encoder. The encoder angle and the rotation angle have a linear relationship. Based on the encoder angle and this linear relationship, the rotation angle of the adjustment shaft is obtained. Then, the rotation matrix of the image module relative to the robot is derived from the rotation angle of the adjustment shaft.

[0095] Step 604: Based on the initial positional relationship between the image module and the module adjustment device, and the rotation matrix of the image module relative to the robot, obtain the pose transformation relationship of the image module posture relative to the robot coordinate system.

[0096] The initial positional relationship between the image module and the module adjustment device is related to the installation position. To facilitate algorithm calibration and attitude calculation, the image module is mounted on the module adjustment device. The center of the image module can be aligned with the pitch axis center of the module adjustment device, thus aligning the origins of the image module coordinate system and the pitch axis coordinate system of the module adjustment device. Alternatively, the center of the image module can be aligned with the yaw axis center of the module adjustment device, thus aligning the origins of the image module coordinate system and the yaw coordinate system of the module adjustment device.

[0097] After installation, the initial position of the image module relative to the robot coordinate system is R_0. Since there is no rotation in the initial state, the initial position relationship between the image module and the module adjustment device can be considered as having only one translation vector T.

[0098] Therefore, the pose transformation relationship of the image module relative to the robot coordinate system can be obtained based on the rotation matrix R and the translation vector T.

[0099]

[0100] Where (x_{robot},y_{robot},z_{robot}) are the coordinates in the robot coordinate system, (x_{mod},y_{mod},z_{mod}) are the coordinates in the module coordinate system, (t_x,t_y,t_z) are the translation vector T, T is a known initial value, and R is the real-time rotation matrix.

[0101] In this embodiment, the image module is mounted on a module adjustment device. This device has an adjustment axis that realizes degrees of freedom. Utilizing the ease of detection of the rotation angle of the adjustment axis, the actual offset information of the image module is obtained. Then, using the rotation angle of the adjustment axis, the rotation matrix of the image module relative to the robot is obtained. Finally, based on the initial positional relationship between the image module and the module adjustment device, and the rotation matrix, the pose transformation relationship of the image module relative to the robot coordinate system is obtained. This method simplifies the calculation of the rotation matrix and improves the computational efficiency of the pose transformation relationship of the image module relative to the robot coordinate system.

[0102] In one embodiment, the adjustment shaft includes an adjustment shaft in a single direction. For example... Figure 7 As shown, the module adjustment device is adjustable in the X-axis, Y-axis or Z-axis direction, that is, the module adjustment device can be set with adjustment axes on the X-axis, Y-axis or Z-axis, and the corresponding adjustment axes are the yaw axis, pitch axis and roll axis respectively.

[0103] The step of obtaining the rotation matrix of the image module relative to the robot based on the rotation angle of the adjustment axis includes: obtaining the rotation matrix of the adjustment axis in the single direction based on the rotation angle of the adjustment axis in the single direction; the rotation matrix of the image module relative to the robot includes the rotation matrix of the adjustment axis in the single direction.

[0104] Specifically, the rotation matrix of the adjustment axis in a single direction is obtained based on the rotation angle of the adjustment axis in that direction. Taking the line axis of the adjustment axis in a single direction as an example, which can be adjusted in the horizontal direction, assuming the rotation angle is α, the rotation matrix in that direction is:

[0105]

[0106] Taking a single-direction adjustment axis as the pitch axis as an example, it can adjust the vertical direction. Assuming the rotation angle is β, the rotation matrix in this direction is:

[0107]

[0108] Taking a single-direction adjustment axis as the roll axis as an example, it can adjust the rotation direction. Assuming the rotation angle is γ, the rotation matrix in that direction is:

[0109]

[0110] For a module adjustment device that includes only a single-direction adjustment axis, the rotation matrix of the image module relative to the robot is the rotation matrix of the adjustment axis in that single direction.

[0111] In this embodiment, the module adjustment device can be used in a single direction to allow the image module to rotate in a single direction. Then, based on the rotation angle of the rotation axis, the pose transformation of the image module relative to the robot coordinate system is calculated. This method simplifies the calculation of the rotation matrix and improves the calculation efficiency of the pose transformation relationship of the image module relative to the robot coordinate system.

[0112] In another embodiment, the adjustment shaft includes adjustment shafts in at least two directions. For example... Figure 7 As shown, the module adjustment device is adjustable in at least two of the X-axis, Y-axis or Z-axis directions, that is, the module adjustment device can be provided with adjustment axes in at least two of the X-axis, Y-axis or Z-axis directions.

[0113] Obtaining the rotation matrix of the image module relative to the robot based on the rotation angle of the adjustment axis includes: obtaining the rotation matrix of the adjustment axis in each direction based on the rotation angle of the adjustment axis in at least two directions; and obtaining the rotation matrix of the image module relative to the robot by multiplying the rotation matrices of the adjustment axis in each direction.

[0114] Specifically, first, based on the rotation angle of the adjustment axis in each direction, the rotation matrix of that adjustment axis is obtained. For example, if the module adjustment device includes a yaw axis and a pitch axis, assuming the rotation angle of the yaw axis is α, then the rotation matrix in that direction is R. x (α), where the pitch axis rotation angle is β, then the rotation matrix in that direction is R. y (β), then the rotation matrix of the image module relative to the robot is the product of the two, which is:

[0115] R = R x (α)R y (β)

[0116] For example, if the module adjustment device includes a yaw axis and a roll axis, and assuming the rotation angle of the yaw axis is α, then the rotation matrix in that direction is R. x (α), where the rotation angle of the roll axis is γ, then the rotation matrix in that direction is R. z(γ), the rotation matrix of the image module relative to the robot is the product of the two, and is:

[0117] R = R x (α)R z (γ)

[0118] For example, if the module adjustment device includes a pitch axis and a roll axis, assuming the rotation angle of the pitch axis is β, then the rotation matrix in that direction is R. y (β), where the rotation angle of the roll axis is γ, then the rotation matrix in that direction is R. z (γ), the rotation matrix of the image module relative to the robot is the product of the two, and is:

[0119] R = R y (β)R z (γ)

[0120] For example, if the module adjustment device includes a yaw axis, a pitch axis, and a roll axis, and assuming the rotation angle of the yaw axis is α, then the rotation matrix in that direction is R. x (α), where the pitch axis rotation angle is β, then the rotation matrix in that direction is R. y (β), where the rotation angle of the roll axis is γ, then the rotation matrix in that direction is R. z (γ), the rotation matrix of the image module relative to the robot is the product of the two, and is:

[0121] R = R x (α)R y (β)R z (γ)

[0122] In this embodiment, by setting adjustment axes in at least two directions of the module adjustment device, the image module is adjustable in at least two directions. Based on the rotation angles in each direction, the rotation matrix of the image module relative to the robot can be calculated. Taking an adjustment axis including a horizontal yaw axis and a vertical pitch axis as an example, the rotation matrix of the adjustment axis in each direction is obtained based on the rotation angles of the adjustment axes in the at least two directions, including: obtaining the rotation matrix of the yaw axis based on the rotation angle of the yaw axis; and obtaining the rotation matrix of the pitch axis based on the rotation angle of the pitch axis.

[0123] Specifically, assuming the angles of the two motor encoders are theta_1 and theta_2, based on the linear relationship between the encoder angles and the rotation angles, the rotation angles of the pitch axis and yaw axis can be obtained as α and β, respectively. Then, the real-time rotation matrix R can be expressed as:

[0124] R = R x (α)R y (β)

[0125]

[0126]

[0127] In this embodiment, by setting the pitch axis and yaw axis of the module adjustment device, the image module is adjustable in both the vertical and horizontal directions. Based on the rotation angles of the pitch axis and yaw axis, the rotation matrix of the image module relative to the robot can be calculated.

[0128] In one embodiment, the robot can trigger a 3D reconstruction command in real time to construct a 3D reconstruction of the environment. The drawback of this method is that it consumes the computing power required for the robot to operate.

[0129] In another embodiment, a 3D reconstruction command can be triggered when at least one of the robot's operating state, recognition result, task type, and task processing result meets the 3D reconstruction conditions.

[0130] The conditions for 3D map reconstruction can include: the robot being in an idle state. That is, the 3D reconstruction command is triggered when the robot is in an idle state. This method utilizes the robot's idle time to trigger 3D map reconstruction.

[0131] The conditions for 3D map reconstruction can also include: the robot's recognition results include the robot recognizing a specific object, the robot not recognizing an object, the robot not recognizing an object, and triggering 3D reconstruction when the robot recognizes a specific object, cannot recognize an object, or cannot recognize an object. This method can trigger 3D reconstruction when the recognition effect is poor, resulting in a larger, more accurate, and more detailed 3D reconstruction result in the world coordinate system. This improves the quality of image recognition.

[0132] In one embodiment, 3D reconstruction is triggered when the task type is an obstacle avoidance task, a navigation task, or a path planning task. This allows for the use of a larger perspective and more accurate and detailed 3D reconstruction results to improve obstacle avoidance, navigation, and path planning performance.

[0133] In one embodiment, the identification result is that a specific type of obstacle has been detected, or the identification result is that the robot has touched an obstacle and the robot is trapped, at least one of these.

[0134] When the robot detects a specific type of obstacle, it triggers a 3D reconstruction command to rebuild a 3D map of the environment in the world coordinate system, facilitating better obstacle avoidance planning. Specific types of obstacles may include fragile obstacles, doorway obstacles, and fecal obstacles.

[0135] When the robot detects that it has touched an obstacle, a 3D reconstruction command is triggered to reconstruct a 3D map of the environment in the world coordinate system in order to identify the obstacle and formulate an obstacle avoidance path.

[0136] When the robot is detected to be trapped, a 3D reconstruction command is triggered.

[0137] In this embodiment, by setting triggering conditions, idle resources can be utilized, or 3D reconstruction can be triggered when needed, thus avoiding the impact of 3D reconstruction occupying resources on the robot's task execution.

[0138] In one embodiment, controlling the robot's image module to rotate in at least one direction includes: determining target offset information of the image module; and controlling the robot's image module to rotate according to the target offset information.

[0139] Specifically, when 3D reconstruction needs to be triggered, the target offset information of the image module can be determined based on at least one of the object information in the environment and the task type currently being performed by the robot. Then, the robot's image module is rotated according to the target offset information to acquire an image from the target's perspective.

[0140] In one embodiment, when the robot performs an object recognition task, the target offset information of the image module can be determined based on the object information. For example, if the entire object is not captured, the target offset information of the image module can be determined based on the object's position to increase the viewing angle of the image module and enable it to capture the entire object.

[0141] In one embodiment, when the robot performs an object recognition task, the semantic recognition function requires the sensor to capture as much overall information about the object as possible to improve the accuracy of the recognition. Generally, a large vertical viewing angle is required. In this case, the vertical viewing angle of the image module can be increased to capture the entire object.

[0142] In one embodiment, when the robot performs a path planning or obstacle avoidance task, the obstacle avoidance function requires the sensors to capture as much information as possible about the robot's surroundings. Generally, the vertical field of view needs to be from the ground to an angle slightly above the robot's body, but the horizontal field of view should be as large as possible. This allows the robot to acquire more surrounding information without rotating its own angle, thus improving obstacle avoidance capabilities. In this case, the horizontal field of view of the image module can be increased to obtain sufficiently rich information about the robot's surroundings.

[0143] In this embodiment, when 3D reconstruction is triggered, the robot's image module is controlled to rotate according to the target offset information. This allows the viewpoint of the image module to be adjusted in real time and flexibly in combination with the task and environmental conditions, thereby acquiring images from the target viewpoint during the 3D reconstruction process.

[0144] In another embodiment, determining the target offset information of the image module includes: determining the viewing angle range of the image module in the vertical direction when the robot's recognition result meets the adjustment conditions of the image module; and determining the target offset information of the image module based on the viewing angle range in the vertical direction.

[0145] Among these tasks, the robot's recognition task is a semantic recognition process. Semantic recognition requires sensors to capture as much overall information about the object as possible to improve accuracy. Generally, a wide vertical field of view is needed; for example, seeing the entire body of a pet increases the probability of identifying its species compared to seeing only its feet.

[0146] Based on this, when the robot recognizes a specific type of object, or when object recognition fails, the viewing angle range of the image module in at least one direction is determined, and the at least one direction includes the vertical direction.

[0147] In one implementation, only the vertical viewing angle range may be determined.

[0148] In one implementation, the horizontal and vertical viewing angle ranges can be determined based on the characteristics of semantic recognition and the information of the object. For example, while expanding the vertical viewing angle range, the corresponding horizontal viewing angle range can be expanded based on the object's position information and missing object information, thereby determining the target viewing angle range of the image module in the horizontal and vertical directions.

[0149] For example, when encountering pet feces, to avoid touching and spreading it everywhere, the field of view needs to be expanded to determine the location of the feces. Similarly, when furniture and shoes are detected, the field of view can be expanded to determine their location for app display and obstacle avoidance during cleaning. When fragile items are detected, the field of view can be expanded to determine their location, allowing for adjustments to the machine's safe distance and speed. When doors or other objects are detected, the field of view can be expanded to determine whether the door is closed based on its type and whether it is currently enclosed. Finally, when thresholds are detected, their location can be determined to inform the machine's action strategy, assisting it in navigating while minimizing the possibility of getting stuck.

[0150] In this embodiment, based on the requirements of robot semantics, when 3D reconstruction is required, the robot is actively controlled to adjust the vertical viewing angle range, which is consistent with the requirement of semantic recognition to capture the overall information of the object to improve the accuracy of recognition. Therefore, the accuracy of semantic recognition can be improved.

[0151] In another embodiment, determining the target offset information of the image module includes: when the robot performs a target task, or when the processing result of performing a target task based on a 2D map meets the adjustment conditions of the image module, determining the viewing angle range of the image module in at least one direction, wherein the at least one direction includes the horizontal direction; and determining the target offset information of the image module based on the viewing angle range of the image module in at least one direction. The robot has path planning and obstacle avoidance requirements. Path planning and obstacle avoidance require the sensors to capture as much information as possible about the robot's surroundings. Generally, the vertical field of view needs to be from the ground to an angle slightly above the robot's body, but the horizontal field of view should be as large as possible. This allows the robot to acquire more surrounding information without rotating its own angle, thus improving obstacle avoidance capabilities.

[0152] Specifically, the adjustment conditions of the image module may include, when the processing result of the task based on the two-dimensional map is a task failure, the target task may include an obstacle avoidance task, a navigation task, or a path planning task.

[0153] Therefore, when the robot needs to perform obstacle avoidance, path planning, or navigation tasks, or when the processing results of the robot's obstacle avoidance, path planning, or navigation tasks based on a 2D map meet the module's adjustment conditions, the viewing angle range of the image module in at least one direction is determined. The robot's path planning result meeting the module's adjustment conditions can be defined as a failure of the robot's path planning. In this case, the viewing angle range of the image module in at least one direction can be determined, where the at least one direction includes the horizontal direction.

[0154] In one implementation, only the horizontal viewing angle range may be determined.

[0155] In one implementation, the horizontal and vertical viewing angle ranges can be determined based on the characteristics of the path planning task and the information of the objects. For example, while expanding the horizontal viewing angle range, the vertical viewing angle range can be expanded based on obstacle information, thereby determining the target viewing angle range of the image module in both the horizontal and vertical directions.

[0156] In this embodiment, based on the requirements of path planning, when 3D reconstruction is required, the robot is actively controlled to adjust the field of view, including the horizontal direction, to meet the requirement of path planning to capture as much information as possible about the surroundings of the robot. Therefore, the accuracy of obstacle avoidance in path planning can be improved.

[0157] Based on the above analysis, it can be determined that robots have different perspective requirements when performing semantic recognition and path planning. The semantic recognition function requires the sensors to capture as much overall information about the object as possible to improve the accuracy of recognition. Generally, a large vertical field of view is required. For example, seeing the whole body of a pet is more likely to identify the type of pet than seeing only its feet. In this case, if the horizontal field of view is sufficient, the vertical field of view does not need to be guaranteed.

[0158] The obstacle avoidance function requires the sensors to capture as much information as possible about the surroundings of the machine. Generally, the vertical field of view needs to be from the ground to an angle slightly above the machine, but the horizontal field of view should be as large as possible. This way, the machine can acquire more information about its surroundings without rotating itself, which can improve its obstacle avoidance ability.

[0159] However, if the viewing angle of the image module is fixed, it is impossible to simultaneously meet the requirements of semantic recognition and obstacle avoidance. In this embodiment, by setting the image module to be rotatable in at least one direction, the viewing angle of the image module is adjustable. Furthermore, it can be adjusted in both the horizontal and vertical directions, thus resolving the conflict between the field of view angles for semantic recognition and obstacle avoidance. The overall solution is to adjust the horizontal and vertical angles of the cleaning robot module to satisfy the field of view angles in both the horizontal and vertical directions.

[0160] In one embodiment, the data processing method for 3D reconstruction, such as Figure 8 As shown, it also includes:

[0161] Step 801: Obtain the robot's location information.

[0162] The robot's position information can be obtained based on IMU sensors.

[0163] Step 804: Using the robot's position information as a reference, obtain multiple frames of target 3D point cloud within the target's viewpoint range in the world coordinate system from the 3D reconstruction results.

[0164] As mentioned earlier, the robot's image module can rotate in at least one direction, acquiring multiple frames of digital and depth images of the environment during the rotation process. Therefore, the 3D reconstruction result includes the 3D point cloud of the environment acquired by the image module from different perspectives. It can obtain and superimpose multiple frames of 3D point clouds from different times and angles onto the world coordinate system to obtain a larger, more accurate, and more detailed 3D reconstruction result in the world coordinate system.

[0165] Based on this 3D map, and using the robot's position information as a reference, multiple frames of target 3D point clouds within the target's viewpoint range in the world coordinate system are obtained from the 3D reconstruction results.

[0166] The target view range can be preset or determined based on the robot's current task.

[0167] Because the 3D reconstruction results have a wider field of view, the extracted multi-frame target 3D point clouds also have a wider field of view, which can provide rich multi-frame target 3D point clouds for subsequent processing.

[0168] Step 806: Process the multi-frame target 3D point cloud.

[0169] Specifically, the processing of multi-frame target 3D point clouds can be determined based on the task currently being performed by the robot. If the current task is recognition, recognition is performed based on the multi-frame target 3D point clouds. If the current task is path planning, path planning is performed based on the multi-frame target 3D point clouds. If the current task is obstacle avoidance, obstacle avoidance is performed based on the multi-frame target 3D point clouds.

[0170] The method in this embodiment obtains multiple frames of target 3D point clouds within the target's viewpoint range in the world coordinate system from the 3D reconstruction results, using the robot's position information as a reference. Since the 3D reconstruction results include 3D point clouds of the environment acquired by the image module with adjustable angle viewpoint range from different viewpoints, the multiple frames of target 3D point clouds extracted from the 3D reconstruction results have multiple different viewpoints. Processing based on the multiple frames of target 3D point clouds can utilize the rich 3D point clouds from multiple viewpoints to improve robot performance.

[0171] In one embodiment, when the robot's recognition result meets the adjustment conditions of the image module, the step of obtaining multiple frames of target 3D point clouds within the target's view range in the world coordinate system from the 3D reconstruction result with reference to the robot's position information includes: obtaining the robot's current view range; and using the robot's position information as a parameter, determining the 3D point cloud of the current view range and the 3D point cloud of the view range adjacent to the current view range from the 3D reconstruction result.

[0172] The adjustment conditions for the image module include the robot recognizing a specific object, the robot being unable to recognize an object, and the robot not recognizing any object.

[0173] When the robot's recognition result is detected to meet the adjustment conditions of the image module, the three-dimensional point cloud of the current view range is first determined in the three-dimensional map, and then the three-dimensional point cloud of the view range adjacent to the current view range is determined, such as the three-dimensional point cloud of the view range adjacent in the vertical direction, so as to extract the multi-frame three-dimensional point cloud with expanded view.

[0174] For example, when encountering pet feces, to avoid touching and spreading it everywhere, the field of view needs to be expanded to determine the location of the feces. Similarly, when furniture and shoes are detected, the field of view can be expanded to determine their location for app display and obstacle avoidance during cleaning. When fragile items are detected, the field of view can be expanded to determine their location, allowing for adjustments to the machine's safe distance and speed. When doors or other objects are detected, the field of view can be expanded to determine whether the door is closed based on its type and whether it is currently enclosed. Finally, when thresholds are detected, their location can be determined to inform the machine's action strategy, assisting it in navigating while minimizing the possibility of getting stuck.

[0175] After acquiring multiple frames of target 3D point cloud, the 3D point cloud is preprocessed, including point cloud filtering, downsampling, and normal estimation, to improve the accuracy of subsequent recognition.

[0176] Then, the 3D point cloud of the current view range and the 3D point cloud of the view range adjacent to the current view range are stitched together to obtain a stitched point cloud; based on the stitched point cloud, at least one of the following is determined: object category, object position, and object pose of the object included in the stitched point cloud.

[0177] Specifically, the recognition process includes: using a deep learning-based object segmentation algorithm to segment the preprocessed point cloud, identifying each object in the point cloud; recognizing each segmented object using a deep learning-based object recognition algorithm, employing a pre-trained model to classify each object into its corresponding category; and estimating the pose of each identified object using a deep learning-based pose estimation algorithm to estimate the object's position and pose.

[0178] Finally, the object category, position, and pose estimation results of the identification results are output to the robot control system so that the robot can perform subsequent statistical recording, navigation strategy, and update and optimize the 3D map.

[0179] Object recognition is achieved by stitching together point clouds. Unlike previous methods that relied on a single frame of data, this approach addresses the challenge of recognizing objects based solely on depth information (RGB). If the frame only contains image data, it's a two-dimensional recognition. Even if it includes depth information, a single frame limits the viewpoint, showing only RGBD from a specific direction. This embodiment, however, fuses RGBD information from multiple perspectives, creating a complete three-dimensional point cloud with color information – a significant improvement. For example, traditional methods rely on photographs to identify people, while this embodiment allows for comprehensive 360*2=720-degree observation without blind spots. Therefore, this method significantly enhances recognition accuracy.

[0180] In another embodiment, when the identified obstacle is not in the center of the field of view, the processing based on the multi-frame target 3D point cloud includes: stitching together the 3D point cloud of the current viewing angle range and the 3D point cloud of the viewing angle range adjacent to the current viewing angle range to obtain a stitched point cloud; identifying at least one of the target clearing area and obstacle information based on the stitched point cloud; determining the target adjustment angle of the image module based on at least one of the target clearing area and obstacle information; controlling the rotation of the image parameters according to the target adjustment angle, and acquiring image data of the environment under different viewing angles during the rotation process.

[0181] Specifically, when an obstacle is not centered in the field of view, the viewing angle needs to be adjusted. This involves acquiring 3D environmental information, i.e., obtaining a reconstructed 3D map, and processing the point cloud data using algorithms such as filtering, segmentation, and registration to obtain 3D data of the environment near the machine. The 3D model is then analyzed and identified, including object recognition and pose estimation, to obtain information such as the area to be cleared and the location and size of obstacles. Based on this information, the angle the gimbal needs to be adjusted is calculated to allow for viewing as much of the area to be cleared as possible from multiple perspectives while avoiding obstacles. The gimbal is then adjusted to the calculated angle for navigation planning, path planning, and obstacle avoidance.

[0182] In another embodiment, when the processing result of the robot performing a task based on a two-dimensional map meets the adjustment conditions of the image module, the step of obtaining a multi-frame target three-dimensional point cloud of the environment within the target range in the world coordinate system from the three-dimensional reconstruction result with reference to the robot's position information includes: obtaining the robot's current position; and obtaining a three-dimensional point cloud around the current position from the three-dimensional reconstruction result with the robot's current position as the center, wherein the surrounding three-dimensional point cloud includes at least the surrounding three-dimensional point cloud in the horizontal direction.

[0183] Specifically, the robot's path planning results satisfy the adjustment conditions of the image module, including failures when performing related tasks based on a 2D map. When the robot performs obstacle avoidance or navigation tasks based on a 2D map, or when path planning fails, the robot's current position and attitude are obtained from the LiDAR. Based on its current position, the robot needs to load a previously established 3D point cloud within a certain range around the robot. This point cloud must include at least the surrounding 3D point cloud in the horizontal direction for navigation, obstacle avoidance, and path rule processing.

[0184] Specifically, the loaded 3D point cloud map is preprocessed, including point cloud filtering, downsampling, and normal estimation, to improve the accuracy of subsequent navigation and obstacle avoidance.

[0185] Then, the robot's posture information is obtained; a local map is constructed based on the 3D point cloud around the current position; and path planning, obstacle avoidance, or navigation processing is performed based on the local map and the robot's posture information.

[0186] Specifically, a 3D map is constructed using 3D point cloud data and recognition results, such as using open-source libraries like Octomap. Then, navigation planning can be performed using this 3D map. This method leverages the rich information in 3D maps and avoids the problem of multiple layers being projected onto a single layer. The planning process takes into account the robot's 3D size, effectively avoiding getting stuck and collisions. It can also combine semantic information to find more suitable navigation paths. Alternatively, the robot can use navigation planning algorithms to plan paths on the 3D point cloud map based on its current and target positions. Furthermore, the robot can use the established 3D map information, combined with real-time sensor data, to implement obstacle avoidance algorithms. For example, based on previous deep learning recognition results and real-time LiDAR data, the robot can determine the type and space occupancy of objects on the current path, and then optimize the planned path, such as fine-tuning parts of the path based on obstacle types and controlling safe distances. Finally, the robot needs to convert the control commands output by the path tracking algorithm into rotation control commands, such as the robot's speed and angular velocity.

[0187] In one embodiment, the method for processing three-dimensional reconstruction data of this application is as follows: Figure 9 As shown,

[0188] The camera and depth sensor are mounted on the module adjustment device. The camera collects RGB information, and the depth sensor collects depth information. An alignment algorithm is used to align the depth information and RGB information. The pose information is obtained from other modules of the robot (such as IMU), and then 3D reconstruction is performed based on the pose information, the aligned depth information and RGB information.

[0189] The specific process of 3D reconstruction has been described in detail in the preceding embodiments and will not be repeated here. This application utilizes an adjustable image module method, which can adaptively open and adjust the angle, thereby improving the robot's intelligence level.

[0190] In another embodiment, a method for processing 3D reconstruction data can be implemented based on an existing 3D map, such as... Figure 10 As shown, the method includes:

[0191] Step 1002: Obtain the robot's location information.

[0192] Step 1004: Using the robot's position information as a reference, obtain multiple frames of target 3D point cloud within the target's viewpoint range in the world coordinate system from the 3D reconstruction results; the 3D reconstruction results are constructed based on image data collected by the robot's image module, which rotates in at least one direction and collects image data of the environment from different viewpoints during the rotation.

[0193] Step 1006: Process the multi-frame target 3D point cloud.

[0194] The method for processing 3D reconstruction data in this embodiment is not limited to any particular method used to construct the 3D reconstructed image. Since the 3D reconstruction result includes 3D point clouds of the environment acquired from different viewpoints by an image module with an adjustable viewing angle range, the multi-frame target 3D point clouds extracted from the 3D reconstruction result have multiple viewpoints. Processing based on these multi-frame target 3D point clouds can utilize the rich 3D point clouds from multiple viewpoints to improve robot performance.

[0195] In another embodiment, when the robot's recognition result meets the adjustment conditions of the image module, the step of obtaining multiple frames of target 3D point cloud within the target's viewpoint range in the world coordinate system from the 3D reconstruction result, using the robot's position information as a reference, includes:

[0196] Obtain the robot's current field of view;

[0197] Using the robot's position information as parameters, the three-dimensional point cloud of the current view range and the three-dimensional point cloud of the view range adjacent to the current view range are determined from the three-dimensional reconstruction results.

[0198] In another embodiment, the processing based on the multi-frame target 3D point cloud includes:

[0199] A stitched point cloud is obtained by stitching together the 3D point cloud of the current view range and the 3D point cloud of the view range adjacent to the current view range; the adjacent view range includes at least the view range adjacent in the vertical direction.

[0200] Based on the stitched point cloud, at least one of the following is determined: object category, object position, and object pose of the objects included in the stitched point cloud.

[0201] In another embodiment, when the identified obstacle is not in the center of the field of view, the processing based on the multi-frame target 3D point cloud includes: stitching together the 3D point cloud of the current view range and the 3D point cloud of the view range adjacent to the current view range to obtain a stitched point cloud.

[0202] Based on the spliced ​​point cloud, at least one of the information of the target clearing area and obstacles is identified;

[0203] The target adjustment angle of the image module is determined based on at least one of the information of the target clearing area and the obstacle;

[0204] The image parameters are rotated according to the target adjustment angle, and image data of the environment from different perspectives are collected during the rotation.

[0205] In another embodiment, when the processing result of the robot performing a task based on a two-dimensional map meets the adjustment conditions of the image module, the step of obtaining a multi-frame target 3D point cloud of the environment within the target range in the world coordinate system from the 3D reconstruction result, with the robot's position information as a reference, includes:

[0206] Get the robot's current position;

[0207] Centered on the robot's current position, obtain the three-dimensional point cloud surrounding the current position from the three-dimensional reconstruction result; the surrounding three-dimensional point cloud includes at least the three-dimensional point cloud surrounding the position in the horizontal direction.

[0208] In another embodiment, the processing based on the multi-frame target 3D point cloud includes:

[0209] Obtain the robot's posture information;

[0210] Construct a local map based on the 3D point cloud surrounding the current location;

[0211] Path planning, obstacle avoidance, or navigation are performed based on the local map and the robot's posture information.

[0212] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0213] Based on the same inventive concept, this application also provides a three-dimensional reconstruction data processing apparatus for implementing the three-dimensional reconstruction data processing method described above. The solutions provided by each apparatus are similar to those described in the above method; therefore, the specific limitations in the apparatus embodiments provided below can be found in the limitations of the method described above, and will not be repeated here.

[0214] In one embodiment, a three-dimensional reconstruction data processing device, such as Figure 11 As shown, it includes:

[0215] The rotation control module 1102 is used to control the robot's image module to rotate in at least one direction and to collect image data of the environment from different perspectives during the rotation.

[0216] Offset acquisition module 1104 is used to acquire the actual offset information of the image module;

[0217] The relationship acquisition module 1106 is used to obtain the pose transformation relationship of the image module posture relative to the robot coordinate system based on the actual offset information.

[0218] The conversion module 1108 is used to obtain multi-frame 3D point clouds of the environment from different perspectives in the world coordinate system based on the image data, the pose transformation relationship of the image module posture relative to the robot coordinate system, and the relative relationship between coordinate systems.

[0219] The fusion module 1110 is used to fuse multiple frames of 3D point clouds of the environment in the world coordinate system to obtain the 3D reconstruction result of the environment.

[0220] In another embodiment, the image data includes digital images and depth images;

[0221] The offset acquisition module is used to acquire the alignment information of the digital image and the depth image, and fuse the digital image and the depth image based on the alignment information to obtain a multi-frame 3D point cloud of the environment in the module coordinate system; according to the pose transformation relationship of the image module posture relative to the robot coordinate system, the multi-frame 3D point cloud of the environment in the module coordinate system, and the relative relationship between the coordinate systems, a multi-frame 3D point cloud of the environment in the world coordinate system is obtained.

[0222] In another embodiment, the image module is mounted on a module adjustment device, the module adjustment device having at least one degree of freedom, and the module adjustment device having adjustment axes to realize each degree of freedom; the actual offset information includes the rotation angle of the adjustment axis;

[0223] The offset acquisition module is used to obtain the rotation matrix of the image module relative to the robot based on the rotation angle of the adjustment axis; and to obtain the pose transformation relationship of the image module posture relative to the robot coordinate system based on the initial position relationship between the image module and the module adjustment device, and the rotation matrix of the image module relative to the robot.

[0224] In another embodiment, when the adjustment axis includes an adjustment axis in a single direction, the relationship acquisition module is used to obtain the rotation matrix of the adjustment axis in the single direction based on the rotation angle of the adjustment axis in the single direction; the rotation matrix of the image module relative to the robot includes the rotation matrix of the adjustment axis in the single direction.

[0225] In another embodiment, when the adjustment axis includes adjustment axes in at least two directions, the relationship acquisition module is used to obtain the rotation matrix of the adjustment axis in each direction according to the rotation angle of the adjustment axis in each of the at least two directions; and to obtain the rotation matrix of the image module relative to the robot according to the product of the rotation matrices of the adjustment axes in each direction.

[0226] In another embodiment, a triggering module is also included, which is used to trigger the image module of the controlled robot to rotate in at least one direction when at least one of the robot's operating state, recognition result, task type, and task processing result meets the three-dimensional reconstruction conditions.

[0227] In another embodiment, the rotation control module includes:

[0228] The offset determination module is used to determine the target offset information of the image module.

[0229] A rotation module is used to control the rotation of the robot's image module based on the target offset information.

[0230] In another embodiment, the offset determination module is used to determine the target viewing angle range of the image module in at least one direction, including the vertical direction, when the robot's recognition result meets the adjustment conditions of the image module; and to determine the target offset information of the image module based on the target viewing angle range of the image module in the at least one direction.

[0231] In another embodiment, the offset determination module is used to determine the target viewing angle range of the image module in at least one direction, where the at least one direction includes the horizontal direction, when the robot performs the target task or the processing result of performing the target task based on the two-dimensional map meets the adjustment conditions of the image module; and to determine the target offset information of the image module based on the target viewing angle range of the image module in at least one direction.

[0232] In another embodiment, a three-dimensional reconstruction data processing device, such as Figure 12 As shown, it includes:

[0233] Information acquisition module 1202 is used to acquire the robot's position information;

[0234] The extraction module 1204 is used to obtain a multi-frame target 3D point cloud within the target's viewpoint range in the world coordinate system from the 3D reconstruction result, with the robot's position information as a reference. The 3D reconstruction result is constructed based on image data collected by the robot's image module. The image module rotates in at least one direction and collects image data of the environment from different viewpoints during the rotation.

[0235] The processing module 1206 is used to process the multi-frame target 3D point cloud.

[0236] In another embodiment, the extraction module is used to obtain the robot's current field of view; using the robot's position information as parameters, it determines the 3D point cloud of the current field of view and the 3D point cloud of the field of view adjacent to the current field of view from the 3D reconstruction results.

[0237] In another embodiment, the processing module is used to stitch together the 3D point cloud of the current view range and the 3D point cloud of the view range adjacent to the current view range to obtain a stitched point cloud; the adjacent view range includes at least the view range adjacent in the vertical direction; and to identify, based on the stitched point cloud, at least one of the object category, object position and object pose of the object included in the stitched point cloud.

[0238] In another embodiment, the processing module is configured to identify at least one of the target clearing area and obstacle information based on the stitched point cloud; determine the target adjustment angle of the image module based on at least one of the target clearing area and obstacle information; control the rotation of the image parameters based on the target adjustment angle; and acquire image data of the environment from different perspectives during the rotation process.

[0239] In another embodiment, when the processing result of the robot performing a task based on a two-dimensional map meets the adjustment conditions of the image module, the extraction module is used to obtain the robot's current position; with the robot's current position as the center, the three-dimensional point cloud around the current position is obtained from the three-dimensional reconstruction result; the surrounding three-dimensional point cloud includes at least the surrounding three-dimensional point cloud in the horizontal direction.

[0240] In another embodiment, the processing module is used to acquire the robot's posture information; construct a local map based on the 3D point cloud around the current position; and perform path planning, obstacle avoidance, or navigation processing based on the local map and the robot's posture information.

[0241] Each module in the aforementioned 3D reconstruction data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0242] In one embodiment, a robot is provided whose internal structure diagram can be as follows: Figure 13 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a threshold recognition method, a door frame orientation recognition method, and a robot threshold crossing control method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the computer device's casing.

[0243] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0244] In one embodiment, a robot is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the methods described in the above embodiments.

[0245] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described in the above embodiments.

[0246] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the methods described in the above embodiments.

[0247] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0248] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0249] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing three-dimensional reconstruction data, characterized in that, Applied to cleaning robots, the method includes: The image module of the robot is controlled to move in at least one direction, horizontal and vertical, and rotate in at least one direction, horizontal and vertical, and to collect image data of the environment from different perspectives during the rotation. The image module is mounted on a module adjustment device, which has at least one degree of freedom and adjustment axes to realize each degree of freedom. Obtain the actual offset information of the image module, wherein the actual offset information includes the rotation angle of the adjustment axis; Based on the rotation angle of the adjustment axis, the rotation matrix of the image module relative to the robot is obtained. Based on the initial positional relationship between the image module and the module adjustment device, and the rotation matrix of the image module relative to the robot, the pose transformation relationship of the image module posture relative to the robot coordinate system is obtained. Based on the image data, the pose transformation relationship of the image module relative to the robot coordinate system, and the relative relationship between coordinate systems, a multi-frame 3D point cloud of the environment from different perspectives in the world coordinate system is obtained. By fusing the multi-frame 3D point cloud of the environment in the world coordinate system, a 3D reconstruction result of the environment is obtained, which is used for path planning, obstacle avoidance, or navigation. The image module controlling the robot rotates in at least one of the horizontal and vertical directions, including: When the robot's recognition result meets the adjustment conditions of the image module, the target viewing angle range of the image module in at least one direction is determined. Based on the target viewing angle range of the image module in at least one direction, the target offset information of the image module is determined. Based on the target offset information, the robot's image module is rotated. The at least one direction includes the vertical direction. When the robot performs a target task, or when the processing result of performing a target task based on a two-dimensional map meets the adjustment conditions of the image module, the target viewing angle range of the image module in at least one direction is determined. Based on the target viewing angle range of the image module in at least one direction, the target offset information of the image module is determined. Based on the target offset information, the rotation of the image module of the robot is controlled. The at least one direction includes the horizontal direction.

2. The method according to claim 1, characterized in that, The image data includes digital images and depth images; The process of obtaining multi-frame 3D point clouds of the environment from different perspectives in the world coordinate system based on the image data, the pose transformation relationship of the image module relative to the robot coordinate system, and the relative relationships between coordinate systems includes: Alignment information of the digital image and the depth image is obtained, and the digital image and the depth image are fused based on the alignment information to obtain a multi-frame 3D point cloud of the environment in the module coordinate system; Based on the pose transformation relationship of the image module posture relative to the robot coordinate system, the multi-frame 3D point cloud of the environment in the module coordinate system, and the relative relationship between the coordinate systems, the multi-frame 3D point cloud of the environment in the world coordinate system is obtained.

3. The method according to claim 1, characterized in that, When the adjustment axis includes a single-direction adjustment axis, obtaining the rotation matrix of the image module relative to the robot based on the rotation angle of the adjustment axis includes: The rotation matrix of the adjustment axis in the single direction is obtained based on the rotation angle of the adjustment axis in the single direction; the rotation matrix of the image module relative to the robot includes the rotation matrix of the adjustment axis in the single direction.

4. The method according to claim 1, characterized in that, When the adjustment axis includes adjustment axes in at least two directions, obtaining the rotation matrix of the image module relative to the robot based on the rotation angle of the adjustment axis includes: Based on the rotation angles of the adjustment axes in at least two directions, the rotation matrices of the adjustment axes in each direction are obtained; The rotation matrix of the image module relative to the robot is obtained by multiplying the rotation matrices of the adjustment axes in each direction.

5. The method according to claim 1, characterized in that, The method further includes: When at least one of the robot's operating state, recognition result, task type, and task processing result meets the three-dimensional reconstruction conditions, the robot's image module is triggered to move in at least one of the horizontal and vertical directions, and rotate in at least one of the horizontal and vertical directions.

6. A method for processing three-dimensional reconstruction data, characterized in that, Applied to cleaning robots, the method includes: Obtain the robot's location information; Using the robot's position information as a reference, multiple frames of target 3D point cloud within the target's viewpoint range in the world coordinate system are obtained from the 3D reconstruction results; the 3D reconstruction results are obtained by processing image data collected by the robot's image module based on the 3D reconstruction data processing method described in any one of claims 1 to 5, wherein the image module moves in at least one direction in the horizontal and vertical directions, rotates in at least one direction in the horizontal and vertical directions, and collects image data of the environment from different viewpoints during the rotation process; Processing is performed based on the multi-frame target 3D point cloud; The processing based on the multi-frame target 3D point cloud includes: path planning, obstacle avoidance, or navigation based on the multi-frame target 3D point cloud.

7. The method according to claim 6, characterized in that, When the robot's recognition result meets the adjustment conditions of the image module, the step of obtaining multiple frames of target 3D point cloud within the target's viewpoint range in the world coordinate system from the 3D reconstruction result, using the robot's position information as a reference, includes: Obtain the robot's current field of view; Using the robot's position information as parameters, the three-dimensional point cloud of the current view range and the three-dimensional point cloud of the view range adjacent to the current view range are determined from the three-dimensional reconstruction results.

8. The method according to claim 7, characterized in that, The processing based on the multi-frame target 3D point cloud includes: A stitched point cloud is obtained by stitching together the 3D point cloud of the current view range and the 3D point cloud of the view range adjacent to the current view range; the adjacent view range includes at least the view range adjacent in the vertical direction. Based on the stitched point cloud, at least one of the following is determined: object category, object position, and object pose of the objects included in the stitched point cloud.

9. The method according to claim 7, characterized in that, When the identified obstacle is not in the center of the field of view, the processing based on the multi-frame target 3D point cloud includes: stitching the 3D point cloud of the current view range and the 3D point cloud of the view range adjacent to the current view range to obtain the stitched point cloud. Based on the spliced ​​point cloud, at least one of the information of the target clearing area and obstacles is identified; The target adjustment angle of the image module is determined based on at least one of the information of the target clearing area and the obstacle; The image module is rotated according to the target adjustment angle, and image data of the environment from different perspectives are collected during the rotation.

10. The method according to claim 6, characterized in that, When the processing result of the robot performing the task based on the two-dimensional map meets the adjustment conditions of the image module, the step of obtaining multiple frames of target 3D point cloud within the target's viewpoint range in the world coordinate system from the 3D reconstruction result, using the robot's position information as a reference, includes: Obtain the current position of the robot; Using the robot's current position as the center, obtain the three-dimensional point cloud surrounding the current position from the three-dimensional reconstruction result; the surrounding three-dimensional point cloud includes at least the three-dimensional point cloud surrounding the position in the horizontal direction.

11. The method according to claim 10, characterized in that, The path planning, obstacle avoidance, or navigation processing based on the multi-frame target 3D point cloud includes: Obtain the robot's posture information; Construct a local map based on the 3D point cloud surrounding the current location; Path planning, obstacle avoidance, or navigation are performed based on the local map and the robot's posture information.

12. A processing device for three-dimensional reconstruction data, characterized in that, Deployed in a cleaning robot, the device includes: A rotation control module is used to control the robot's image module to move in at least one direction (horizontal and vertical) and rotate in at least one direction (horizontal and vertical), and to collect image data of the environment from different perspectives during the rotation. The image module is mounted on a module adjustment device, which has at least one degree of freedom and adjustment axes to realize each degree of freedom. An offset acquisition module is used to acquire the actual offset information of the image module, the actual offset information including the rotation angle of the adjustment axis; The relationship acquisition module is used to obtain the rotation matrix of the image module relative to the robot based on the rotation angle of the adjustment axis, and to obtain the pose transformation relationship of the image module posture relative to the robot coordinate system based on the initial position relationship between the image module and the module adjustment device, and the rotation matrix of the image module relative to the robot. The conversion module is used to obtain multi-frame 3D point clouds of the environment from different perspectives in the world coordinate system based on the image data, the pose transformation relationship of the image module posture relative to the robot coordinate system, and the relative relationship between coordinate systems. The fusion module is used to fuse the multi-frame 3D point cloud of the environment in the world coordinate system to obtain the 3D reconstruction result of the environment; The rotation control module includes an offset determination module and a rotation module. The offset determination module is used to determine the target viewing angle range of the image module in at least one direction when the robot's recognition result meets the adjustment conditions of the image module, and to determine the target offset information of the image module based on the target viewing angle range in the at least one direction, where the at least one direction includes the vertical direction; when the robot performs a target task, or when the processing result of performing a target task based on a 2D map meets the adjustment conditions of the image module, the module determines the target viewing angle range of the image module in at least one direction, and to determine the target offset information of the image module based on the target viewing angle range in the at least one direction, where the at least one direction includes the horizontal direction; the rotation module is used to control the rotation of the robot's image module based on the target offset information.

13. A processing device for three-dimensional reconstruction data, characterized in that, Deployed in a cleaning robot, the device includes: The information acquisition module is used to acquire the robot's position information; The extraction module is used to obtain a multi-frame target 3D point cloud within the target's viewpoint range in the world coordinate system from the 3D reconstruction result, using the robot's position information as a reference. The 3D reconstruction result is obtained by processing image data collected by the robot's image module based on the 3D reconstruction data processing method described in any one of claims 1 to 5. The image module moves in at least one direction (horizontal or vertical) and rotates in at least one direction (horizontal or vertical), and collects image data of the environment from different viewpoints during the rotation. The processing module is used to perform path planning, obstacle avoidance, or navigation processing based on the multi-frame target 3D point cloud.

14. A robot comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

Images