Robot movement control method and robot

Abstract

The application provides a robot movement control method and a robot, wherein the method comprises: in response to the robot reaching a current waypoint corresponding to a current time step, acquiring image information and attitude information of the robot at the current waypoint; determining a current semantic point cloud according to the image information and the attitude information; determining environment memory topological information of the current time step according to the current waypoint and the current semantic point cloud; determining a next waypoint of the current waypoint according to the environment memory topological information, and controlling the robot to move to the next waypoint. The application can convert instantaneous observation into structured environment memory information through the binding of the waypoint, the semantic point cloud and the environment memory topological information, realize the continuous accumulation of long-term spatial structure knowledge, realize the online mapping of an unknown environment, solve the problems of spatial memory fragmentation and the lack of long-term structure knowledge accumulation in the prior art, and further improve the robustness and efficiency of navigation decision-making.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a robot movement control method and a robot. Background Technology

[0002] Object navigation is a key task in embodied artificial intelligence and robot interaction, requiring the seamless integration of visual perception, natural language understanding, and spatial reasoning to enable robots to navigate in unknown environments to find designated target objects.

[0003] Existing target navigation methods are typically based on learning (reinforcement learning, imitation learning) or zero-shot methods (open vocabulary scene understanding, map-guided exploration).

[0004] However, these two types of target navigation methods in the existing technology generally rely on the concatenation of temporary visual observations or simple map representations, which easily leads to target confusion, path redundancy or even complete failure in long-term navigation tasks. At the same time, they lack real-time adaptability to unknown environments and lack effective support from high-level semantic context such as room type and object distribution, resulting in poor navigation performance during target navigation and affecting the success rate of robot interaction tasks. Summary of the Invention

[0005] The purpose of this application is to provide a robot movement control method and a robot to address the shortcomings of the prior art, thereby solving the problem of poor navigation performance in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a robot movement control method, the method comprising: When the robot reaches the current waypoint corresponding to the current time step, the robot's image information and attitude information at the current waypoint are obtained. Based on the image information and the pose information, a current semantic point cloud is determined, wherein the current semantic point cloud includes sub-point clouds of multiple dimensions that are spatially registered. Based on the current waypoint and the current semantic point cloud, the environmental memory topology information of the current time step is determined. The environmental memory topology information includes multiple nodes and attribute information of each node. Each node represents a spatial point in the environment in which the robot is located. The attribute information of each node is used to indicate the semantic features of the spatial point. Based on the environmental memory topology information, the next waypoint of the current waypoint is determined, and the robot is controlled to move to the next waypoint.

[0007] Secondly, another embodiment of this application provides a robot, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the robot is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0008] The beneficial effects of this application are as follows: By responding to the robot's arrival at the current waypoint corresponding to the current time step, the image information and attitude information of the robot at the current waypoint are obtained. Based on the image information and attitude information, the current semantic point cloud is determined, and based on the current waypoint and the current semantic point cloud, the environmental memory topology information of the current time step is determined. By binding waypoints, semantic point clouds, and environmental memory topology information, the instantaneous observations of the robot at the current time step can be transformed into structured environmental memory information, forming an evolvable environmental cognition model. This enables the continuous accumulation of long-term spatial structure knowledge, online mapping of unknown environments, and solves the problems of fragmented spatial memory and lack of long-term structural knowledge accumulation in existing technologies. Furthermore, through this globally structured memory of environmental memory topology information, the next waypoint of the current waypoint can be determined, and the robot can be controlled to move to the next waypoint. This reduces path drift caused by single-frame false detection or occlusion, avoids getting lost due to local observation failure, thereby improving the robustness and efficiency of navigation decisions. At the same time, it increases the success rate in long-distance tasks.

[0009] Furthermore, in practical applications, there is no need for offline training for specific scenarios or predefined environment-related prior knowledge. The robot can directly start navigation upon arriving in an unknown environment, significantly reducing the adaptation cost of engineering deployment and achieving perfect adaptation in zero-sample navigation scenarios. In addition, it reduces computing and storage overhead, adapts to robot edge deployment, and enhances human-robot collaboration friendliness. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic flowchart of a robot movement control method provided in an embodiment of this application; Figure 2 A flowchart illustrating the process of determining the current semantic point cloud in the robot movement control method provided in this application embodiment; Figure 3A schematic diagram of a process for obtaining the current scene base point cloud in the robot movement control method provided in the embodiments of this application; Figure 4 A flowchart illustrating the process of determining the point cloud of each current object in the robot movement control method provided in this application embodiment; Figure 5 A schematic diagram of a process for obtaining the currently navigable point cloud in the robot mobility control method provided in the embodiments of this application; Figure 6 A schematic flowchart illustrating the process of determining the current obstacle point cloud in the robot movement control method provided in this application embodiment; Figure 7 A schematic flowchart illustrating the process of obtaining environmental memory topology information in the robot mobility control method provided in this application embodiment; Figure 8 This is a flowchart illustrating the process of determining the attribute information of a newly added node in the robot movement control method provided in this application embodiment. Figure 9 This is a schematic flowchart illustrating the process of obtaining the environmental memory topology information of the current time step in the robot motion control method provided in this application embodiment; Figure 10 This is a schematic diagram of the robot structure provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0013] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0014] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0015] Existing target navigation methods are typically based on learning (reinforcement learning, imitation learning) or zero-shot methods (open vocabulary scene understanding, map-guided exploration).

[0016] However, these two types of target navigation methods in the existing technology have the following drawbacks: First, spatial memory is fragmented and lacks a mechanism for accumulating long-term environmental structural knowledge, which can easily lead to target confusion, path redundancy, or even complete failure in long-term navigation tasks. Second, the application of topological structures is limited. Traditional topological maps are mostly statically predefined or generated based on pre-calculated paths, lacking real-time adaptability to unknown environments and not deeply integrated with language-driven spatial reasoning. Third, semantic information is separated from geometric structure, and navigation decisions lack effective support from high-level semantic contexts such as room type and object distribution. Finally, some methods still require task-specific training, have insufficient zero-shot generalization ability, and are difficult to adapt to open-vocabulary object categories and diverse indoor scenes, limiting deployment costs and adaptability.

[0017] Therefore, these two types of target navigation methods in the existing technology generally rely on the concatenation of temporary visual observations or simple map representations, which easily leads to target confusion, path redundancy or even complete failure in long-term navigation tasks. At the same time, they lack real-time adaptability to unknown environments and lack effective support from high-level semantic contexts such as room type and object distribution, resulting in poor navigation performance during target navigation and affecting the success rate of robot interaction tasks.

[0018] Based on the aforementioned problems, this application proposes a robot mobility control method. By responding to the robot's arrival at the current waypoint corresponding to the current time step, it acquires the robot's image and posture information at the current waypoint. Based on this information, it determines the current semantic point cloud, addressing the issues of fragmented spatial memory and lack of long-term structural knowledge accumulation in existing technologies. This allows for the determination of environmental memory topology information for the current time step based on the current waypoint and semantic point cloud, forming an evolvable environmental cognition model. This enables online mapping of unknown environments. Furthermore, the method uses the environmental memory topology information to determine the next waypoint and controls the robot to move to it. This reduces path drift caused by single-frame false detections or occlusion, improving navigation robustness. It also increases the success rate in long-distance tasks and reduces deployment costs. In addition, it enhances human-robot collaborative friendliness.

[0019] First, the relevant background of the robot movement control method provided in the embodiments of this application will be explained.

[0020] It should be understood that the background related to the robot movement control method provided in the embodiments of this application can be deployed in any device that needs to perform dynamic environmental perception and movement, such as a robot, specifically a humanoid robot, a humanoid robot, or a robot dog.

[0021] By example, by implementing the robot mobility control method provided in the embodiments of this application, the robot can more intelligently understand the environment and perform mobility tasks or other complex tasks.

[0022] For example, a robot can be a robot dog. Specifically, a robot dog is a robot that mimics the appearance of a canine and typically walks on four legs.

[0023] Specifically, a robot dog may include a mechanical structure, a drive system, a sensing system, a control system, and a power supply system.

[0024] For example, the mechanical structure includes a fuselage body and a leg structure, the leg structure including multiple legs, each leg including a hip joint and a knee joint, and may also include an ankle joint, the mechanical structure being used to perform specific actions, such as movement.

[0025] The drive system includes joint motors, reducers, and transmission structures. The joint motors control the rotational or linear motion of the joints in the leg structure, providing precise power output. The reducers lower the rotational speed and increase the torque, enabling the robot dog to stand and carry heavy loads. The transmission structure, including timing belts and connecting rods, converts the rotational motion of the motors into leg swinging motion.

[0026] The sensing system includes a vision sensor and a lidar sensor. Optionally, the sensing system may also include a torque sensor, an inertial measurement unit, and a tactile sensor. The sensing system is used to detect environmental information in real time to obtain real-time environmental information for controlling the robot dog. Optionally, the vision sensor may include a binocular camera and a depth camera.

[0027] Optionally, the vision sensor can be located on the head, front end, or top of the robot dog.

[0028] Optionally, the lidar can be fixed to the head, front end, or top of the robot dog.

[0029] The control system includes a processor, a memory, and a communication module. The memory stores machine-readable instructions that the processor can execute. When the robot dog's control system is running, the processor executes the machine-readable instructions to process and store the sensor data obtained by the sensing system, and executes the steps of the robot movement control method provided in this application embodiment to control the robot dog's movement.

[0030] Optionally, the robot dog may also include a human-computer interaction interface to enable human-computer interaction.

[0031] The robot movement control method provided in this application will be described in detail below with reference to several embodiments.

[0032] Figure 1 This is a flowchart illustrating a robot movement control method provided in an embodiment of this application, with reference to... Figure 1 As shown, the executing entity of this method is any electronic device with processing capabilities, such as the aforementioned robot dog, and the method includes: S101. In response to the robot reaching the current waypoint corresponding to the current time step, obtain the robot's image information and attitude information at the current waypoint.

[0033] It should be noted that during the robot's movement, waypoint generation and movement are performed periodically. The current time step is a relative temporal concept, dynamically advancing with the robot's navigation progress. For example, when the robot is at time step t, its current waypoint is the one at time step t. The pose anchor point determined by the decision and reached through motion control is used as the spatiotemporal reference in steps S101-S104 executed at time step t. That is, in this embodiment, the current time step is taken as time step t, and the starting point of the next time step (i.e., the next waypoint) is determined from the current waypoint at the current time step t.

[0034] Here, the current waypoint corresponding to the current time step refers to the discrete cognitive anchor point formed by the robot during continuous navigation. The current waypoint corresponds to a spatial point in real space. The current waypoint corresponding to the current time step is the next waypoint of the previous time step.

[0035] Optionally, responding to the robot reaching the current waypoint corresponding to the current time step includes: when the robot reaches the position corresponding to the current waypoint, determining whether the robot meets the preset pose convergence conditions according to the preset pose convergence conditions; if so, determining that the robot has reached the current waypoint corresponding to the current time step. The preset pose convergence conditions may include at least one of the following: position error < 5 cm, orientation error < 3°, angular velocity / linear velocity < threshold, and completion of sensor data acquisition synchronization, etc.

[0036] Optionally, a sensing system can be used to acquire image and attitude information of the robot at the current waypoint. The image information includes RGB images and depth images. The attitude information includes: position p t =(x, y, z) and orientation q t Among them, the direction towards P t These can be represented using Euler angles (roll, pitch, yaw) or quaternions. Roll is the roll angle, pitch is the pitch angle, and yaw is the yaw angle. The quaternion consists of (w, x', y', z'), where w is a scalar, x' is the x-axis vector in 3D space, y' is the y-axis vector in 3D space, and z' is the z-axis vector in 3D space.

[0037] S102. Determine the current semantic point cloud based on image information and pose information.

[0038] Optionally, after obtaining the image information and attitude information, the current semantic point cloud can be constructed based on the robot's image information and attitude information at the current waypoint, using the current waypoint as the spatiotemporal anchor point.

[0039] The current semantic point cloud includes multiple spatially registered sub-point clouds. Specifically, each point cloud is an independent topological entity, and the sub-point clouds share physical addressing, geometric origin, and semantic heterogeneity. Each sub-point cloud originates from the same observation but carries functions under different task-specific semantic dimensions.

[0040] In one example, after obtaining image information and pose information, each sub-point cloud can be constructed based on the image information and pose information according to a pre-configured construction strategy for each sub-point cloud, and used as the current semantic point cloud.

[0041] In one example, after obtaining image information and pose information, the sub-point clouds of the previous time step can be incrementally updated online according to the image information and pose information and the pre-configured generation strategy of each sub-point cloud to obtain the sub-point clouds of the current time step, which are used as the current semantic point cloud.

[0042] The current semantic point cloud is a collection of multiple sub-point clouds that are spatially registered in a unified global coordinate system and have clear semantic divisions. For example, the multiple sub-point clouds may include one or more of the following: scene base point cloud, object point cloud, navigable point cloud, front edge point cloud, interactive intent point cloud, illumination-sensitive point cloud, and memory confidence point cloud.

[0043] Specifically, the scene base point cloud represents the main structure of the robot's environment, serving as a geometric reference skeleton during navigation. The object point cloud represents identifiable or interactive dynamic and semi-static objects in the robot's environment, providing category information about objects. The navigable point cloud represents the actually safe traversable areas in the robot's environment, serving as feasibility input for path planning. The leading edge point cloud represents the boundary lines or points between explored and unexplored areas in the robot's environment, providing exploration targets for the robot. The interaction intent point cloud represents the surface area of ​​the object most likely to point to or request interaction in the current field of view. The illumination-sensitive point cloud represents areas in the robot's environment where visual features are easily lost due to changes in illumination. The memory confidence point cloud represents the stability of each spatial point in the robot's environment in long-term memory.

[0044] S103. Determine the environmental memory topology information for the current time step based on the current waypoint and the current semantic point cloud.

[0045] Optionally, after obtaining the current semantic point cloud, it can be dynamically constructed and updated using the current waypoint and the current semantic point cloud to obtain the environmental memory topology information of the current time step, thereby capturing the scene connection relationship, adjacency structure and semantic information in the environment through the environmental memory topology information of the current time step.

[0046] The environmental memory topology information includes multiple nodes and their attribute information. Each node represents a spatial point in the robot's environment, and its attribute information indicates the semantic features of the spatial point. Specifically, the essence of environmental memory topology information is spatial memory. The attribute information of each node refers to the topological description data representing the semantic state of the surrounding environment of that spatial location, obtained by the robot using the spatial location corresponding to that node as the query origin and performing spatial neighborhood retrieval and statistical analysis on the constructed current semantic point cloud.

[0047] Optionally, the environmental memory topology information may also include the edges corresponding to each node and the robot's historical movement node sequence, wherein the edges corresponding to each node are used to represent the robot's explored paths, and the historical movement node sequence is used to represent the robot's movement trajectory.

[0048] For example, taking the first time step as an example, when the robot arrives at the first time step, the first node can be created, and the attribute information of the first node can be generated based on the current semantic point cloud of the first waypoint.

[0049] For example, the attribute information of each node includes at least: identifier, location information, neighboring object category information, room type, and number of frontier points. The attribute information of each node may also include: neighboring point cloud confidence.

[0050] Specifically, location information refers to the node's three-dimensional spatial coordinates and orientation in the global coordinate system. Neighborhood object category information refers to the statistical distribution of the semantic categories to which all points in the current object point cloud belong within a preset radius *r* centered on the node. Room type refers to the high-level semantic discrimination result of the functional area type of the physical space to which the current waypoint belongs. Number of leading-edge points refers to the total number of leading-edge points in the current leading-edge point cloud contained within the current node's neighborhood, and is a scalar integer. Neighborhood point cloud confidence is a quantitative assessment value of the quality and consistency of point cloud data within the neighborhood centered on the current node, reflecting the reliability of the memory in that area.

[0051] In one example, the category information of neighboring objects can be determined by the current object point cloud.

[0052] In another example, the voxel coverage density in the neighborhood of a node can be determined by the current scene base point cloud; the multi-view observation consistency in the neighborhood of a node can be determined by the current object point cloud; the proportion of interpolation points in the neighborhood of a node can be determined by the current navigable point cloud; and a weighted fusion is performed based on the voxel coverage density in the neighborhood of a node, the multi-view observation consistency in the neighborhood of a node, and the proportion of interpolation points in the neighborhood of a node to obtain the confidence of the neighborhood point cloud of a node. Thus, by embedding the memory quality assessment mechanism into the topological node, the robot has the metacognitive ability to "know where it is uncertain," enabling the decision-making process to dynamically reduce the weight of low-confidence nodes.

[0053] S104. Based on the environmental memory topology information, determine the next waypoint of the current waypoint and control the robot to move to the next waypoint.

[0054] Optionally, after obtaining the environmental memory topology information of the current time step, the next waypoint of the current waypoint can be determined based on the environmental memory topology information and the current waypoint, and the robot can be controlled to move to the next waypoint.

[0055] In one example, after obtaining the environmental memory topology information, task-related information and the sequence of historical moving nodes corresponding to the robot before the current waypoint can be obtained. The task-related information, environmental memory topology information, and historical moving node sequence are then input into a pre-trained visual language model, which determines and generates the next waypoint.

[0056] It should be noted that after determining the next waypoint from the current waypoint, path planning can be performed between the current waypoint and the next waypoint, resulting in multiple intermediate path points. In this embodiment, each intermediate path point only guides the robot's movement during its movement and does not participate in the determination of semantic point cloud and environmental memory topology information. Furthermore, for ease of explanation, this embodiment uses a one-to-one correspondence between the current time step and the current waypoint; however, adjustments can be made based on actual circumstances in practical applications.

[0057] In this embodiment, by responding to the robot's arrival at the current waypoint corresponding to the current time step, image and attitude information of the robot at the current waypoint are acquired. Based on the image and attitude information, the current semantic point cloud is determined, and based on the current waypoint and the current semantic point cloud, the environmental memory topology information of the current time step is determined. By binding waypoints, semantic point clouds, and environmental memory topology information, the robot's instantaneous observations at the current time step can be transformed into structured environmental memory information, forming an evolvable environmental cognition model. This enables the continuous accumulation of long-term spatial structure knowledge, online mapping of unknown environments, and solves the problems of fragmented spatial memory and lack of long-term structural knowledge accumulation in existing technologies. Furthermore, through this globally structured memory of environmental memory topology information, the next waypoint of the current waypoint is determined, and the robot is controlled to move to the next waypoint. This reduces path drift caused by single-frame false detections or occlusions, avoids getting lost due to local observation failures, thereby improving the robustness and efficiency of navigation decisions and increasing the success rate in long-distance tasks.

[0058] Furthermore, in practical applications, there is no need for offline training for specific scenarios or predefined environment-related prior knowledge. The robot can directly start navigation upon arriving in an unknown environment, significantly reducing the adaptation cost of engineering deployment and achieving perfect adaptation in zero-sample navigation scenarios. In addition, it reduces computing and storage overhead, adapts to robot edge deployment, and enhances human-robot collaboration friendliness.

[0059] The following is an illustrative example of the process for determining the current semantic point cloud.

[0060] In one possible implementation, the current semantic point cloud includes: the current scene base point cloud and the current object point cloud in multiple dimensions. Figure 2 This is a flowchart illustrating the process of determining the current semantic point cloud in the robot movement control method provided in this application embodiment, with reference to... Figure 2 As shown, in S102 above, determining the current semantic point cloud based on image information and pose information includes: S201. Based on the image information and posture information, incrementally update the historical scene base point cloud of the robot in the previous time step to obtain the current scene base point cloud.

[0061] Optionally, the scene point cloud to be added can be determined based on the image information and pose information, and the scene point cloud to be added can be added to the historical scene base point cloud of the previous time step to obtain the initial scene base point cloud of the current time step. The initial scene base point cloud of the current time step can be optimized to obtain the current scene base point cloud.

[0062] Optionally, the incremental scene point cloud at the current time step can be determined based on image information, posture information, and the historical scene base point cloud of the previous time step. Based on the incremental scene point cloud, the robot's historical base point cloud at the previous time step can be incrementally updated to obtain the current scene base point cloud.

[0063] Among them, the current scene base point cloud refers to the scene base point cloud at the current time step.

[0064] S202. Determine the point cloud of each current object based on the image information, pose information, and the current scene base point cloud.

[0065] Optionally, after obtaining the current scene base point cloud, the point clouds of each current object can be determined based on the image information, pose information, and the current scene base point cloud.

[0066] Each current object point cloud includes one or more of the following: current object point cloud, current navigable point cloud, current front edge point cloud, current interaction intent point cloud, current illumination-sensitive point cloud, and current memory confidence point cloud.

[0067] In one example, a pre-trained open vocabulary detection model, combined with task instructions, can be used to locate at least one pixel region of the noun phrase in the task instructions within an RGB image. Each pixel region is then back-projected into 3D space, registered with the current scene's base point cloud, and interactive weights are assigned to overlapping points, thereby generating an interactive intent point cloud. The open vocabulary detection model could, for example, include the GroundingDINO model.

[0068] In another example, the local contrast variance of each pixel in the RGB image can be calculated and combined with a preset illuminance map to identify low-illuminance or high-light overflow areas, thereby obtaining at least one target area. The pixels in each target area are then back-projected into 3D points to obtain a light-sensitive point cloud.

[0069] By incrementally updating the historical scene base point cloud of the robot at the previous time step using image and posture information, the current scene base point cloud is obtained. Based on the current scene base point cloud, the point clouds of each current object are determined. This enables the unification of long-term continuity of geometric memory, hierarchical dependence of semantic representation, and online evolution of topological structure in robot navigation, fundamentally breaking through the inherent bottlenecks of fragmented spatial memory, semantic-geometric separation, and static topological rigidity in existing technologies.

[0070] In one possible implementation, the image information includes: an RGB image and a depth image. Figure 3 This is a flowchart illustrating the process of obtaining the current scene base point cloud in the robot movement control method provided in this application embodiment, with reference to... Figure 3 As shown, in step S201 above, the robot incrementally updates the historical scene base point cloud of the previous time step based on image information and posture information to obtain the current scene base point cloud, including: S301. Based on the depth image and the robot's camera intrinsic parameter matrix, determine multiple first spatial points corresponding to the depth image.

[0071] Optionally, each pixel in the depth image can be transformed to the camera coordinate system based on the robot's camera intrinsic parameter matrix to obtain multiple first spatial points corresponding to the depth image.

[0072] For example, the first spatial point X c The following formula (1) can be used to determine this: (1) Where d=D t (u,v), D t Let u be the depth image, v be the x-coordinate of the depth image, and K be the camera intrinsic parameter matrix.

[0073] S302. Based on the RGB image, add color attributes to each first space point.

[0074] Optionally, color attributes can be added to each first spatial point based on the pixel values ​​of each pixel in the RGB image.

[0075] The color attribute of the first spatial point can be its color value. For example, the first spatial point X... c Color value c = I t (u,v). Among them, I t It is an RGB image.

[0076] S303. Based on the attitude information, transform each first spatial point to the global coordinate system to obtain multiple current spatial points.

[0077] Optionally, after obtaining each first spatial point with color attributes, the first spatial points are subjected to rigid body transformation according to the attitude information, and the first spatial points are transformed to the global coordinate system to obtain multiple current spatial points.

[0078] For example, the current spatial point X w The following formula (2) can be used to determine this: (2) Where, q t For orientation, R(q) t ) represents the rotation matrix corresponding to the orientation, p t For position, X c This is the first spatial point.

[0079] S304. Determine the first incremental point cloud of each current spatial point and the historical scene base point cloud.

[0080] Optionally, after obtaining each current spatial point, it can be determined whether each current spatial point falls within the covered voxels of the historical scene base point cloud. If not, the current spatial point is determined as the first incremental point cloud, thereby retaining only the newly added part of the environment, eliminating duplicate observations, and significantly compressing the amount of data.

[0081] S305. Based on the pre-trained voxel mesh model, update the first incremental point cloud to the historical scene base point cloud to obtain the current scene base point cloud.

[0082] Optionally, after obtaining the first incremental point cloud, the first incremental point cloud is updated to the historical scene base point cloud according to the pre-trained voxel mesh model to obtain the current scene base point cloud, so as to reduce the point cloud density, improve the calculation effect, eliminate measurement noise, and at the same time maintain the integrity of geometric features.

[0083] For example, the scene base point cloud Mscene can be obtained by referring to the following formula (3): (3) Where Mscene is the scene base point cloud, VoxelDownsample is the voxel mesh model, and X... w For the current spatial point, r pcd This represents the resolution of the voxel mesh model.

[0084] By determining the first incremental point cloud of each current spatial point and the historical scene base point cloud, the first incremental point cloud is updated to the historical scene base point cloud through a pre-trained voxel mesh model to obtain the current scene base point cloud. This allows only the incremental point cloud to be injected in the current time step, naturally suppressing repeated mapping and redundant storage. At the same time, updating through the voxel mesh model can establish a long-term scene base memory with temporal continuity and spatial consistency while ensuring geometric accuracy.

[0085] In one possible implementation, the current object point cloud has multiple dimensions, including: the current object point cloud, the current navigable point cloud, the current obstacle point cloud, and the current front point cloud. Figure 4 This is a flowchart illustrating the process of determining the point cloud of each current object in the robot movement control method provided in this application embodiment, with reference to... Figure 4 As shown, in step S202 above, the point cloud of each current object is determined based on image information, pose information, and the current scene base point cloud, including: S401. Based on the image information and posture information, incrementally update the historical object point cloud of the robot in the previous time step to obtain the current object point cloud.

[0086] Optionally, at least one object at the current time step can be identified based on image information and pose information, and the historical object point cloud of the previous time step can be incrementally updated based on each object to obtain the current object point cloud.

[0087] Optionally, the current object point cloud can be obtained by filtering from the current scene base point cloud according to a preset object category directory.

[0088] For example, a pre-trained semantic segmentation model can perform semantic segmentation on RGB images in image information based on a preset object category directory, and identify multiple categories of objects in the RGB image and the regions where each category of objects is located.

[0089] For example, the objects of each category are traversed. For the object of the current category, based on the region where the object of the current category is located, multiple corresponding pixels of the object of the current category in the depth image are determined. Then, each corresponding pixel is transformed into the camera coordinate system according to the robot's camera intrinsic parameter matrix to obtain multiple second spatial points corresponding to the depth image. Specifically, the process of transforming each corresponding pixel into the camera coordinate system according to the robot's camera intrinsic parameter matrix to obtain multiple second spatial points corresponding to the depth image can be implemented with reference to the aforementioned formula (1).

[0090] For example, after obtaining multiple second spatial points corresponding to the depth image, color data is added to each second spatial point according to the RGB image. Specifically, it can be implemented with reference to the aforementioned S302.

[0091] For example, after obtaining each second spatial point with color attributes, a rigid body transformation is performed on each second spatial point according to the pose information, and each second spatial point is transformed to the global coordinate system to obtain multiple object spatial points. Specifically, the object spatial points can be calculated with reference to the aforementioned formula (2).

[0092] For example, after obtaining the spatial points of each object, the second incremental point cloud of each object spatial point and the historical object point cloud is determined, and the second incremental point cloud is updated to the historical object point cloud according to the pre-trained voxel mesh model to obtain the current object point cloud.

[0093] Specifically, the point cloud of objects in the c-th category The following formula (4) can be used as a reference: (4) in, For the point cloud of the c-th category, VoxelDownsample is a voxel mesh model. For the j-th object point of the c-th category, r pcd This represents the resolution of the voxel mesh model.

[0094] S402. Based on the current scene base point cloud and the current object point cloud, update the robot's historical navigable point cloud at the previous time step to obtain the current navigable point cloud.

[0095] Optionally, after obtaining the current scene base point cloud, semantic construction is performed based on the current scene base point cloud and the current object point cloud, and the historical navigable point cloud of the robot in the previous time step is updated to obtain the current navigable point cloud.

[0096] For example, geometric filtering is performed on the current scene base point cloud to obtain a first navigable point cloud set, and semantic enhancement is performed on the current object point cloud to obtain a second navigable point cloud set. Based on the first and second navigable point cloud sets, the robot's historical navigable point cloud at the previous time step is updated to obtain the current navigable point cloud. This ensures that no possible ground areas are missed while ensuring that special structures are correctly understood and processed, thereby enabling the robot to navigate safely and intelligently in the real world.

[0097] S403. Determine the current obstacle point cloud based on the current scene base point cloud.

[0098] In one example, the current obstacle point cloud is obtained by filtering from the current scene base point cloud. Specifically, the filtering can be based on the navigable height range, adding point clouds outside the navigable height range to the current obstacle point cloud.

[0099] S404. Determine the current leading edge point cloud based on the current scene base point cloud, the current navigable point cloud, and the current obstacle point cloud.

[0100] Optionally, based on the current scene base point cloud, a discrete continuous grid space corresponding to the current scene base point cloud is generated. In the discrete continuous grid space corresponding to the current scene base point cloud, the grid state of each grid in the discrete continuous grid space is determined by the current navigable point cloud and the current obstacle point cloud. Based on the grid state of each grid, the neighborhood relationship of each grid is determined. Based on the neighborhood relationship of each grid, the current frontier point cloud is generated. This achieves efficient spatial reasoning while maintaining semantic information, completes boundary detection with linear time complexity, and naturally obtains noise resistance and smoothness through the discretization characteristics of the grid. Ultimately, it supports real-time, incremental exploration planning.

[0101] For example, the continuous space can be divided into Nx segments based on the spatial boundaries of the current scene's base point cloud. There are Ny grids, each with a size of rg. rg. Specifically, it can be determined by referring to the following formula (5): (5) Where, x min The minimum x-coordinate of the spatial boundary of the current scene's base point cloud. max The maximum x-coordinate of the spatial boundary of the current scene's base point cloud, y min The minimum y-coordinate of the spatial boundary of the current scene's base point cloud. max This represents the maximum y-coordinate of the spatial boundary of the current scene's base point cloud.

[0102] For example, after obtaining each grid, the point clouds in the current navigable point cloud and the current obstacle point cloud are sequentially projected into each grid to obtain the grid state of each grid.

[0103] Specifically, iterate through all point clouds in the current navigable point cloud and the current obstacle point cloud. For the current point cloud p that has been traversed, determine the corresponding grid of the current point cloud p based on the horizontal and vertical coordinates of the current point cloud p, and determine the grid state of the corresponding grid based on the type of the current point cloud p.

[0104] For example, for the current point cloud p, the corresponding mesh φ(p) can be obtained by referring to the following formula (6): (6) Where px is the x-coordinate of the current point cloud and py is the y-coordinate of the current point cloud.

[0105] Specifically, if the current point cloud p is a navigable point cloud, the grid state of the corresponding grid is determined to be navigable; if the current point cloud p is an obstacle point cloud, the grid state of the corresponding grid is determined to be obstacle.

[0106] Specifically, after traversing all point clouds in the current navigable point cloud and the current obstacle point cloud, all grids in the non-navigable state and the non-obstacle state are determined to be in the unknown state.

[0107] For example, for grids whose grid state is unknown, a preset boundary grid judgment condition is used to determine whether the grid state of each grid is a boundary state, thereby obtaining each grid in a boundary state. The boundary grid judgment condition includes: the grid is not in a navigable state, is not in an obstacle state, is adjacent to at least one grid in a navigable state, and is not directly adjacent to a grid in an obstacle state.

[0108] For example, after obtaining each mesh of the boundary state, the boundary points can be generated by projecting each mesh of the boundary state, thereby obtaining the current frontier point cloud Mfro. Specifically, it can be obtained by referring to the following formula (7): (7) in, The first in the current cutting-edge point cloud Mfro The frontier point cloud, (i k i k ) is the first The coordinates of the grid at each boundary state, Z floor x is the preset ground plane elevation. min The minimum x-coordinate of the spatial boundary of the current scene's base point cloud, y min The minimum y-coordinate of the spatial boundary of the current scene's base point cloud.

[0109] By using image information, pose information, and the current scene base point cloud, the current object point cloud, current navigable point cloud, current obstacle point cloud, and current frontier point cloud are determined separately. Each object point cloud is responsible for a core concern: the current object point cloud focuses on the semantic target, the current navigable point cloud focuses on the feasible region, the current obstacle point cloud focuses on the safety boundary, and the current frontier point cloud focuses on the exploration direction. This separation of concerns at the architectural level avoids redundant and ineffective computations, reducing complexity from O(N^4) to O(4N). Furthermore, at the semantic level, it achieves intelligence through hierarchical understanding, security through multi-layered protection, real-time performance through incremental updates, interpretability through layered visualization, performance through space-for-time tradeoffs, and scalability through plug-and-play functionality.

[0110] In one possible implementation, Figure 5 This is a flowchart illustrating the process of obtaining the currently navigable point cloud in the robot mobility control method provided in this application embodiment, with reference to... Figure 5 As shown, S402 updates the robot's historical navigable point cloud at the previous time step based on the current scene base point cloud and the current object point cloud, to obtain the current navigable point cloud, including: S501. According to the preset navigable height range, extract at least one first base point from the current scene base point cloud.

[0111] Optionally, at least one first base point can be extracted from the current scene base point cloud according to a preset navigable height range.

[0112] For example, the navigable altitude range can be determined by a preset ground elevation Z. floor The navigable altitude range is obtained by comparing the altitude with a preset altitude error threshold δ. Specifically, the navigable altitude range can be [Z]. floor -δ,Z floor +δ].

[0113] S502. Based on the current object point cloud, select at least one second base point.

[0114] Optionally, the current object point cloud is filtered to select the point cloud corresponding to special passable structures in the current object point cloud to obtain at least one second base point.

[0115] In one example, the robot also stores navigable labels for objects of various categories. These navigable labels indicate whether an object of a particular category is a navigable, accessible structure. For example, the navigable label for an object of the "stairs" category is "navigable".

[0116] Specifically, from the current object point cloud, according to the navigable labels of various object categories, the corresponding point cloud is searched to obtain at least one second base point.

[0117] In another example, the current object point cloud can be identified by a pre-trained recognition model to determine whether each point cloud is a navigable point. If so, it is used as a second base point.

[0118] S503. Take each first base point and each second base point as a base increment point, and traverse each base increment point. For the current base increment point, determine whether to generate an interpolation point corresponding to the base increment point based on the position of the current base increment point. If so, perform interpolation processing on the base increment point to generate at least one interpolation point corresponding to the base increment point.

[0119] Optionally, each first base point and each second base point are treated as a base increment point, and each base increment point is traversed. For the current base increment point, a distance judgment is made based on the position corresponding to the current base increment point to determine whether to generate an interpolation point corresponding to the base increment point. If so, the base increment point is interpolated to generate at least one interpolation point corresponding to the base increment point.

[0120] For example, the current distance is calculated based on the position of the current basic increment point and the robot's current position. It is then determined whether the current distance is greater than a preset step size threshold. If so, the interpolation point corresponding to the basic increment point is determined, thereby solving the problem of limited sensor field of view and creating a continuous passable surface. In addition, under this processing method, even if a wall blocks part of the area, a passable path can be inferred, improving the success rate of navigation.

[0121] Specifically, regarding the current basic incremental point X i You can refer to the following formula (8) to determine whether the current distance is greater than the preset step size threshold: (8) Among them, X i As the current base increment point, P stand Let Δ be the robot's current location. step This is the preset step size threshold.

[0122] For example, the interpolation point P k It can be generated by referring to the following formula (9): (9) Among them, P k X is the interpolation point. i As the current base increment point, P stand Let Δ be the robot's current location.step This is the preset step size threshold.

[0123] Optionally, after obtaining all interpolation points, all interpolation points can be filtered to retain only those whose height is still within the navigable height range, thus obtaining the interpolation points corresponding to each basic incremental point.

[0124] S504. Based on the pre-trained voxel grid model, update each basic increment point and the interpolation point corresponding to each basic increment point to the historical navigable point cloud to obtain the current navigable point cloud.

[0125] Optionally, based on the pre-trained voxel mesh model, each basic increment point and the interpolation point corresponding to each basic increment point are updated to the historical navigable point cloud to obtain the current navigable point cloud, thereby reducing the point cloud density, improving the computational effect, eliminating measurement noise, and maintaining the integrity of geometric features.

[0126] For example, each basic increment point and its corresponding interpolation point are added to the historical navigable point cloud, and the historical navigable point cloud is uniformly downsampled using a pre-trained voxel grid model to obtain the current navigable point cloud. Specifically, the navigable point cloud Mnav can be obtained by referring to the following formula (10): (10) Where Mnav is a navigable point cloud, VoxelDownsample is a voxel mesh model, and r pcd This represents the resolution of the voxel mesh model.

[0127] By geometrically filtering the current scene's base point cloud, multiple first base points are obtained. Height constraints ensure complete coverage of ordinary ground areas. By semantically enhancing the current object point cloud, multiple second base points are obtained. Semantic information allows special structures such as stairs and ramps to be incorporated into the navigable space, solving the problem that a single method cannot cope with the complexity of real-world environments. Interpolation fills the gaps in sensor observations, constructing a continuous, navigable surface. Voxel downsampling reduces data density and noise while maintaining geometric features. The resulting navigable point cloud not only provides a continuous and uniform spatial foundation for path planning but also supports adaptive navigation strategies through the semantic information of point origins. This enables safe and efficient navigation decisions in complex dynamic environments.

[0128] In one possible implementation, Figure 6 This is a flowchart illustrating the process of determining the current obstacle point cloud in the robot movement control method provided in this application embodiment, with reference to... Figure 6 As shown, S403 above determines the current obstacle point cloud based on the current scene base point cloud, including: S601. According to the preset first height threshold, at least one candidate point cloud is extracted from the current scene base point cloud.

[0129] Optionally, at least one candidate point cloud is extracted from the current scene base point cloud according to a preset first height threshold. The first height threshold can be Z... floor +δ。 Z floor δ is the preset ground plane elevation, and δ is the preset height error threshold.

[0130] S602. Perform spatial clustering on each candidate point cloud to obtain multiple spatially connected point clusters.

[0131] Optionally, spatial clustering is performed on each candidate point cloud to obtain multiple spatially connected point clusters, thereby classifying spatially connected points into the same obstacle species. Spatial clustering includes density clustering based on Euclidean distance.

[0132] S603. Determine the obstacle type for each spatial connected point cluster, and extract at least one obstacle sub-point cloud from each spatial connected point cluster based on the obstacle type determination result.

[0133] Optionally, the obstacle type of each spatial connected point cluster is determined by a preset obstacle type identification condition, and the obstacle type determination result of each spatial connected point cluster is obtained. Based on the obstacle type determination result, at least one obstacle sub-point cloud is extracted from each spatial connected point cluster.

[0134] For example, the preset obstacle type recognition conditions may include: discrete obstacle recognition conditions and vertical obstacle recognition conditions.

[0135] The obstacle recognition conditions for discrete types are: minimum bounding box height H ≥ 0.1m and base area S ≤ 0.5m². Specifically, the minimum bounding box height and base area of ​​the spatially connected point clusters are calculated and judged. If the discrete obstacle recognition conditions are met, all point clouds in the corresponding spatially connected point cluster are treated as obstacle sub-point clouds, thereby ensuring that the robot avoids collisions when facing discrete obstacles.

[0136] The condition for recognizing vertical obstacles is that the normal vector is perpendicular to gravity. Specifically, the normal vector of the spatial connected point cluster is obtained through PCA analysis, and the angle between the normal vector and the direction of gravity is calculated. If the angle between the normal vector and the direction of gravity is less than a preset angle range, it is determined to be a vertical obstacle. The upper edge points of the corresponding spatial connected point cluster are extracted and used as obstacle sub-point clouds, thereby ensuring that the robot defines a passable boundary when facing vertical obstacles.

[0137] S604. Merge the point clouds of each obstacle and denoise them by voxel downsampling to generate the current obstacle point cloud.

[0138] Optionally, the union of each obstacle sub-point cloud is calculated, and voxel downsampling and denoising are performed using a pre-trained voxel mesh model to generate the current obstacle point cloud Mobs.

[0139] For example, the current obstacle point cloud Mobs can be generated by voxel downsampling and denoising using a pre-trained voxel mesh model, as described in the aforementioned formula (3).

[0140] By using a first height threshold, at least one candidate point cloud is extracted from the current scene base point cloud. By determining the obstacle type, geometric feature-driven intelligent classification can be achieved, enabling differentiated representation of different types of obstacles. This can significantly reduce the amount of point cloud data while ensuring navigation safety, and at the same time provide clearer obstacle boundary information for path planning.

[0141] The above provides an illustrative explanation of the process of determining the current semantic point cloud. It can be understood that after obtaining the current semantic point cloud, the environmental topology information of the current time step can be determined based on the current semantic point cloud. The following provides an illustrative explanation.

[0142] In one possible implementation, Figure 7 This is a schematic flowchart illustrating the process of obtaining environmental memory topology information in the robot mobility control method provided in this application embodiment, with reference to... Figure 7 As shown, in S103 above, the environmental memory topology information for the current time step is determined based on the current waypoint and the current semantic point cloud, including: S701. Create a new node corresponding to the current waypoint, and determine the attribute information of the new node based on the current waypoint and the current semantic point cloud.

[0143] Optionally, a new node corresponding to the current waypoint is created, and attribute information of the new node is generated based on the current waypoint and the current semantic point cloud.

[0144] In one example, the first node can be created when the robot reaches the first waypoint corresponding to the first time step.

[0145] In another example, when the robot reaches a waypoint that is not the first time step, a new node can be created based on the previous node. Specifically, the identifier of the new node is obtained by incrementing the identifier of the previous node.

[0146] For example, after creating a new node, the location information of the new node is determined based on the location of the current waypoint, the neighborhood object category information of the new node is determined based on the location of the current waypoint and the current object point cloud in the current semantic point cloud, the number of leading points of the new node is determined based on the current leading point cloud, and the room type of the new node is determined based on the current object point cloud.

[0147] S702. Add the newly added node and its attribute information to the robot's environmental memory topology information in the previous time step to obtain the environmental memory topology information in the current time step.

[0148] Optionally, the newly added node and its attribute information can be added to the robot's environmental memory topology information in the previous time step to obtain the environmental memory topology information in the current time step.

[0149] For example, when the robot reaches the first waypoint corresponding to the first time step, the newly added node and its attribute information can be used as the environmental memory topology information of the current time step; when the robot reaches a non-first waypoint corresponding to a non-first time step, the newly added node and its attribute information are added to the environmental memory topology information of the robot in the previous time step to obtain the environmental memory topology information of the current time step.

[0150] By using the current waypoint and the current semantic point cloud, the attribute information of the newly added node is determined, and the newly added node and its attribute information are added to the environmental memory topology information of the robot in the previous time step to obtain the environmental memory topology information of the current time step. This enables the robot to learn in exploration and reason in memory, solving the problems of target confusion and path redundancy caused by memory fragmentation in long-term navigation.

[0151] In one possible implementation, the attribute information of the newly added node includes: location information, neighbor object category information, room type, and number of frontier points. Figure 8 This is a flowchart illustrating the process of determining the attribute information of a newly added node in the robot movement control method provided in this application embodiment, with reference to... Figure 8 As shown, in step S701 above, the attribute information of the newly added node is determined based on the current waypoint and the current semantic point cloud, including: S801. Determine the location information based on the current waypoint.

[0152] Optionally, the position information of the new node can be determined based on the attitude information of the current waypoint.

[0153] For example, the position p in the attitude information of the current waypoint t Transform (x, y, z) to the global coordinate system and omit the z-axis to obtain n. k =(xk , y k ), which serves as the location information for the newly added node k.

[0154] S802. Determine the category information of neighboring objects based on the location information and the current object point cloud in the current semantic point cloud.

[0155] Optionally, with the position of the newly added node k as the center and the preset first perception radius r as the radius, the objects around the newly added node k are identified from the current object point cloud in the current semantic point cloud to obtain the neighborhood object category information.

[0156] For example, neighborhood object category information may include {chair, table}.

[0157] By using location information and the current object point cloud in the current semantic point cloud, the category information of neighboring objects can be determined, which can ensure that the semantic information is strongly correlated with the robot's current decision pose, avoid semantic ambiguity caused by global statistics, and realize atomic-level binding between spatial topological nodes and local semantic fields, thereby improving navigation accuracy.

[0158] S803. Obtain the current panoramic image corresponding to the current waypoint, and determine the room type based on the current panoramic image.

[0159] It is understandable that image information can be obtained at each waypoint. Therefore, the image information of multiple historical waypoints and the current waypoint can be fused to obtain the current panoramic image corresponding to the current waypoint.

[0160] Optionally, the current panoramic image is input into a pre-trained visual language model, which identifies the room type of the newly added node. Only the current panoramic image and the text prompt need to be input to identify the most matching semantic label, thereby improving the zero-shot open vocabulary generalization ability in the navigation process and also improving robustness.

[0161] S804. Determine the number of leading points based on the location information and the current leading point cloud in the current semantic point cloud.

[0162] Optionally, the number of leading points can be calculated based on location information, a preset second perception radius, and the current leading point cloud in the current semantic point cloud.

[0163] For example, the number of front points f k The following formula (11) can be used to obtain the result: (11) Among them, f k Let Mfro be the number of frontier points, and Mfro be the current frontier point cloud. topoThe second sensing radius is a preset value, which can be the same as the first sensing radius r, n k This provides the location information for the newly added node k.

[0164] By determining the number of leading points using location information and the current leading point cloud in the current semantic point cloud, discrete geometric features can be transformed into continuous and computable topological state variables, providing a new dimension of input for intelligent decision-making. This enables robots to have metacognitive abilities such as "knowing where they are, how much is unknown around them, and whether it is worthwhile to continue exploring," thereby improving navigation accuracy.

[0165] In one possible implementation, Figure 9 This is a flowchart illustrating the process of obtaining the environmental memory topology information at the current time step in the robot movement control method provided in this application embodiment, with reference to... Figure 9 As shown, S702 adds the newly added node and its attribute information to the robot's environmental memory topology information from the previous time step, thus obtaining the environmental memory topology information for the current time step, including: S901. Add the newly added node and its attribute information to the robot's environmental memory topology information in the previous time step to obtain the current candidate environmental memory topology information.

[0166] Optionally, the newly added node and its attribute information can be added to the robot's environmental memory topology information in the previous time step to obtain the current candidate environmental memory topology information.

[0167] S902. Determine whether to merge nodes based on the memory topology information of the current candidate environment.

[0168] Optionally, all nodes in the current candidate environment memory topology information, except for the newly added node, are traversed. For the current node that is traversed, it is determined whether to merge the newly added node and the current node in the current candidate environment memory topology information based on the distance between the newly added node and the current node and the relative relationship between the newly added node and the current node.

[0169] For example, if the spatial distance between the newly added node and the current node is less than a preset distance threshold, and there are no obstacles between the newly added node and the current node, then it is determined to merge the newly added node and the current node.

[0170] Specifically, it can be obtained by referring to the following formula (12): (12) Where, n i For the location information of the newly added node i, n j d represents the position information of the current node j. merge The distance threshold is set to a preset value, Mobs represents the current obstacle point cloud, and Z represents the distance threshold. floorZ is the preset ground level elevation. ceiling For the preset ceiling elevation, z w For point cloud X w The height value, L (n i , n j ) represents the straight-line path between the newly added node i and the current node j.

[0171] By measuring the distance between the newly added node and the current node, as well as their relative relationship, we can determine whether to merge the newly added node and the current node in the current candidate environment's memory topology information. This method can quickly filter out obviously irrelevant distant nodes, ensuring computational efficiency, and can also completely avoid erroneous merging of geometrically adjacent but semantically conflicting nodes, making the merging result cognitively reasonable and task-safe.

[0172] S903. If so, merge the nodes of the current candidate environment memory topology information to obtain the environment memory topology information of the current time step.

[0173] Optionally, if it is determined to merge the current node and the newly added node, then the current node and the newly added node are merged according to the attribute information of the current node and the attribute information of the newly added node to obtain the merged node and the attribute information of the merged node. Then, the current node, the newly added node, the attribute information of the current node and the attribute information of the newly added node are deleted from the current candidate environment memory topology information, and the merged node and the attribute information of the merged node are added.

[0174] For example, the identifiers of the current node and the newly added node are compared, and the identifier of the node that was created earlier is retained as the identifier of the merged node.

[0175] For example, the two-dimensional centroids of the current node and the newly added node are calculated based on the position information of the current node and the position information of the newly added node, and used as the position information of the merged node.

[0176] For example, based on the location information of the merged node, the neighbor object category information of the merged node is recalculated with reference to the aforementioned S802.

[0177] For example, the room type of the earlier created node is retained as the room type of the merged node.

[0178] For example, based on the location information of the merged nodes, the number of leading edge points of the merged nodes is recalculated with reference to the aforementioned S804.

[0179] S904. If not, use the current candidate environment memory topology information as the environment memory topology information of the current time step.

[0180] Optionally, if it is determined not to merge, the current candidate environment memory topology information is used as the environment memory topology information for the current time step.

[0181] By merging nodes of the current candidate environment memory topology information, the environment memory topology information of the current time step is obtained. Through spatial distance-driven, attribute-aware, and configurable topology compression, it is possible to avoid memory explosion while ensuring that key semantic information is not lost.

[0182] In one possible implementation, determining the next waypoint of the current waypoint based on the environmental memory topology information in S104 above includes: At the current time step, acquire the robot's current task information, the current panoramic image corresponding to the current waypoint, and the historical navigation information corresponding to the current waypoint; input the current task information, the current panoramic image, the historical navigation information, and the pre-constructed environmental memory topology information into the pre-trained VLM model, and the VLM model predicts the node recommendation information, preferred motion direction, and target state information; based on the node recommendation information, preferred motion direction, and target state information, determine the next waypoint from the currently navigable point cloud in the current semantic point cloud.

[0183] The current task information refers to the task the robot needs to perform. This information indicates the target object. The current panoramic image corresponding to the current waypoint is a panoramic image obtained by fusing images acquired from the current waypoint and all historical waypoints preceding it.

[0184] The historical navigation information corresponding to the current waypoint refers to the navigation information corresponding to all historical waypoints before the current waypoint. The historical navigation information includes at least the historical movement node sequence.

[0185] Optionally, the historical navigation information corresponding to the current waypoint also includes: a sequence of historical waypoint semantic labels and a sequence of historical task completion statuses. The sequence of historical waypoint semantic labels includes the room type associated with each historical waypoint, and the sequence of historical task completion statuses includes the completion status of similar or related tasks in the past at each historical waypoint.

[0186] Optionally, the current task information, current panoramic image, historical navigation information, environmental memory topology information, and preset prompt words are input into a pre-trained visual language model, which then performs reasoning to obtain node recommendation information, preferred movement direction, and target state information.

[0187] Among them, the target status information is used to indicate the completion status of the current task information in the current panoramic image.

[0188] Among them, the current task information is used to provide high-level semantic target constraints, the current panoramic image is used to provide omnidirectional, unobstructed instantaneous environmental observation, the historical navigation information is used to provide spatiotemporal experience memory and task execution constraints, and the environmental memory topology information is used to provide structured, reasonable long-term environmental knowledge.

[0189] The node recommendation information includes the identifier k' of the nodes prioritized for exploration by the visual language model, and the preferred motion direction d. VLM This refers to the direction that the visual language model determines may contain the target object.

[0190] The target status information indicates the completion status of the current task information in the current panoramic image. The target status information includes: completed and incomplete.

[0191] Specifically, the target state information is "complete," meaning the target object corresponding to the current task information exists in the current panoramic image. The target state information is "incomplete," meaning the target object corresponding to the current task information does not exist in the current panoramic image. For example, the target state information g... found ∈{0,1}.

[0192] Optionally, while outputting target state information, the visual language model can also output target hit information corresponding to the target state information. Target hit information includes: hit and miss. A hit means that the category of the target object is determined to exist by the visual language model, that is, the target object belongs to the preset object category directory. A miss means that the category of the target object is determined not to exist by the visual language model, that is, the target object does not belong to the preset object category directory.

[0193] Optionally, based on the target status information, the waypoint determination mode for the next waypoint is determined, and based on the node recommendation information, the waypoint determination mode, and the preferred direction of movement, the next waypoint for the current waypoint is determined.

[0194] The waypoint determination modes include exploration mode and non-exploration mode.

[0195] In one example, when the target status information is "completed", the waypoint determination mode is determined to be non-exploration mode; when the target status information is "incomplete", the waypoint determination mode is determined to be exploration mode.

[0196] In another example, when the target status information is "completed" and the target hit information is "hit", the waypoint determination mode is determined to be non-exploration mode; when the target status information is "incomplete" and the target hit information is "missed", the waypoint determination mode is determined to be exploration mode.

[0197] Optionally, each point in the current navigable point cloud is taken as a candidate waypoint, and the target actionable parameters of each candidate waypoint are calculated based on node recommendation information, waypoint determination mode and preferred direction of movement; based on the target actionable parameters of each candidate waypoint, the next waypoint of the current waypoint is determined from each candidate waypoint.

[0198] By using target state information, the waypoint determination pattern for the next waypoint is determined, and based on the waypoint determination pattern, the next waypoint is determined. This upgrades the starting conditions for navigation decisions from physical quantities such as time, number of steps, and spatial distance to target state information, a high-level cognitive state that represents the progress of the semantic task. This makes the navigation process strongly bound to the semantics of the task, and enables the navigation decision-making process to adaptively switch between efficiently expanding cognitive boundaries and achieving high-precision arrival, thereby improving the adaptive capability of the navigation process.

[0199] The following is an illustrative explanation of the process for determining the target actionable parameters for each candidate waypoint. Before calculating the target actionable parameters for each candidate waypoint, it should be understood that all candidate waypoints constitute a candidate point cloud set. In the candidate point cloud set, for any candidate waypoint, the target point cloud in multiple dimensions can be determined. When calculating the target actionable parameters for that candidate waypoint, the sub-target actionable parameters of each dimension can be calculated based on the target point cloud in each dimension, thereby calculating the target actionable parameters for the candidate waypoint.

[0200] For any candidate waypoint p i And the target point cloud S in any dimension, and the actionable parameters of the candidate waypoints as sub-targets in that dimension. This can be achieved by referring to the following formulas (13)-(16): (13) (14) (15) (16) in, q is a preset error coefficient, and q is any point in the target point cloud.

[0201] Based on this, the following explains the target action parameters for each candidate waypoint calculated.

[0202] In one possible implementation, the above steps calculate the target actionable parameters for each candidate waypoint based on node recommendation information, waypoint determination patterns, and preferred movement directions, including: If the waypoint determination mode is exploration mode, determine the directional guidance parameters of the candidate waypoints based on the preferred direction of movement; determine the node guidance parameters of the candidate waypoints based on the node recommendation information; determine the frontier exploration parameters of the candidate waypoints based on the current frontier point cloud in the current semantic point cloud; determine the historical avoidance parameters of the candidate waypoints based on historical navigation information; calculate the sum of the directional guidance parameters, node guidance parameters, frontier exploration parameters, and historical avoidance parameters as the target actionable parameters of the candidate waypoints.

[0203] Optionally, if the waypoint determination mode is exploration mode, the directional guidance parameters of the candidate waypoints are calculated based on the preferred direction of movement. These directional guidance parameters are the actionable parameters of the sub-targets in the directional dimension.

[0204] For example, according to the preferred direction of motion d VLM Determine the candidate point cloud set along the preferred motion direction d. VLM The point set is used as the target point cloud S in the direction dimension. dir And the candidate waypoint p is calculated by referring to the aforementioned formulas (13)-(16). i Directional guidance parameters .

[0205] By using directional guidance parameters, the visual reasoning ability of a visual language model can be used to identify possible directions to the target during navigation.

[0206] Optionally, if the waypoint determination mode is exploration mode, the node guidance parameters of the candidate waypoints are calculated based on the node identifiers in the node recommendation information. These node guidance parameters are the actionable parameters of the sub-targets at the node level.

[0207] For example, based on the node identifier k' in the node recommendation information, the node corresponding to node N is found from the pre-constructed current frontier point cloud. k The set of associated frontier points serves as the target point cloud S in the node dimension. node And the candidate waypoint p is calculated by referring to the aforementioned formulas (13)-(16). i Node boot parameters .

[0208] By using node-guided parameters, high-potential areas can be prioritized for exploration during navigation by leveraging the topological reasoning capabilities of the visual language model.

[0209] Optionally, if the waypoint determination mode is exploration mode, the front exploration parameters of the candidate waypoints are calculated based on the current front edge point cloud in the pre-constructed current semantic point cloud. These front edge exploration parameters are the actionable parameters of the sub-targets in the front edge dimension.

[0210] For example, the current leading edge point cloud Mfro is obtained from the pre-constructed point cloud as the target point cloud in the leading edge dimension, and the candidate waypoint p is calculated according to the aforementioned formulas (13)-(16). i Frontier exploration parameters Among them, the current cutting-edge point cloud Mfro can be implemented with reference to S905.

[0211] By exploring cutting-edge parameters, a wide range of exploration pressures can be provided during navigation, supplementing the targeted guidance provided by the visual language model.

[0212] Optionally, if the waypoint determination mode is exploration mode, historical avoidance parameters of the candidate waypoints are calculated based on historical navigation information. These historical avoidance parameters are the actionable parameters of the sub-targets in the historical trajectory dimension.

[0213] For example, based on historical navigation information, a set of points consisting of historical trajectory points is determined as the target point cloud S in the historical trajectory dimension. hist And the candidate waypoint p is calculated according to the aforementioned formulas (13)-(16). i Historical avoidance parameters .

[0214] By using historical avoidance parameters, it is possible to avoid repeatedly visiting already explored areas during navigation.

[0215] For example, the sum of the direction guidance parameters, node guidance parameters, frontier exploration parameters, and historical avoidance parameters of the candidate waypoint is calculated as the target actionable parameters of the candidate waypoint.

[0216] In exploration mode, by using directional guidance parameters, node guidance parameters, frontier exploration parameters, and historical avoidance parameters, the target action parameters of candidate waypoints are calculated. This allows for the expansion of cognitive boundaries while achieving a balance between high-level semantics and low-level geometry, avoiding over-reliance on a single information source, and guiding the robot to conduct multi-dimensional exploration. This balances exploration efficiency, target proximity, and safety performance. Furthermore, it enhances the robustness of the exploration process.

[0217] The above describes the process of determining the target action parameters of candidate waypoints when the waypoint determination mode is exploration mode. The following describes the process of determining the target action parameters of candidate waypoints when the waypoint determination mode is non-exploration mode.

[0218] In one possible implementation, the above steps calculate the target actionable parameters for each candidate waypoint based on node recommendation information, waypoint determination patterns, and preferred movement directions, including: If the waypoint determination mode is non-exploration mode, determine the directional guidance parameters of the candidate waypoints based on the preferred direction of movement; determine the neighboring guidance parameters of the candidate waypoints based on the current object point cloud; calculate the sum of the directional guidance parameters and the neighboring guidance parameters as the target action parameters of the candidate waypoints.

[0219] Optionally, if the waypoint determination mode is non-exploration mode, the directional guidance parameters of the candidate waypoints are calculated based on the preferred direction of movement. These directional guidance parameters are the actionable parameters of the sub-targets in the directional dimension.

[0220] For example, according to the preferred direction of motion d VLM Determine the candidate point cloud set along the preferred motion direction d. VLM The point set is used as the target point cloud S in the direction dimension. dir And the candidate waypoint p is calculated by referring to the aforementioned formulas (13)-(16). i Directional guidance parameters .

[0221] By using directional guidance parameters, the visual reasoning ability of a visual language model can be used to identify possible directions to the target during navigation.

[0222] In one example, based on the current object point cloud, a set of point clouds with the same category as the target object is determined as the target point cloud S. sem And the candidate waypoint p is calculated by referring to the aforementioned formulas (13)-(16). i Proximity guidance parameters for candidate waypoints .

[0223] By using proximity guidance parameters, the robot can be guided to precisely approach the identified target instance during navigation.

[0224] Optionally, the sum of the directional guidance parameters and the neighboring guidance parameters is calculated as the target actionable parameters for the candidate waypoints.

[0225] In non-exploration mode, actionable parameters for candidate waypoints are calculated using directional and proximity guidance parameters. This prevents the robot from exploring irrelevant areas during navigation once the target has been visually confirmed. Simultaneously, it guides the agent closer to the confirmed target instance, achieving precise localization. It also reduces computational complexity and accelerates waypoint selection. Furthermore, retaining directional guidance parameters ensures a smooth transition from exploration to target acquisition, maintaining navigation continuity and preventing abrupt directional changes.

[0226] In one possible implementation, the above steps, based on the target actionable parameters of each candidate waypoint, determine the next waypoint from each candidate waypoint, including: Determine the hazard score of each candidate waypoint, and based on the hazard score of each candidate waypoint, select at least one optional waypoint from each candidate waypoint; obtain the maximum value of the target actionable parameter in each optional waypoint, and obtain the target waypoint corresponding to the maximum value, and use the target waypoint as the next waypoint of the current waypoint.

[0227] In one example, all obstacle points in the current panoramic image can be obtained based on the current obstacle point cloud (Mobs).

[0228] Alternatively, it can be based on candidate waypoints p i And all obstacle points, calculate the shortest distance of each candidate waypoint relative to all obstacle points. Specifically, it can be implemented with reference to the aforementioned formula (14), thereby capturing the instantaneous collision risk by the shortest distance of each candidate waypoint relative to all obstacle points.

[0229] Optionally, the maximum and minimum distances from all candidate waypoints to obstacles can be calculated based on all candidate waypoints and all obstacle points. Specifically, this can be achieved by referring to the aforementioned formulas (15)-(16), thereby characterizing the local spatial redundancy through the maximum distance from all candidate points to obstacles, and revealing the spatial constraint strength through the minimum distance from all candidate points to obstacles.

[0230] Optionally, the hazard score of each candidate waypoint can be calculated based on the shortest, maximum, and minimum distances relative to all obstacle points. Specifically, this can be achieved using the following formula (13).

[0231] By calculating the shortest, maximum, and minimum distances of each candidate waypoint relative to all obstacle points, the hazard score of each candidate waypoint is obtained. Through the fusion of three-dimensional distance features, a quantitative description of the local risk field topology of the candidate waypoints can be achieved.

[0232] Optionally, the maximum value of the target actionable parameter among the optional waypoints is obtained, and the target waypoint corresponding to the maximum value is obtained, and the target waypoint is used as the next waypoint of the current waypoint.

[0233] By identifying candidate waypoints and calculating their target actionable parameters based on node recommendation information, waypoint determination patterns, and preferred movement directions, the next waypoint is determined from among these candidate waypoints according to these parameters. This method enables the calculation of target actionable parameters through multi-source heterogeneous parameter coupling, ensuring that even if a single signal fails, the remaining dimensions still provide strong constraints. This avoids waypoint decision collapse caused by distortion in single-frame visual observations or missing local geometric information, significantly improving navigation robustness in long-term, highly dynamic indoor environments. Furthermore, it achieves spatial alignment and causal weighting of four types of information: natural language task intent, panoramic visual observation, structured environmental topology, and spatiotemporal navigation experience, ensuring that navigation decisions possess both high-level semantic rationality and low-level motion feasibility.

[0234] This application also provides a robot, such as... Figure 10 As shown, Figure 10 The schematic diagram of the robot structure provided in this application embodiment includes: a processor 1001 and a memory 1002, and optionally, a bus 1003. The memory 1002 stores machine-readable instructions executable by the processor 1001. When the robot is running, the processor 1001 and the memory 1002 communicate through the bus 1003, and the processor 1001 executes the machine-readable instructions to perform the steps of the above-described robot movement control method.

[0235] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot movement control method described above.

[0236] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A robot movement control method, characterized in that, include: When the robot reaches the current waypoint corresponding to the current time step, the robot's image information and attitude information at the current waypoint are obtained. Based on the image information and the pose information, a current semantic point cloud is determined, wherein the current semantic point cloud includes sub-point clouds of multiple dimensions that are spatially registered. Based on the current waypoint and the current semantic point cloud, the environmental memory topology information of the current time step is determined. The environmental memory topology information includes multiple nodes and attribute information of each node. Each node represents a spatial point in the environment in which the robot is located. The attribute information of each node is used to indicate the semantic features of the spatial point. Based on the environmental memory topology information, the next waypoint of the current waypoint is determined, and the robot is controlled to move to the next waypoint.

2. The robot movement control method according to claim 1, characterized in that, The current semantic point cloud includes: the current scene base point cloud and the current object point cloud in multiple dimensions; Determining the current semantic point cloud based on the image information and the pose information includes: Based on the image information and the posture information, the robot incrementally updates the historical scene base point cloud of the previous time step to obtain the current scene base point cloud. The point clouds of each current object are determined based on the image information, the pose information, and the current scene base point cloud.

3. The robot movement control method according to claim 2, characterized in that, The image information includes: RGB image and depth image; The step of incrementally updating the robot's historical scene base point cloud at the previous time step based on the image information and the pose information to obtain the current scene base point cloud includes: Based on the depth image and the robot's camera intrinsic parameter matrix, determine multiple first spatial points corresponding to the depth image; Based on the RGB image, add color attributes to each of the first spatial points; Based on the attitude information, each of the first spatial points is transformed to the global coordinate system to obtain multiple current spatial points; Determine the first incremental point cloud of each current spatial point and the historical scene base point cloud; Based on the pre-trained voxel mesh model, the first incremental point cloud is updated to the historical scene base point cloud to obtain the current scene base point cloud.

4. The robot movement control method according to claim 2, characterized in that, The current object point cloud in multiple dimensions includes: current object point cloud, current navigable point cloud, current obstacle point cloud, and current front edge point cloud; The step of determining each current object point cloud based on the image information, the pose information, and the current scene base point cloud includes: Based on the image information and the posture information, the robot incrementally updates the historical object point cloud of the previous time step to obtain the current object point cloud. Based on the current scene base point cloud and the current object point cloud, the robot's historical navigable point cloud at the previous time step is updated to obtain the current navigable point cloud; The current obstacle point cloud is determined based on the current scene base point cloud; The current front-line point cloud is determined based on the current scene base point cloud, the current navigable point cloud, and the current obstacle point cloud.

5. The robot movement control method according to claim 4, characterized in that, The step of updating the robot's historical navigable point cloud at the previous time step based on the current scene base point cloud and the current object point cloud to obtain the current navigable point cloud includes: According to the preset navigable altitude range, at least one first base point is extracted from the current scene base point cloud; Based on the current object point cloud, at least one second base point is obtained by filtering. Each of the first base points and each of the second base points is taken as a base increment point. Each base increment point is traversed. For the current base increment point, it is determined whether to generate an interpolation point corresponding to the current base increment point based on the position of the current base increment point. If so, the base increment point is interpolated to generate at least one interpolation point corresponding to the base increment point. Based on the pre-trained voxel grid model, each of the basic increment points and the interpolation points corresponding to each of the basic increment points are updated to the historical navigable point cloud to obtain the current navigable point cloud.

6. The robot movement control method according to claim 4, characterized in that, Determining the current obstacle point cloud based on the current scene base point cloud includes: At least one candidate point cloud is extracted from the current scene base point cloud according to a preset first height threshold. Spatial clustering is performed on each of the candidate point clouds to obtain multiple spatially connected point clusters; Obstacle types are determined for each of the spatial connected point clusters, and at least one obstacle sub-point cloud is extracted from each of the spatial connected point clusters based on the obstacle type determination results; The obstacle sub-point clouds are merged and denoised by voxel downsampling to generate the current obstacle point cloud.

7. The robot movement control method according to claim 1, characterized in that, The step of determining the environmental memory topology information for the current time step based on the current waypoint and the current semantic point cloud includes: Create a new node corresponding to the current waypoint, and determine the attribute information of the new node based on the current waypoint and the current semantic point cloud; The newly added node and its attribute information are added to the robot's environmental memory topology information in the previous time step to obtain the environmental memory topology information in the current time step.

8. The robot movement control method according to claim 7, characterized in that, The attribute information of the newly added node includes: location information, neighbor object category information, room type, and number of frontier points; The step of determining the attribute information of the newly added node based on the current waypoint and the current semantic point cloud includes: The location information is determined based on the current waypoint; Based on the location information and the current object point cloud in the current semantic point cloud, the neighborhood object category information is determined; Obtain the current panoramic image corresponding to the current waypoint, and determine the room type based on the current panoramic image; The number of leading edge points is determined based on the location information and the current leading edge point cloud in the current semantic point cloud.

9. The robot movement control method according to claim 7, characterized in that, The step of adding the newly added node and its attribute information to the robot's environmental memory topology information in the previous time step to obtain the environmental memory topology information in the current time step includes: The newly added node and its attribute information are added to the robot's environmental memory topology information in the previous time step to obtain the current candidate environmental memory topology information; Determine whether to merge nodes in the current candidate environment memory topology information; If so, the nodes of the current candidate environment memory topology information are merged to obtain the environment memory topology information of the current time step; If not, the current candidate environment memory topology information shall be used as the environment memory topology information of the current time step.

10. A robot, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the robot is running, are executed by the processor to perform the steps of the robot motion control method as described in any one of claims 1 to 9.

Images