Method, apparatus and system for task execution of robot

Abstract

The application provides a task execution method, device and system of a robot. A target space coordinate and a natural language instruction transmitted by an augmented reality device are received, wherein the natural language instruction is a task instruction issued by a target object wearing the augmented reality device and requiring the robot to execute, and the target space coordinate is a space position of a target object based on a gaze of the target object in a coordinate system of the augmented reality device when the augmented reality device receives the natural language instruction, and is calculated through coordinate conversion. The robot is controlled to move to a target position according to the target space coordinate, and a task corresponding to the natural language instruction is executed at the target position. The problem that a service robot in the prior art cannot accurately position a target object position corresponding to a voice instruction, resulting in low execution task accuracy, is solved, the training threshold and deployment cost of the service robot are reduced, and an efficient and low-cost solution is provided for service intelligent equipment in a family and office scenario.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a robot task execution method, a robot task execution device, and a robot task execution system. Background Technology

[0002] Collaborative robots and humanoid robots, as important products in the field of high-end equipment manufacturing, are widely used in industrial and household services. However, they share common shortcomings in artificial intelligence human-computer interaction, such as over-reliance on speech semantic parsing and an inability to truly understand the physical task objectives, physical object positions, and spatial constraints corresponding to speech commands. In other words, they lack the ability to accurately perceive the spatial coordinates of the physical world.

[0003] In existing technologies, collaborative robots and humanoid robots require massive amounts of pre-trained data and powerful artificial intelligence learning models to collect sensor fusion information in real time in order to complete tasks. This makes it difficult for robots to quickly and accurately locate targets in unfamiliar environments without prior training. This limits the deep application of next-generation intelligent interaction technologies in the field of service robots and makes it difficult to meet the intelligent and personalized service needs of modern times. Summary of the Invention

[0004] The main objective of this application is to provide a robot task execution method, a robot task execution device, and a robot task execution system, so as to at least solve the problem in the prior art that service robots cannot accurately locate the position of the target object corresponding to the voice command, resulting in low accuracy in task execution.

[0005] To achieve the above objectives, according to one aspect of this application, a robot task execution method is provided, comprising: receiving target spatial coordinates and natural language instructions transmitted by an augmented reality device, wherein the natural language instructions are task instructions issued by a target object wearing the augmented reality device and requiring the robot to perform them, and the target spatial coordinates are calculated by the augmented reality device through coordinate transformation based on the spatial position of the target object being viewed by the target object in the coordinate system of the augmented reality device when the natural language instructions are received; controlling the robot to move to a target position according to the target spatial coordinates, and performing the task corresponding to the natural language instructions at the target position.

[0006] Optionally, performing a task corresponding to the natural language instruction at the target location includes: identifying the target object at the target location and determining the semantic category of the target object; if the semantic category belongs to a preset common item category, storing the spatial coordinates of the target location and the semantic name of the target object in a storage device, wherein the semantic name is a category label determined by visual recognition of the target object; if the augmented reality device is not connected, and if the natural language instruction including the semantic name is received again, retrieving the spatial coordinates corresponding to the semantic name from the storage device, and navigating to the target location based on the spatial coordinates to perform the task.

[0007] Optionally, the calculation process of the target spatial coordinates includes: determining the relative coordinates of the target object in the coordinate system of the augmented reality device, and the pose matrix of the augmented reality device relative to the base coordinate system of the robot, wherein the pose matrix is ​​used to characterize the posture and position information of the augmented reality device in the base coordinate system of the robot; and calculating the target spatial coordinates of the target object in the base coordinate system of the robot based on the relative coordinates and the pose matrix using a preset coordinate transformation formula.

[0008] Optionally, determining the relative coordinates of the target object in the coordinate system of the augmented reality device includes: acquiring pupil position information of the target object and calculating a gaze vector based on the pupil position information; constructing a scene point cloud based on a scene depth map built into the augmented reality device, and calculating the intersection of the gaze vector and the scene point cloud, and determining the pixel region where the target object is located based on the intersection; determining the center pixel coordinates of the target object based on image features within the pixel region; and converting the center pixel coordinates into the relative coordinates of the target object in the coordinate system of the augmented reality device through inverse projection transformation, based on the current pose data of the augmented reality device and the depth value corresponding to the center pixel coordinates in the scene depth map.

[0009] Optionally, determining the pose matrix of the augmented reality device relative to the robot's base coordinate system includes: obtaining a first pose matrix of the augmented reality device relative to the world coordinate system and a second pose matrix of the robot relative to the world coordinate system; inverting the second pose matrix to obtain an inverted second pose matrix; and performing matrix multiplication between the inverted second pose matrix and the first pose matrix to obtain the pose matrix of the augmented reality device relative to the robot's base coordinate system.

[0010] Optionally, controlling the robot to move to the target location based on the target spatial coordinates includes: acquiring a global map of the environment where the target location is located, wherein the global map is pre-constructed using simultaneous localization and mapping technology; performing global path planning using a global path planning algorithm with the target spatial coordinates as the endpoint in the global map to obtain a target path; and controlling the robot to move along the target path until it reaches the target location corresponding to the target spatial coordinates.

[0011] Optionally, performing a task corresponding to the natural language instruction at the target location includes: after the robot reaches the target location, acquiring a target image, wherein the target image is an image of the area where the target location is located; extracting feature points of the target object from the target image, and calibrating the target spatial coordinates based on the feature points to obtain calibrated target spatial coordinates; and performing a task corresponding to the natural language instruction based on the calibrated target spatial coordinates.

[0012] Optionally, executing the task corresponding to the natural language instruction based on the calibrated target spatial coordinates includes: obtaining the operation pose information or target operation box of the target object based on the calibrated target spatial coordinates, and determining a collision-free operation path based on the operation pose information or the target operation box, wherein the operation pose information is used to characterize the position and posture reached by the robot's end effector, and the target operation box is used to characterize the operation position, posture, and range; controlling the robot's end effector to move along the collision-free operation path to a pre-operation pose, wherein the pre-operation pose is the spatial pose of the end effector relative to the surface of the target object, along a preset approach direction and maintaining a preset safety distance; selecting a corresponding operation strategy according to the shape of the target object, and controlling the end effector to perform the target operation based on the operation strategy to complete the task corresponding to the natural language instruction.

[0013] According to another aspect of this application, a task execution device for a robot is provided, comprising: a receiving unit for receiving target spatial coordinates and natural language instructions transmitted by an augmented reality device, wherein the natural language instructions are task instructions issued by a target object wearing the augmented reality device and requiring the robot to perform them, and the target spatial coordinates are calculated by the augmented reality device through coordinate transformation based on the spatial position of the target object being viewed by the target object in the coordinate system of the augmented reality device when the natural language instructions are received; and an execution unit for controlling the robot to move to a target position according to the target spatial coordinates and performing a task corresponding to the natural language instructions at the target position.

[0014] According to another aspect of this application, a robot task execution system is provided, comprising: an augmented reality device; a robot, the robot being communicatively connected to the augmented reality device, the robot including a controller for executing any of the robot task execution methods described above.

[0015] The technical solution of this application receives target spatial coordinates and natural language commands transmitted by an augmented reality device. The natural language commands are task instructions issued by the target object wearing the augmented reality device, requiring the robot to execute them. The target spatial coordinates are calculated by the augmented reality device through coordinate transformation based on the spatial position of the target object being viewed by the target object in the augmented reality device's coordinate system upon receiving the natural language commands. The robot is then controlled to move to the target position based on the target spatial coordinates and execute the task corresponding to the natural language command at the target position. This solution, by converting the target being viewed by the human eye into precise spatial coordinates, achieves accurate binding between voice commands and physical locations, solving the problem of fuzzy positioning caused by the robot's lack of spatial awareness, and improving the accuracy of task execution. This addresses the problem in existing technologies where service robots cannot accurately locate the target object corresponding to the voice command, resulting in low task execution accuracy. It also lowers the training threshold and deployment cost of service robots, providing an efficient and low-cost solution for service-oriented intelligent equipment in home and office scenarios. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A flowchart illustrating a task execution method for a robot according to an embodiment of this application is shown.

[0018] Figure 2 A schematic diagram of a robot task execution system provided according to an embodiment of this application is shown;

[0019] Figure 3 A schematic diagram of the coordinate system layout for a robot task execution method according to an embodiment of this application is shown;

[0020] Figure 4 A flowchart illustrating a specific robot task execution method according to an embodiment of this application is shown.

[0021] Figure 5 A schematic diagram of a service robot navigation process according to an embodiment of this application is shown;

[0022] Figure 6 A structural block diagram of a robot task execution device provided according to an embodiment of this application is shown. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] As described in the background section, existing robots cannot truly understand the physical task objectives, physical object positions, and spatial constraints corresponding to voice commands, and cannot quickly and accurately locate targets in unfamiliar environments without training. To address the problem that service robots in the prior art cannot accurately locate the target object position corresponding to voice commands, resulting in low accuracy in task execution, embodiments of this application provide a robot task execution method, a robot task execution device, and a robot task execution system.

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] Figure 1 This is a flowchart illustrating a robot task execution method according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0029] Step S101: Receive target spatial coordinates and natural language instructions transmitted by the augmented reality device. The natural language instructions are task instructions issued by the target object when wearing the augmented reality device, which require the robot to perform. The target spatial coordinates are calculated by the augmented reality device through coordinate transformation based on the spatial position of the target object being viewed by the target object in the coordinate system of the augmented reality device when the natural language instructions are received.

[0030] Specifically, the robot task execution method in this embodiment is mainly applied to a system consisting of an augmented reality device, a service robot, and a communication network between the two. The augmented reality device is worn by the target object (i.e., the user). The robot in this embodiment can be a service robot or a humanoid robot. This embodiment uses a service robot as an example and augmented reality glasses (AR glasses) as an example for explanation. Specifically, the user wears AR glasses with eye tracking and visual capture functions and issues natural language commands. After receiving the user's voice input, the AR glasses process it and extract the natural language command including the task intent, such as: "Give me the cup on the table." At the same time, the AR glasses capture the user's gaze direction and determine the target object currently being looked at based on the gaze direction. It calculates the three-dimensional spatial position of the target object in the AR glasses' own coordinate system and calculates the target spatial coordinates in the robot's base coordinate system through coordinate transformation. Then, the target spatial coordinates and the natural language command are sent to the service robot, which receives the target spatial coordinates and the natural language command.

[0031] Step 102: Control the robot to move to the target position according to the target spatial coordinates, and execute the task corresponding to the natural language command at the target position.

[0032] Specifically, after receiving the target spatial coordinates, they are converted into target pose parameters in the robot's base coordinate system. The robot is then driven to move, and its spatial position and attitude are adjusted through kinematic calculations, enabling the robot to accurately reach the physical position indicated by the target spatial coordinates, thus providing a stable operational reference for subsequent task execution.

[0033] The system analyzes the action semantics in natural language commands to identify the corresponding task type and operation parameters. If the task involves robotic arm operation, it drives the robotic arm's end effector to move from a standby position to a working position near the target location. Subsequently, based on the analyzed task type, it controls the robotic arm's end effector to perform specific physical actions, such as controlling the gripper to close to grasp the target object or controlling the robotic arm to apply thrust to move the object. If the task only involves robot body response, the robot directly executes preset voice feedback, light prompts, or posture display actions to complete the entire task flow.

[0034] This embodiment transforms the target gazed at by the human eye into precise spatial coordinates, achieving accurate binding between voice commands and physical locations. This solves the problem of ambiguous positioning caused by the robot's lack of spatial awareness and improves the accuracy of task execution.

[0035] In some embodiments of this application, performing a task corresponding to the natural language instruction at the target location includes: identifying the target object at the target location and determining the semantic category of the target object; if the semantic category belongs to a preset common item category, storing the spatial coordinates of the target location and the semantic name of the target object in a storage device, wherein the semantic name is a category label determined by visual recognition of the target object; if the augmented reality device is not connected, and a natural language instruction including the semantic name is received, retrieving the spatial coordinates corresponding to the semantic name from the storage device, and navigating to the target location based on the spatial coordinates to perform the task.

[0036] Specifically, once the robot moves to the target location, it first acquires images and performs feature analysis on the target object using its onboard visual sensors (such as an RGB-D camera or a depth camera). The control unit then processes the acquired images using a built-in object recognition algorithm (such as a convolutional neural network) to identify the specific shape of the target object and determine its corresponding semantic category (e.g., "water cup," "file," or "coffee cup"). Simultaneously, it establishes a mapping relationship between the precise spatial coordinates of the target object in the current coordinate system and its semantic name.

[0037] If the identified semantic category belongs to a pre-set "frequently used item categories" database (i.e., a storage device, which can be pre-set or dynamically updated based on the user's historical usage frequency), the fixed-location memory mechanism is triggered. The spatial coordinates of the target object (including X, Y, Z positions and attitude information such as yaw angle) and the semantic name determined through visual recognition (i.e., category label, such as "blue water cup") are recorded and written into the storage device. This process achieves basic data accumulation for "one-time guidance and multiple uses," enabling the robot to remember the fixed storage locations of frequently used items.

[0038] During subsequent task execution, if the augmented reality device is not connected (e.g., the user is not wearing AR glasses or the AR function is not enabled), and the user issues a natural language command containing a stored semantic name (e.g., the user directly says, "Go get that water cup"), the local retrieval function will be automatically activated. The system will search the storage device for spatial coordinate records matching that semantic name. Once the retrieval is successful, the robot can autonomously plan its movement trajectory and navigate to the target location based on the stored historical spatial coordinates, without relying on real-time guidance from an external AR device, and then perform the corresponding grasping or manipulation tasks.

[0039] This embodiment achieves automatic memorization and offline reuse of frequently used item locations. When a target object is identified as a frequently used item, the robot automatically stores its spatial coordinates and semantic name in the database. Subsequently, even if the AR device is not connected, the robot can directly retrieve the coordinates based on semantic instructions and autonomously navigate to perform tasks. This mechanism eliminates continuous dependence on AR glasses, achieving "one-time guidance, multiple reuses," significantly reducing the operational threshold and training cost of robots in non-AR-assisted scenarios, improving the convenience and practicality of human-computer interaction, and lowering the large-scale deployment cost and application threshold of high-end intelligent equipment.

[0040] In some embodiments of this application, the calculation process of the target spatial coordinates includes: determining the relative coordinates of the target object in the coordinate system of the augmented reality device, and the pose matrix of the augmented reality device relative to the base coordinate system of the robot, wherein the pose matrix is ​​used to characterize the attitude and position information of the augmented reality device in the base coordinate system of the robot; and calculating the target spatial coordinates of the target object in the base coordinate system of the robot based on the relative coordinates and the pose matrix using a preset coordinate transformation formula.

[0041] Specifically, in this embodiment, to accurately map the target focused by the human eye onto the robot's workspace, spatial transformation calculations between multiple coordinate systems are performed. Taking AR glasses as an example, firstly, the AR glasses use their built-in eye-tracking sensors and vision cameras to determine the three-dimensional relative coordinates of the target object with respect to the augmented reality device's own coordinate system (AR coordinate system). Simultaneously, the pose matrix of the AR glasses relative to the robot's base coordinate system is acquired in real time. This pose matrix includes the translation vector and rotation matrix of the augmented reality device on the robot's base, used to characterize the relative geometric relationship between the two in space.

[0042] Based on the aforementioned relative coordinates and pose matrix, mathematical operations are performed using a preset coordinate transformation formula to calculate the target object's spatial coordinates in the robot's base coordinate system. This calculation process utilizes the inverse operation principle of homogeneous transformation matrices to transform the target object from the AR coordinate system to the world coordinate system (or unified reference system), and then further to the robot's base coordinate system. For example, if the relative coordinates of the target object in the AR glasses' coordinate system are known to be... A T T This relative coordinate reflects the three-dimensional spatial position of the target object relative to the center of the AR glasses lens. The pose matrix of the AR glasses relative to the robot's base coordinate system is obtained. R T A The pose matrix describes the position translation and attitude rotation of the AR glasses in the robot's base coordinate system, establishing a spatial mapping relationship between the two coordinate systems.

[0043] Based on the determined relative coordinates and pose matrix, the target space coordinates of the target object in the robot base coordinate system are calculated using a preset coordinate transformation formula. R T T The specific calculation process follows the rules of matrix multiplication, that is, it uses the pose matrix. R T A relative coordinates A T T Perform coordinate transformation, the formula is as follows: R T T = R T A A T T This formula converts the local coordinates of the target object from the perspective of AR glasses into the global working coordinates from the perspective of the robot, thereby ensuring that the robot can accurately understand the specific position of the target object in its own space, providing accurate coordinate basis for subsequent navigation and task execution.

[0044] By establishing a spatial mapping relationship between the augmented reality device's coordinate system and the robot's base coordinate system, and using a pose matrix to transform the relative coordinates of the target object, the target position from the user's perspective is accurately converted into coordinates within the robot's workspace. This effectively solves the problem of target positioning deviation caused by changes in the wearing position or angle of augmented reality devices (such as AR glasses), achieving spatial alignment of the human and machine perspectives. This provides the robot with high-precision navigation guidance, improving the robot's accuracy in recognizing target objects and its task execution success rate in complex dynamic scenes.

[0045] In some embodiments of this application, determining the relative coordinates of the target object in the coordinate system of the augmented reality device includes: acquiring pupil position information of the target object and calculating a gaze vector based on the pupil position information; constructing a scene point cloud based on a scene depth map built into the augmented reality device and calculating the intersection of the gaze vector and the scene point cloud, and determining the pixel region where the target object is located based on the intersection; determining the center pixel coordinates of the target object based on image features within the pixel region; and converting the center pixel coordinates into the relative coordinates of the target object in the coordinate system of the augmented reality device through inverse projection transformation based on the current pose data of the augmented reality device and the depth value corresponding to the center pixel coordinates in the scene depth map.

[0046] Specifically, determining the relative coordinates of a target object within the coordinate system of the augmented reality device is primarily based on the deep fusion of human eye-tracking technology and AR visual sensors. First, the AR glasses use built-in eye-tracking sensors to collect real-time pupil position information while the user is wearing the glasses. Combining this with the geometric relationship between the eye center and the lens, the user's current gaze vector is calculated. This gaze vector represents the spatial direction of the user's intended focus, providing initial guidance for subsequent target localization.

[0047] AR glasses acquire single-frame or continuous-frame depth images, or scene depth maps, within the current field of view using a built-in depth camera. This depth map is a two-dimensional matrix, where each valid pixel corresponds to a depth value, representing the distance of the scene point corresponding to that pixel along the optical axis (Z-axis) in the camera coordinate system. To construct a 3D scene point cloud, each valid pixel in the scene depth map is traversed. Using the camera's intrinsic parameter matrix and the corresponding depth value, the two-dimensional pixel coordinates are converted to spatial point coordinates in the 3D camera coordinate system using an inverse projection formula. The set of converted 3D coordinates for all valid pixels constitutes the current scene point cloud. Points in the scene point cloud are traversed, and the perpendicular distance from point P to the line of the viewing vector is calculated. If the perpendicular distance is less than a preset threshold (e.g., 5-10 cm) and the depth value of point P is greater than a preset threshold, then point P is considered to be within the viewing range. The point closest to the optical center of the AR glasses and satisfying the distance threshold condition is selected as the viewing intersection point, or a ray casting method is used to find the collision point with the minimum depth value along the viewing direction as the final intersection point.

[0048] The calculated intersection points are projected back onto the 2D image plane of the AR glasses in 3D space to obtain the corresponding pixel coordinates. Considering the slight jitter error in human eye focusing and the certain volume of the target object, a preset search window radius (e.g., 50 pixels) is set with these pixel coordinates as the center to determine a rectangular or circular area, which is the pixel area where the target object is located.

[0049] Within this pixel region, image features are further extracted to determine the precise target center. Specifically, edge detection or color segmentation is performed on the image patch within this pixel region to identify the foreground region belonging to the target object. The geometric center or centroid of this foreground region is calculated to obtain the center pixel coordinates of the target object. For example, if the target object is a water glass, the outline of the glass is extracted within the pixel region using color thresholding, and the center point of the outline is calculated as the center pixel coordinates; if the target is a document, the center of the document is located using Optical Character Recognition (OCR) or edge features. Finally, using the center pixel coordinates and the depth value at the corresponding position in the scene depth map, a subsequent inverse projection transformation is performed to obtain the precise relative coordinates.

[0050] Converting the center pixel coordinates to relative coordinates in the augmented reality (AR) device's coordinate system is achieved through the inverse projection transformation of the pinhole camera model. First, the intrinsic parameter matrix of the AR glasses camera is obtained. This matrix includes the camera's focal length and the offset of the optical center on the image plane, describing the camera's internal optical geometry. Simultaneously, the depth value corresponding to the center pixel coordinates of the target object is read from the scene depth map. It should be noted that the depth value here refers to the perpendicular distance of the target object along the optical axis (i.e., the Z-axis) to the camera's imaging plane in the AR device's coordinate system. The center pixel coordinates are normalized to obtain coordinates on the normalized plane. The normalized coordinates are then scaled using the obtained depth value to calculate the target object's three-dimensional spatial coordinates in the AR device's coordinate system.

[0051] Through the aforementioned inverse projection transformation, the two-dimensional image feature points that the user's visual focus is mapped to three-dimensional relative coordinates in the augmented reality device's coordinate system. These relative coordinates, with the AR device's lens center as the origin, accurately describe the target object's positional relationship within the AR device's local space. This coordinate data forms the basis for subsequent coordinate transformations, which will then be combined with the AR device's pose matrix relative to the robot's base to further transform the target object's position into the robot's workspace coordinate system.

[0052] In some embodiments of this application, determining the pose matrix of the augmented reality device relative to the robot's base coordinate system includes: obtaining a first pose matrix of the augmented reality device relative to the world coordinate system and a second pose matrix of the robot relative to the world coordinate system; inverting the second pose matrix to obtain an inverted second pose matrix; and performing matrix multiplication between the inverted second pose matrix and the first pose matrix to obtain the pose matrix of the augmented reality device relative to the robot's base coordinate system.

[0053] Specifically, in this embodiment, the pose matrix of the augmented reality device relative to the robot's base coordinate system is determined by introducing the world coordinate system as an intermediate reference and deriving and calculating it using the relative transformation relationships of the coordinate systems. First, two spatial pose information pieces are obtained: one is the pose matrix of the augmented reality device relative to a unified world coordinate system (i.e., the first pose matrix). W T A The second is the robot's pose matrix relative to the same world coordinate system (i.e., the second pose matrix). W T R The world coordinate system can be a pre-established fixed global coordinate system or an environment map coordinate system constructed in real time using SLAM technology. The first pose matrix describes the translation and rotation of the augmented reality device in world space, and the second pose matrix describes the translation and rotation of the robot in world space.

[0054] To establish a direct coordinate mapping between the augmented reality device and the robot, it is necessary to invert the second pose matrix to obtain the inverse transformation matrix of the robot relative to the world coordinate system (i.e., the inverted second pose matrix). W T R ) -1 The inverse transformation matrix is ​​used to transform the coordinate system from the world coordinate system back to the robot's base coordinate system. Mathematically, if the coordinates of a point in the world coordinate system are known, multiplying by (… W T R ) -1 This gives the coordinates of the point in the robot's base coordinate system. Perform matrix multiplication to obtain the inverted second pose matrix (...). W T R ) -1 With the first pose matrix W T A Multiplying these matrices allows us to calculate the pose matrix of the augmented reality device relative to the robot's base coordinate system. R T A The specific calculation formula is as follows: R T A =( W T R ) -1W T A .

[0055] Through this series of coordinate transformations, the pose of the augmented reality device from the world's perspective is converted to the pose from the robot's perspective. This pose matrix fully represents the positional offset and angular deviation of the augmented reality device relative to the robot. It is a key parameter for subsequently transforming the target object from the coordinate system of the augmented reality device to the robot's base coordinate system, ensuring the consistency and accuracy of spatial positioning during human-robot collaboration.

[0056] In some embodiments of this application, controlling a robot to move to a target location based on the target spatial coordinates includes: obtaining a global map of the environment where the target location is located, wherein the global map is pre-constructed using simultaneous localization and mapping technology; in the global map, using the target spatial coordinates as the endpoint, performing global path planning using a global path planning algorithm to obtain a target path; and controlling the robot to move along the target path until it reaches the target location corresponding to the target spatial coordinates.

[0057] Specifically, the robot moves based on the received target spatial coordinates, primarily relying on a pre-built global map of the environment. This global map is generated using Simultaneous Localization and Mapping (SLAM) technology, during the robot's early deployment or idle periods, by scanning and modeling the environment using its onboard LiDAR, vision sensors, and other perception devices. This global map not only includes the geometric structure information of the environment (such as the positions of walls, obstacles, and furniture) but also establishes a mapping relationship between the map coordinate system in the environment and the robot's base coordinate system, providing a precise spatial reference for the robot's autonomous navigation.

[0058] The received target spatial coordinates are used as the endpoint for path planning, and a global path planning algorithm (e.g., A) is invoked. The algorithm searches the global map for the optimal path from the robot's current starting position to the target position. It comprehensively considers the free space and obstacle distribution on the map to plan a collision-free, shortest-distance, or least time-consuming global trajectory—the target path. Based on this target path, and using underlying motion control algorithms such as the Dynamic Window Approach (DWA), the global path is decomposed into specific motor control commands. These commands drive the robot to perform corresponding straight-line driving and turning actions, monitoring its own pose in real time and correcting deviations, smoothly moving along the planned target path until it reaches the target position corresponding to the target spatial coordinates. Throughout the movement, the robot continuously performs closed-loop control through sensor feedback to ensure navigation accuracy and safety, preparing for subsequent grasping or interaction tasks.

[0059] By combining a pre-built global map with a global path planning algorithm, the robot is provided with an optimal navigation benchmark at the macro level, effectively avoiding the blindness and computational redundancy caused by local search. At the same time, the use of Simultaneous Localization and Mapping (SLAM) technology ensures the real-time nature and accuracy of the map environment, enabling the robot to accurately convert the target spatial coordinates into an executable global path in a known environmental structure. This improves the robot's navigation efficiency, the rationality of path planning, and the accuracy of reaching the target location in complex service scenarios.

[0060] In some embodiments of this application, performing a task corresponding to the natural language instruction at the target location includes: after the robot arrives at the target location, acquiring a target image, wherein the target image is an image of the area where the target location is located; extracting feature points of the target object from the target image, and calibrating the target spatial coordinates based on the feature points to obtain calibrated target spatial coordinates; and performing a task corresponding to the natural language instruction based on the calibrated target spatial coordinates.

[0061] Specifically, when the robot reaches the target location through global path planning, a vision-based secondary calibration mechanism is introduced to ensure the accuracy of task execution. The robot activates its onboard vision sensors (such as an RGB camera or a depth camera) to acquire real-time images of the area where the target location is located, obtaining a current target image containing the target object and its surrounding environment. This target image reflects the actual physical world state perceived by the robot and can capture positional deviations caused by map errors, robot positioning drift, or minor movements of environmental objects.

[0062] Feature extraction processing is performed on the acquired target image to identify and extract key feature points (such as corner points, edge features, or specific texture features) of the target object. Using feature matching algorithms from computer vision, the extracted feature points are correlated and matched with pre-stored map features or the initial spatial coordinates transmitted by the augmented reality device. By calculating the projection transformation matrix between the image coordinate system and the world coordinate system, the true 3D position of the target object in the current real-world scene can be accurately calculated. Based on this calculation result, the target spatial coordinates previously calculated by the augmented reality device are corrected and calibrated to obtain calibrated target spatial coordinates. This process effectively eliminates the positioning errors accumulated during long-distance navigation, achieving a leap from coarse to fine positioning. Based on the calibrated target spatial coordinates, the robotic arm or related work unit is driven to perform specific tasks (such as grasping, placing, or manipulating) corresponding to natural language commands.

[0063] Because the calibrated coordinates are closer to the actual physical location of the target object, the robot can plan the end effector trajectory of the robotic arm more accurately, avoiding collisions or grasping failures, thus improving the success rate and stability of the robot in performing tasks, especially in service scenarios that are not static or have slight obstructions.

[0064] In some embodiments of this application, the task corresponding to the natural language instruction is executed based on the calibrated target spatial coordinates, including: obtaining the operation pose information or target operation box of the target object based on the calibrated target spatial coordinates, and determining a collision-free operation path based on the operation pose information or target operation box, wherein the operation pose information is used to characterize the position and posture reached by the robot's end effector, and the target operation box is used to characterize the operation position, posture, and range; controlling the robot's end effector to move along the collision-free operation path to a pre-operation pose, wherein the pre-operation pose is the spatial pose of the end effector relative to the surface of the target object, along a preset approach direction and maintaining a preset safety distance; selecting a corresponding operation strategy according to the shape of the target object, and controlling the end effector to perform the target operation based on the operation strategy to complete the task corresponding to the natural language instruction.

[0065] Specifically, after the robot obtains the calibrated target space coordinates, it needs to further acquire the target object's operational pose information or target bounding box to plan a collision-free operation path. High-precision 3D point cloud or depth images of the target object are acquired using a vision sensor. Target detection and pose estimation algorithms (such as 6D pose estimation) are then used to calculate the precise pose of the target object in the robot's base coordinate system. Based on this pose information, a 3D bounding box or fine mesh model of the target object, i.e., the target bounding box, is constructed. Combining the robot's own kinematic model and information about surrounding obstacles, a trajectory from the current robotic arm configuration to the target grasping / operation pose is planned within the robot's workspace. This trajectory planning employs a sampling-based path planning algorithm, searching the configuration space for a continuous path that does not interfere with environmental obstacles or the robot itself—a collision-free operation path. If the operational pose information is used, the inverse kinematics solution path for the end effector to reach that pose is directly planned, and collision constraints are verified. If the target bounding box is used, the approach, insertion, or contact path of the end effector relative to the box is planned.

[0066] After path planning is completed, the robot's end effector moves along a collision-free operation path to the pre-operation pose. The pre-operation pose refers to the spatial pose of the end effector when it reaches the vicinity of the target object's surface, along a preset approach direction (usually the normal direction or a direction perpendicular to the contact surface), and maintaining a preset safe distance (e.g., 5-10 cm). The purpose of setting the pre-operation pose is to allow for final visual fine-tuning or buffering before force-controlled contact, preventing the robot from directly colliding with the target object at high speed, thus avoiding damage or misjudgment. At this position, the robot can restart the vision system for sub-millimeter-level pose correction, confirming the relative positional relationship between the end effector and the target object, ensuring the accuracy of subsequent operations.

[0067] The robot dynamically selects the appropriate operation strategy based on the shape, material, and specific semantics of the natural language command. For example, if the target object is a rigid cylinder (such as a water glass) and the command is "pick up the water glass," the "grip" strategy is selected, the end effector uses a parallel gripper, and the control strategy is to close the gripper to a set torque to ensure a stable grip without damaging the object. If the target object is a flexible fabric (such as clothing) and the command is "arrange the clothing," the "drag" or "grab a corner" strategy is selected, the end effector uses a flexible finger or suction cup, and the control strategy is adjusted to low-speed control to prevent tearing the clothing. If the target object is a fragile item (such as a glass) and the command is "move the glass," the "touch and push" strategy is selected, the end effector uses a force control mode to push the object along the specified direction with minimal contact force. Through this adaptive selection of strategies based on shape recognition, the robot can flexibly respond to the operation requirements of objects with different shapes, improving the robustness and safety of operations.

[0068] Based on the selected operational strategy, the robot arm's end effector is controlled to perform the target operation to complete the task corresponding to the natural language command. Specific forms of the target operation include: for graspable objects, performing a "grasp" action, i.e., the end effector closes and captures the object; for objects requiring interaction, performing "press," "rotate," or "insert" actions; for scenarios requiring object movement, if the robot itself has a mobile chassis, the target operation can include "follow-movement," i.e., the robot arm maintains the grasping state while the chassis moves to a new position along the command path; for fixed-station operations, it can include a "place" action, i.e., precisely placing the object into a designated area. Throughout the entire target operation process, the robot monitors the force sensor feedback in real time. If abnormal resistance or slippage is detected, the operation immediately stops and is replanned to ensure the task is completed safely and accurately.

[0069] By combining the operation pose / target operation frame with collision-free path planning and pre-operation pose mechanism, the robot arm achieves precise connection and safe obstacle avoidance from navigation to micro-operation, significantly improving operation safety and alignment accuracy. At the same time, it adaptively selects operation strategies based on the shape of the target object to perform diverse tasks, overcoming the shortcomings of poor adaptability of traditional robots and greatly improving the success rate and flexibility of service robots in complex and unstructured scenarios.

[0070] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the robot task execution method of this application will be described in detail below with reference to specific embodiments.

[0071] This embodiment relates to a specific task execution method for a robot, which is applied to a system including service robots and AR glasses, such as... Figure 2As shown, the system includes an AR glasses system and a service robot system. The AR glasses system includes a visual capture unit, a human eye focus recognition unit, a sensor unit, a semantic recognition unit, a coordinate analysis module, and a wireless transmission subunit. The service robot system includes a data transmission unit, a navigation unit, a sensor unit, an actuator unit, a semantic recognition unit, an artificial intelligence unit, a storage unit, and a control motherboard.

[0072] Figure 3 This is a layout diagram, such as Figure 3 As shown, a base coordinate system R:O1-X1Y1Z1 for the robot is created in the world coordinate system W:O-XYZ, with the pose matrix relative to the world coordinate system as follows: w T R AR glasses coordinate system A: O2-X2Y2Z2 relative to the world coordinate system pose matrix: w T A Target coordinate system T: O3-X3Y3Z3, relative world coordinate system pose matrix: w T T .in R: rotation matrix; P: coordinate translation vector.

[0073] Figure 4 A flowchart illustrating a specific robot task execution method provided in this application embodiment is shown below. Figure 4 As shown, an AR glasses and service robot hardware system is built, and the AR glasses are embedded with software adapted to the hardware units. When the user issues a command, the AR glasses perform semantic analysis to determine whether the visually captured object matches the semantically analyzed object. If they match, the AR glasses calculate the target point and send it to the service robot. The service robot navigates to the target point and starts the specific visual analysis task. Then, the robot executes the task and determines whether the target point is a commonly used fixed point. If so, the robot records the point and associates it with the corresponding task.

[0074] Specifically, it includes the following steps:

[0075] Step 1: Hardware Configuration. Select compatible AR glasses and a service robot to complete the basic hardware setup. The AR glasses should be a smart AR device with high-definition visual capture and eye-tracking capabilities. It should have a built-in visual capture camera, eye focus sensor, coordinate resolution chip, and wireless transmission module, enabling real-time capture of image information of the area focused by the human eye and rapid resolution of 3D spatial coordinates. The service robot should be a collaborative robot with autonomous navigation and grasping capabilities. It should have a built-in control motherboard, visual sensor, distance sensor, mechanical actuator, and storage module. The control motherboard receives coordinate data transmitted from the AR glasses and controls the robot to complete navigation and subsequent tasks. The storage module stores coordinate data for some fixed tasks.

[0076] The communication between the AR glasses and the robot includes:

[0077] Communication protocol stack: Physical layer / link layer: Wi-Fi 6 (802.11ax) is used to ensure high bandwidth and anti-interference capabilities in complex indoor environments; Transport layer: TCP is used to ensure reliable delivery of key data such as coordinates and instructions; Application layer: The lightweight JSON-RPC protocol is used to serialize data.

[0078] The communication process includes: Session establishment: After the AR glasses are powered on, a TCP three-way handshake is initiated through the previously bound robot IP / hostname. Both parties complete TLS encryption authentication based on a pre-shared key (PSK) to establish a secure data channel. Heartbeat and synchronization: After the connection is established, both parties send heartbeat packets to each other at a frequency of 10Hz and synchronize the timestamp and map ID of the world coordinate system to ensure the timeliness of coordinate data. Task instruction issuance: After semantic parsing is completed, the AR glasses package the task type, target object ID, and activation signal of the current human eye gaze state into an instruction frame and send it. Coordinate data push: After the human eye stabilizes its gaze, the coordinate parsing chip calculates the transformation matrix RTT of the target relative to the robot base in real time, and pushes it continuously via UDP (low overhead) at a frequency of about 30Hz in the form of a binary floating-point array (x,y,z,qx,qy,qz,qw) until the gaze ends or the robot confirms. Feedback loop: After the robot successfully receives the coordinates, it sends back an ACK; after reaching the target area and recognizing the object, it sends back a confirmation image to the AR glasses via a video stream, completing the loop.

[0079] A person's position is calculated using visual-inertial odometry (VIO). The camera in the AR glasses captures images, and the inertial measurement unit (IMU) collects acceleration and angular velocity. The increment matrix A is obtained by estimating the relative motion between camera frames, calculating the offset and rotation of the relative motion. W T A =Previous Frame W T A A. The person's position is calculated in real time using this method. The camera detects keyframes of the constructed sparse point cloud map and performs loop closure correction to eliminate accumulated errors.

[0080] The robot's position is calculated using an adaptive Monte Carlo localization algorithm. The robot's onboard wheel encoder and IMU provide linear and angular velocities. The algorithm employs particle filtering and scattering: a large number of particles are generated near the robot to predict its new pose. All particles are compared with data acquired by the existing LiDAR, and the particles with the highest matching degree are assigned high weights. High-weighted particles are copied, while low-weighted particles are deleted. Finally, a weighted average of all particles is calculated, and this particle pose represents the latest robot coordinates. W T R .

[0081] Target point location calculation: Target locking: By acquiring the pupil position and gaze vector of the human eye through an eye-tracking sensor, combined with the scene depth map acquired by the camera built into the AR glasses, the intersection of the gaze vector and the scene point cloud is calculated to lock the target object region focused by the human eye; simultaneously, the YOLO instance segmentation algorithm is used to identify the object category and extract the complete pixel contour of the object. Coordinate calculation: Based on the glasses pose output by the AR glasses' own visual VIO, combined with the pixel depth information of the depth map, the pixel coordinates of the target object are converted into three-dimensional coordinates in the AR glasses coordinate system, and finally the homogeneous transformation matrix of the target object in the world coordinate system is output. W T T And save it.

[0082] Step 2: Software adaptation and command activation. Human eye focusing recognition algorithm and coordinate analysis algorithm are embedded in AR glasses to complete software debugging and adaptation. When a human wearing AR glasses issues a natural language command, the command is converted into an electrical signal through the voice recognition module built into the AR glasses, which automatically activates the visual capture function of AR glasses and enters the task target coordinate capture process.

[0083] The semantic deep parsing is performed using a multimodal semantic deep parsing framework that is collaborative between the edge and cloud. The specific process includes:

[0084] Multimodal Input: The AR glasses simultaneously capture both audio and first-person video streams. Automatic Speech Recognition (ASR): The AR glasses' built-in microphone array captures the user's speech, which is then converted to text using an edge-side speech recognition model, supporting noise reduction and echo cancellation. Entity Extraction: Based on a finely tuned BERT model, Named Entity Recognition (NER) is performed on the text, extracting three core entity categories: Action entities (grabbing, delivering, pouring water, placing, etc.); Target entities (objects such as cups, documents, tools, etc.); Location entities (destination such as desktops, water dispensers, etc.). Intent Classification: Combining the entity extraction results with the command context, a classification model determines the user's core intent, categorizing it into three types: "single-step operation" (e.g., "bring the cup over"), "multi-step task" (e.g., "pour a cup of water and bring it over"), and "query." Task decomposition and action sequence generation: For multi-step tasks, the robot decomposes them into an ordered sequence of actions according to its execution logic. For example, "pour a glass of warm water" is decomposed into: grab the glass → move to the water dispenser → fill the glass with water → move to the user's location → deliver the glass, while matching the coordinates of corresponding fixed / common points. Task instruction generation: Finally, structured JSON machine instructions are generated, including action type, target label, spatial constraints, etc., and sent out through a communication protocol.

[0085] Current mainstream AR glasses typically include an infrared eye-tracking sensor, an RGB-D depth camera, and a VIO positioning module. The infrared camera captures the pupil position and gaze vector, while the depth camera obtains scene depth information to calculate the spatial position of the gaze point. Existing target capture capabilities only serve local AR device interactions (such as selecting a virtual UI or displaying annotation information) and cannot be directly adapted to robot collaboration scenarios. This embodiment makes three targeted improvements:

[0086] Semantic linkage conditional activation mechanism: In existing solutions, the visual capture module is often in the background standby or running all the time; in this embodiment, the target capture and coordinate calculation module is only activated when the voice semantic parsing determines it to be a "spatial operation command" (grab, deliver, move, etc.). Non-operation commands do not activate the visual computing power, reducing device power consumption and avoiding false triggers caused by random eye scanning.

[0087] The robot's base coordinate system coordinate adaptation output: The target coordinates output by existing AR glasses are based on their own coordinate system or AR world coordinate system, which cannot be directly used by the robot navigation system; This embodiment adds a coordinate transformation adaptation layer, which converts the target coordinates calculated by the AR end into standard pose data in the robot's base coordinate system in real time through pre-calibrated coordinate system alignment parameters and homogeneous transformation formulas, which can be directly input into the robot navigation and motion planning module without secondary conversion.

[0088] The existing solutions for target recognition optimization focusing on the Region of Interest (ROI) mostly involve full-screen target detection, which consumes a lot of computing power. In this embodiment, after activation and capture, only the ROI with a 15° field of view centered on the human eye's focal point is selected for object recognition and precise coordinate calculation. This greatly improves the calculation speed and reduces the probability of misidentification of background objects, making it more suitable for scenarios requiring rapid robot response.

[0089] Step 3: Target Coordinate Capture and Transmission. After the AR glasses are activated, the human eye focusing sensor captures the position of the target object being viewed in real time, and the visual camera simultaneously captures images of the target area. The coordinate resolution algorithm combines image information and spatial positioning data to quickly resolve the three-dimensional spatial coordinates (X, Y, Z) of the target object relative to the AR glasses. , combined Figure 3 According to the contravariant formula: B T A =( W T B ) -1W T A Since R is an orthogonal matrix, we have: ;

[0090] Target → AR glasses: A T T =( W T A ) -1W T T ;

[0091] AR glasses → Robot's base coordinate system: R T A =( W T R ) -1W T A ;

[0092] according to A T T =( W T A ) -1W T T and R T A =( W T R ) -1W T A Obtain the target → robot's base coordinate system: R T T = R T A A T T =( W T R )-1W T A A T T .

[0093] The AR glasses transmit the target coordinate data to the service robot's control motherboard via a wireless transmission subunit.

[0094] The above explains the basic principles of coordinate transformation, illustrating the inverse T of an orthogonal matrix T. -1 The calculation method is as follows: A: AR glasses coordinate system; T: target point coordinate system; R: robot coordinate system; W: world coordinate system. A T T Let T be the coordinate matrix of the target point T in the AR glasses coordinate system. Similarly, other coordinate systems can be transformed according to the above formula.

[0095] Step 4: Robot navigation and task execution. Figure 5 A schematic diagram of the service robot navigation process, such as Figure 5 As shown, the process includes the following steps: SLAM mapping, target point acquisition, current pose calculation, global path planning, local path planning, chassis motion execution, and stopping the robot when it reaches the target position; if so, navigation ends. Specifically, the robot typically completes SLAM mapping before the task begins. After the robot's control motherboard receives coordinate data transmitted from the AR glasses and calculates the target point, it uses its own IMU, encoder, and LiDAR to detect its pose data in real time. The robot then uses a static SLAM map to perform AR navigation. The algorithm begins with global path planning, and then uses the DWA algorithm to plan local paths along the optimal path while driving the motors to move the chassis. This continues until the mainboard controls the navigation unit to move the robot to the target coordinate area. The robot then uses a vision sensor to identify the specific shape of the target object, controls the mechanical arm to complete the grasping action, and then executes subsequent operations according to the instructions. The entire process requires no human intervention, achieving precise and efficient execution.

[0096] Global Path Planning (A) The specific calculation process of the algorithm is as follows: The global path is based on the two-dimensional occupancy grid map constructed by the robot's SLAM, and uses A... The heuristic search algorithm for finding the optimal path involves the following steps:

[0097] Map preprocessing: The raster map is binarized, marking passable grids (value 0) and obstacle grids (value 1); the expansion radius is set according to the robot chassis size, and the obstacle grids are expanded to avoid robot edge collisions. Start and end point mapping: The robot's current pose (starting point) and target object coordinates (ending point) are converted from the world coordinate system to the raster coordinates of the raster map. A Algorithm Iterative Search: Initialize the OpenList and ClosedList. Add the starting point to the OpenList and calculate the cost function f(n) = g(n) + h(n) for the starting point, where g(n) is the actual movement cost from the starting point to the current grid (horizontal / vertical movement cost is 1, diagonal movement cost is 2); h(n) is the heuristically estimated cost from the current grid to the destination, calculated using Euclidean distance to ensure the algorithm's adoptability. Iteratively select the grid with the smallest f(n) in the OpenList as the current node and move it to the ClosedList. Traverse the 8 neighboring grids of the current node, skipping obstacle grids and grids in the ClosedList. Calculate a new g value for each neighboring grid. If the new g value is smaller or the grid is not in the OpenList, update its g value, f value, and parent node, and add it to the OpenList. When the destination is added to the OpenList, the algorithm terminates. Starting from the destination, backtrack to the parent node to obtain the global path in the form of a grid sequence.

[0098] The optimal path is A The algorithm theoretically assumes the optimal path cost function f(n) = g(n) + h(n) is at its minimum, satisfying the adoption condition. The core process of DWA local path planning is as follows: Determine the velocity search space: Based on the robot's current velocity and acceleration / deceleration limits, calculate the range of linear velocity v and angular velocity ω achievable in the next cycle (dynamic window). Sample candidate velocities: Discretely sample within the window with a fixed step size, generating several (v, ω) combinations. Simulate the forward trajectory: For each (v, ω) combination, simulate the robot's trajectory within 2-3 seconds. Score and select the best trajectory: Weight each trajectory. Orientation score: Whether the trajectory end aligns with the global path target direction; Safety score: Minimum distance of the trajectory from obstacles; Efficiency score: Magnitude of linear velocity. Issue the optimal velocity: The (v, ω) with the highest score is sent to the motor for execution. Repeat the above process in the next cycle.

[0099] Object Shape Recognition: After the robot reaches the target coordinate point directly in front, it activates its onboard RGB-D structured light depth camera to perform multi-view scanning of the target area. First, it runs lightweight instance segmentation (YOLOv8-seg) to select all suspected targets in the RGB image and compares them with the semantic features in the command to lock onto the target. The depth point cloud of the corresponding region of the segmentation mask is extracted and iteratively registered with the nearest point in the pre-stored CAD model of the object or the template point cloud saved during the initial guidance, thereby accurately obtaining the object's 6D pose (precise position and orientation) and geometric bounding box. If it is an unknown flexible object, a feasible grasping box is directly inferred from the point cloud using a grasping pose detection algorithm.

[0100] Grasping Action Execution: Based on the obtained target 6D pose and bounding box, the robotic arm planner generates a collision-free grasping path. Pre-grasping Pose: The robotic arm end effector is controlled to move to a safe distance directly above or in front of the target. Grasping Strategy: A strategy is selected based on the object's shape (two-finger concentric grasp for cylindrical objects, suction cup or adaptive underactuated hand grasp for irregular objects). The end effector approaches the object in a straight line along the normal direction of the object's surface until the force feedback sensor detects that the contact force exceeds a threshold, or the visual servo determines that the grasping point has been reached. Closing and Lifting: The fingers close to firmly grasp the object, and the robotic arm lifts along the normal direction, completing the grasp. Subsequently, it coordinates with the navigation chassis to perform the transportation task.

[0101] Step 5: Fixed-point memory and autonomous execution. When the robot performs a task related to a fixed object, the AI ​​model connected to the robot will automatically identify the object as a commonly used fixed object, store its 3D coordinate information in the fixed-point database of the storage module, and label the object name. Subsequently, when a human issues a related command again without wearing AR glasses, the robot's control board will automatically retrieve the corresponding coordinates from the fixed-point database. Without the assistance of AR glasses, it can autonomously navigate to the target location and complete the task in conjunction with sensors, achieving "one-time guidance, multiple reuses" and greatly reducing the amount of repetitive training.

[0102] In this embodiment, "frequently used points" and "fixed points" are no longer strictly distinguished and are uniformly included in the "fixed point database" for management. However, logical layering is achieved through confidence levels and status markers: Automatic identification and storage: After the robot successfully grasps and completes the task for the first time, the AI ​​model it accesses performs semantic attribute analysis on the grasped item. If the item category belongs to "static high-reusability entities" (such as coffee tables, refrigerators, specific workstation slots, etc., determined by their semantic tags and long-term static characteristics), the AI ​​model stores the item's coordinates, multi-angle visual features, and item name into the fixed point database and marks the status as "calibrated". Dynamic correction: If visual repositioning detects a slight shift in the coordinates of the fixed item during subsequent task execution, the robot will update the old coordinates in the database using a real-time coordinate moving average. However, if the shift exceeds a threshold, it will request manual or AR glasses to re-guide it. Recall logic: Without AR glasses, the robot receives an instruction and directly and accurately retrieves the item name in the database to call the corresponding fixed coordinate point. This mechanism can be summarized as: "one-time teaching, dynamic memory, long-term reuse".

[0103] The following is an example based on a specific application scenario:

[0104] Initial guidance: The elderly person, wearing AR glasses, sits on the sofa and says, "I need to take my medicine." The AR glasses interpret the intention as "take the medicine box on the coffee table," automatically activating eye tracking. The elderly person looks at the white medicine box on the coffee table, and the glasses determine the precise coordinates of the medicine box in the robot's coordinate system, sending it to the home service robot.

[0105] Execution and Memory: The robot plans its path globally, moves to the coffee table, visually recognizes the "white medicine box," grasps it, and smoothly navigates to deliver it to the elderly person. After completing the task, the AI ​​identifies the "white medicine box" as a high-frequency, fixed-location item and automatically stores its coordinates, visual features, and "medicine box" label into a fixed-location database.

[0106] Glasses-free reuse: The next day, the elderly person did not need to wear AR glasses. They only needed to say softly, "Get me the medicine." After receiving the voice, the robot directly retrieved the storage coordinates of the "medicine box" from the fixed location database, autonomously planned its way to the coffee table, and successfully grabbed and delivered the medicine again.

[0107] Dynamic adjustment: If the pillbox is slightly moved on a certain day, after the robot reaches the fixed coordinates, the depth camera can search in a small area nearby, match the point cloud of the white pillbox, fine-tune the grasping pose to complete the task, and update the new coordinates to the database.

[0108] The entire process embodies intelligent collaboration of "one-time guidance, multiple uses".

[0109] This application also provides a robot task execution device. It should be noted that the robot task execution device of this application can be used to execute the robot task execution method provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0110] The following describes the task execution device for the robot provided in the embodiments of this application.

[0111] Figure 6 This is a structural block diagram of a robot task execution device according to an embodiment of this application. Figure 6 As shown, the device includes a receiving unit 61 and an execution unit 62. The receiving unit receives target spatial coordinates and natural language instructions transmitted by the augmented reality device. The natural language instructions are task instructions issued by the target object when wearing the augmented reality device, requiring the robot to perform the task. The target spatial coordinates are calculated by the augmented reality device through coordinate transformation based on the spatial position of the target object being viewed by the target object in the coordinate system of the augmented reality device when the natural language instructions are received. The execution unit controls the robot to move to the target position according to the target spatial coordinates and performs the task corresponding to the natural language instructions at the target position.

[0112] This embodiment transforms the target gazed at by the human eye into precise spatial coordinates, achieving accurate binding between voice commands and physical locations. This solves the problem of ambiguous positioning caused by the robot's lack of spatial awareness and improves the accuracy of task execution.

[0113] In some embodiments of this application, the execution unit includes an identification module, a storage module, and a navigation module. The identification module identifies the target object at the target location and determines the semantic category of the target object. The storage module stores the spatial coordinates of the target location and the semantic name of the target object in a storage device if the semantic category belongs to a preset common item category. The semantic name is a category label determined by visual recognition of the target object. The navigation module, when the augmented reality device is not connected, retrieves the spatial coordinates corresponding to the semantic name from the storage device and navigates to the target location based on the spatial coordinates to perform the task when a natural language instruction including the semantic name is received.

[0114] This embodiment achieves automatic memorization and offline reuse of the locations of frequently used items. When a target object is identified as a frequently used item, the robot automatically stores its spatial coordinates and semantic name in the database; subsequently, even if the AR device is not connected, the robot can directly retrieve the coordinates based on semantic instructions and autonomously navigate to perform tasks. This mechanism eliminates the continuous dependence on AR glasses, achieving "one-time guidance, multiple reuses," significantly reducing the operating threshold and training cost of the robot in non-AR-assisted scenarios, and improving the convenience and practicality of human-computer interaction.

[0115] In some embodiments of this application, the apparatus includes a determining unit and a calculating unit. The determining unit is used to determine the relative coordinates of the target object in the coordinate system of the augmented reality device, and the pose matrix of the augmented reality device relative to the base coordinate system of the robot, wherein the pose matrix is ​​used to characterize the attitude and position information of the augmented reality device in the base coordinate system of the robot; the calculating unit is used to calculate the target spatial coordinates of the target object in the base coordinate system of the robot based on the relative coordinates and the pose matrix using a preset coordinate transformation formula.

[0116] By establishing a spatial mapping relationship between the augmented reality device's coordinate system and the robot's base coordinate system, and using a pose matrix to transform the relative coordinates of the target object, the target position from the user's perspective is accurately converted into coordinates within the robot's workspace. This effectively solves the problem of target positioning deviation caused by changes in the wearing position or angle of augmented reality devices (such as AR glasses), achieving spatial alignment of the human and machine perspectives. This provides the robot with high-precision navigation guidance, improving the robot's accuracy in recognizing target objects and its task execution success rate in complex dynamic scenes.

[0117] In some embodiments of this application, the determining unit includes an acquisition module, a calculation module, a determining module, and a conversion module. The acquisition module acquires pupil position information of the target object and calculates a gaze vector based on the pupil position information. The calculation module constructs a scene point cloud based on the scene depth map built into the augmented reality device and calculates the intersection of the gaze vector and the scene point cloud, determining the pixel region where the target object is located based on the intersection. The determining module determines the center pixel coordinates of the target object based on image features within the pixel region. The conversion module converts the center pixel coordinates into the relative coordinates of the target object in the augmented reality device's coordinate system through inverse projection transformation, based on the current pose data of the augmented reality device and the depth value corresponding to the center pixel coordinates in the scene depth map.

[0118] Through the aforementioned inverse projection transformation, the two-dimensional image feature points that the user's visual focus is mapped to three-dimensional relative coordinates in the augmented reality device's coordinate system. These relative coordinates, with the AR device's lens center as the origin, accurately describe the target object's positional relationship within the AR device's local space. This coordinate data forms the basis for subsequent coordinate transformations, which will then be combined with the AR device's pose matrix relative to the robot's base to further transform the target object's position into the robot's workspace coordinate system.

[0119] In some embodiments of this application, the determining unit includes a first acquisition module, an inversion module, and a calculation module. The first acquisition module is used to acquire the first pose matrix of the augmented reality device relative to the world coordinate system and the second pose matrix of the robot relative to the world coordinate system; the inversion module is used to invert the second pose matrix to obtain the inverted second pose matrix; the calculation module is used to perform matrix multiplication between the inverted second pose matrix and the first pose matrix to obtain the pose matrix of the augmented reality device relative to the base coordinate system of the robot.

[0120] Through this series of coordinate transformations, the pose of the augmented reality device from the world's perspective is converted to the pose from the robot's perspective. This pose matrix fully represents the positional offset and angular deviation of the augmented reality device relative to the robot. It is a key parameter for subsequently transforming the target object from the coordinate system of the augmented reality device to the robot's base coordinate system, ensuring the consistency and accuracy of spatial positioning during human-robot collaboration.

[0121] In some embodiments of this application, the execution unit includes a second acquisition module, a planning module, and a control module. The second acquisition module is used to acquire a global map of the environment where the target location is located, the global map being pre-constructed using simultaneous localization and mapping (SLAM) technology; the planning module is used to perform global path planning in the global map, with the target spatial coordinates as the endpoint, using a global path planning algorithm to obtain the target path; the control module is used to control the robot to move along the target path until it reaches the target location corresponding to the target spatial coordinates.

[0122] By combining a pre-built global map with a global path planning algorithm, the robot is provided with an optimal navigation benchmark at the macro level, effectively avoiding the blindness and computational redundancy caused by local search. At the same time, the use of Simultaneous Localization and Mapping (SLAM) technology ensures the real-time nature and accuracy of the map environment, enabling the robot to accurately convert the target spatial coordinates into an executable global path in a known environmental structure. This improves the robot's navigation efficiency, the rationality of path planning, and the accuracy of reaching the target location in complex service scenarios.

[0123] In some embodiments of this application, the execution unit includes a third acquisition module, an extraction module, and an execution module. The third acquisition module is used to acquire a target image after the robot reaches the target position, wherein the target image is an image of the area where the target position is located; the extraction module is used to extract feature points of the target object from the target image and calibrate the target spatial coordinates based on the feature points to obtain calibrated target spatial coordinates; the execution module is used to execute a task corresponding to the natural language instruction based on the calibrated target spatial coordinates.

[0124] Because the calibrated coordinates are closer to the actual physical location of the target object, the robot can plan the end effector trajectory of the robotic arm more accurately, avoiding collisions or grasping failures, thus improving the success rate and stability of the robot in performing tasks, especially in service scenarios that are not static or have slight obstructions.

[0125] In some embodiments of this application, the execution module includes an acquisition submodule, a control submodule, and a selection submodule. The acquisition submodule is used to acquire the operation pose information or target operation frame of the target object based on the calibrated target spatial coordinates, and to determine a collision-free operation path based on the operation pose information or the target operation frame. The operation pose information characterizes the position and attitude reached by the robot's end effector, and the target operation frame characterizes the operation position, attitude, and range. The control submodule is used to control the robot's end effector to move along the collision-free operation path to a pre-operation pose. The pre-operation pose is the spatial pose of the end effector relative to the surface of the target object, along a preset approach direction, and maintaining a preset safe distance. The selection submodule is used to select a corresponding operation strategy based on the shape of the target object, and to control the end effector to perform the target operation based on the operation strategy to complete the task corresponding to the natural language instruction.

[0126] By combining the operation pose / target operation frame with collision-free path planning and pre-operation pose mechanism, the robot arm achieves precise connection and safe obstacle avoidance from navigation to micro-operation, significantly improving operation safety and alignment accuracy. At the same time, it adaptively selects operation strategies based on the shape of the target object to perform diverse tasks, overcoming the shortcomings of poor adaptability of traditional robots and greatly improving the success rate and flexibility of service robots in complex and unstructured scenarios.

[0127] The robot's task execution device includes a processor and a memory. The receiving unit, execution unit, etc., are all stored as program units in the memory, and the processor executes the program units stored in the memory to achieve the corresponding functions. All the modules are located in the same processor; alternatively, the modules may be located in different processors in any combination.

[0128] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0129] This invention provides a robot task execution system, comprising: an augmented reality device; a robot, wherein the robot is communicatively connected to the augmented reality device, the robot includes a controller, and the controller is used to execute any of the robot task execution methods described above.

[0130] This invention provides a computer-readable storage medium that includes a stored program, wherein the program, when running, controls the device containing the computer-readable storage medium to execute the task execution method of the robot.

[0131] This invention provides a processor for running a program, wherein the program executes the task execution method of the robot.

[0132] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the robot's task execution method described above. The device described herein can be a server, PC, PAD, mobile phone, etc.

[0133] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform the steps of initializing the task execution method of the robot described above.

[0134] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0135] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0140] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0141] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0144] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for robot task execution, characterized in that, include: The system receives target spatial coordinates and natural language instructions transmitted by an augmented reality device. The natural language instructions are task instructions issued by the target object when wearing the augmented reality device, which require the robot to execute. The target spatial coordinates are calculated by the augmented reality device through coordinate transformation based on the spatial position of the target object being viewed by the target object in the coordinate system of the augmented reality device when the natural language instructions are received. The robot is controlled to move to the target location based on the target spatial coordinates, and then performs the task corresponding to the natural language command at the target location.

2. The method according to claim 1, characterized in that, Executing the task corresponding to the natural language instruction at the target location includes: Identify the target object at the target location and determine the semantic category of the target object; If the semantic category belongs to a preset common item category, the spatial coordinates of the target location and the semantic name of the target object are stored in the storage device. The semantic name is a category label determined by visual recognition of the target object. If the augmented reality device is not connected, and a natural language instruction including the semantic name is received, the spatial coordinates corresponding to the semantic name are retrieved from the storage device, and the user navigates to the target location based on the spatial coordinates to perform the task.

3. The method according to claim 1, characterized in that, The calculation process of the target spatial coordinates includes: The relative coordinates of the target object in the coordinate system of the augmented reality device are determined, as well as the pose matrix of the augmented reality device relative to the base coordinate system of the robot. The pose matrix is ​​used to characterize the attitude and position information of the augmented reality device in the base coordinate system of the robot. Based on the relative coordinates and the pose matrix, the target spatial coordinates of the target object in the robot's base coordinate system are calculated using a preset coordinate transformation formula.

4. The method according to claim 3, characterized in that, Determining the relative coordinates of the target object in the coordinate system of the augmented reality device includes: Collect pupil position information of the target object, and calculate the gaze vector based on the pupil position information; A scene point cloud is constructed based on the scene depth map built into the augmented reality device, and the intersection of the gaze vector and the scene point cloud is calculated. The pixel region where the target object is located is determined based on the intersection. The center pixel coordinates of the target object are determined based on the image features within the pixel region; Based on the current pose data of the augmented reality device and the depth value corresponding to the center pixel coordinates in the scene depth map, the center pixel coordinates are converted into the relative coordinates of the target object in the coordinate system of the augmented reality device through inverse projection transformation.

5. The method according to claim 3, characterized in that, Determining the pose matrix of the augmented reality device relative to the robot's base coordinate system includes: Obtain the first pose matrix of the augmented reality device relative to the world coordinate system, and the second pose matrix of the robot relative to the world coordinate system; Invert the second pose matrix to obtain the inverted second pose matrix; The inverted second pose matrix is ​​multiplied by the first pose matrix to obtain the pose matrix of the augmented reality device relative to the robot's base coordinate system.

6. The method according to claim 1, characterized in that, Controlling the robot to move to the target position according to the target spatial coordinates includes: Obtain a global map of the environment where the target location is located, wherein the global map is pre-constructed using simultaneous localization and mapping (SLAM) technology; In the global map, with the target spatial coordinates as the endpoint, a global path planning algorithm is used to perform global path planning to obtain the target path; The robot is controlled to move along the target path until it reaches the target position corresponding to the target spatial coordinates.

7. The method according to claim 1, characterized in that, Executing the task corresponding to the natural language instruction at the target location includes: After the robot reaches the target location, a target image is acquired, wherein the target image is an image of the area where the target location is located; Feature points of the target object are extracted from the target image, and the target spatial coordinates are calibrated based on the feature points to obtain calibrated target spatial coordinates; Execute the task corresponding to the natural language instruction based on the calibrated target space coordinates.

8. The method according to claim 7, characterized in that, Executing the task corresponding to the natural language instruction based on the calibrated target space coordinates includes: Based on the calibrated target space coordinates, the operation pose information or target operation box of the target object is obtained, and a collision-free operation path is determined based on the operation pose information or the target operation box. The operation pose information is used to characterize the position and posture reached by the robot's end effector, and the target operation box is used to characterize the operation position, posture and range. The robot's end effector is controlled to move along the collision-free operation path to a pre-operation pose, wherein the pre-operation pose is the spatial pose of the end effector relative to the surface of the target object, along a preset approach direction and maintaining a preset safe distance. The corresponding operation strategy is selected based on the shape of the target object, and the end effector of the robotic arm is controlled to perform the target operation based on the operation strategy to complete the task corresponding to the natural language instruction.

9. A task execution device for a robot, characterized in that, include: The receiving unit is used to receive target spatial coordinates and natural language instructions transmitted by the augmented reality device. The natural language instructions are task instructions issued by the target object when wearing the augmented reality device, which require the robot to execute. The target spatial coordinates are calculated by the augmented reality device through coordinate transformation based on the spatial position of the target object being viewed by the target object in the coordinate system of the augmented reality device when the natural language instructions are received. An execution unit is used to control the robot to move to the target position according to the target spatial coordinates, and to perform the task corresponding to the natural language instruction at the target position.

10. A task execution system for a robot, characterized in that, include: Augmented reality devices; A robot, which is communicatively connected to the augmented reality device, the robot including a controller for performing the task execution method of the robot as described in any one of claims 1 to 8.

Images