Robot operation method, robot, storage medium, apparatus, and program product

Abstract

The application provides a robot operation method, a robot, a storage medium, equipment and a program product. The method comprises the following steps: generating operation planning information for achieving a target task according to text information and / or visual information in a device specification in a corresponding document format of a target device; after receiving an observation image containing the target device, aligning an operable component in the target device with an operable component in the device specification to obtain a first correspondence relationship between a component identifier of the operable component in the target device and a component name of the operable component in the device specification; and based on the operation planning information and the first correspondence relationship, controlling the robot to execute a corresponding operation state for a corresponding operable component of the operation step to achieve the operation step. The application can improve the adaptability and universality of the robot operation equipment.

Description

Technical Field

[0001] The present application relates to the field of robot operation technology, and in particular to a robot operation method, a robot, a storage medium, a device and a program product. Background Art

[0002] With the rapid development of intelligent manufacturing and service robotics, robots are increasingly being used in daily life and industrial production. In the field of equipment operation and maintenance, robots can automate complex tasks, improving efficiency and reducing human errors. However, existing robotic operation methods rely on pre-programmed action sequences or instantaneous sensor-based decision-making. These methods typically require specialized programming for each specific device, resulting in limited adaptability and versatility when dealing with diverse and complex equipment.

[0003] Based on this, the present application provides a robot operation method, a robot, a storage medium, a device and a program product to improve the relevant technology. Summary of the Invention

[0004] The purpose of this application is to provide a robot operation method, a robot, a storage medium, a device and a program product to improve the adaptability and versatility of the robot operation device.

[0005] The purpose of this application is achieved by the following technical solutions:

[0006] In the first aspect, the present application provides a robot operation method, the method comprising: generating operation planning information for achieving a target task based on text information and / or visual information in a device manual in a document format corresponding to the target device; the operation planning information is used to indicate the component name and operation status of an operable component in the target device corresponding to at least one operation step; after receiving an observation image containing the target device, aligning the operable component in the target device with the operable component in the device manual to obtain a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device manual; based on the operation planning information and the first correspondence, controlling the robot to execute a corresponding operation state for the operable component corresponding to the operation step to achieve the operation step.

[0007] In some embodiments, before generating the operation planning information, the method also includes: converting the device manual in the document format corresponding to the target device into one or more page images; performing text recognition on the page images to obtain the text information, and / or performing visual element detection on the page images to obtain the visual information.

[0008] In some embodiments, the performing text recognition on the page image to obtain the text information includes: performing text recognition on the page image using an OCR optical character recognition model to obtain the text information; and / or, the performing visual element detection on the page image to obtain the visual information includes: performing visual element detection on the page image using an MLLM multimodal large language model to obtain the visual information.

[0009] In some embodiments, the aligning of the operable component in the target device with the operable component in the device manual to obtain a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device manual includes: when the corresponding structural model of the target device is available, aligning the operable component in the structural model with the operable component in the device manual to obtain a second correspondence between the component model identification of the operable component in the structural model and the component name of the operable component in the device manual; aligning the target device with the structural model according to the object posture of the target device in the observed image to obtain a third correspondence between the component identification of the operable component in the target device and the component model identification of the operable component in the structural model; and determining the first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device manual based on the second correspondence and the third correspondence.

[0010] In some embodiments, the visual information includes a device image, which contains a component name. Aligning the operable component in the target device with the operable component in the device manual to obtain a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device manual includes: marking the component identification of the operable component on the corresponding operable component in the observation image; inputting the observation image with the component identification and the device image containing the component name into the MLLM multimodal large language model to identify the first correspondence between the component identification and the component name.

[0011] In some embodiments, marking the component identification of the operable component on the corresponding operable component in the observed image includes: using a detection model to detect the bounding box of the operable component in the observed image; using the bounding box as an input of a segmentation model to generate a mask of the operable component; superimposing the mask of the operable component on the corresponding operable component in the observed image; and marking the component identification of the operable component on the mask of the corresponding operable component in the observed image.

[0012] In some embodiments, masks of different operable components correspond to different colors and / or transparencies; and / or, in the observed image, the component identifier of the operable component is located at the geometric center of the corresponding mask.

[0013] In some embodiments, based on the operation planning information and the first correspondence, the robot is controlled to execute the corresponding operation state for the operable component corresponding to the operation step, including: when the corresponding structural model of the target device is available, based on the operation planning information and the first correspondence, determining the component model identification and operation state corresponding to one or more operation steps; designing the operation action of the robot to execute the corresponding operation state for the corresponding operable component according to the component model identification and operation state corresponding to the operation step and the target structural features of the corresponding operable component in the structural model; and controlling the robot to execute the operation action for the corresponding operation state for the operable component.

[0014] In some embodiments, the target structural feature of the operable component is determined according to the component type of the operable component.

[0015] In some embodiments, based on the operation planning information and the first correspondence, the robot is controlled to perform the corresponding operation state for the operable components corresponding to the operation steps, including: when the corresponding structural model of the target device is unavailable, based on the operation planning information and the first correspondence, determining the component identification and operation state corresponding to one or more operation steps; processing the component identification and operation state corresponding to the operation steps to obtain operation information in a language format; using an open vocabulary operation model to process the operation information to control the robot to perform the operation action of the corresponding operation state for the operable components.

[0016] In a second aspect, the present application provides a robot comprising a control module, wherein the control module is configured to execute any one of the above methods to implement one or more operation steps.

[0017] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0018] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.

[0019] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, implements any of the above methods.

[0020] The present application provides a robot operation method, a robot, a storage medium, a device and a program product. First, based on the text information and visual information (such as charts, diagrams, etc.) contained in the device manual of the target device, operation planning information for achieving the target task is generated. The planning information indicates the component name of the operable component corresponding to each operation step and its expected operation state. After receiving an observation image containing the target device, the operable components in the actual device are aligned with the operable components described in the device manual, and a first correspondence between the component identification and the component name is established. Subsequently, based on the operation planning information and the first correspondence, the robot is controlled to perform operation actions of the corresponding operation state for the operable components, thereby realizing the operation steps of the task planning. The present application ensures the consistency between the actual device and the instruction manual information through a precise component alignment mechanism, thereby improving the accuracy and reliability of the operation planning. Secondly, based on the effective integration of high-level task planning and low-level operation control, the robot can flexibly respond to complex and changeable operation tasks, thereby enhancing the adaptability and versatility of the robot operation. In addition, this method supports efficient operations in different scenarios with and without the assistance of structural models, broadens the application scope of robots, achieves a significant improvement in the intelligence and automation level of robot operations, and improves the efficiency and accuracy of robot operations on various devices such as household appliances. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present application is further described below with reference to the accompanying drawings and specific implementation methods.

[0022] Figure 1 It is a framework diagram of a ManualPlan model provided in an embodiment of the present application.

[0023] Figure 2 It is a flowchart of a robot operation method provided in an embodiment of the present application.

[0024] Figure 3 This is an alignment flow chart of a ManualPlan model provided in an embodiment of the present application (the structural model is available).

[0025] Figure 4 This is a robot operation diagram (structural model is available) that combines ManualPlan and CAD model provided in an embodiment of the present application.

[0026] Figure 5This is a robot operation diagram combining ManualPlan and OmniManip provided in an embodiment of the present application (the structural model is not available).

[0027] Figure 6 This is a structural block diagram of a robot provided in an embodiment of the present application.

[0028] Figure 7 This is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative work are within the scope of protection of this application.

[0030] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly indicate the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0031] With the rapid development of intelligent manufacturing and service robotics, robots are increasingly being used in daily life and industrial production. In the field of equipment operation and maintenance, robots can automatically perform complex operational tasks, improving efficiency and reducing human errors. However, related robotic operation methods rely on pre-programmed action sequences or instantaneous sensor-based decision-making. These methods typically require specialized programming for each specific device, resulting in insufficient adaptability and versatility when faced with diverse and complex equipment. In addition, related methods lack efficient parsing and understanding capabilities when processing text and visual information in equipment manuals, making it difficult to automatically generate operation plans for specific tasks.

[0032] Device manuals, crucial documents guiding users in the proper use of equipment, contain extensive operating procedures, component information, and safety precautions. However, translating these human-readable manuals into robot-executable instructions remains a pressing challenge. Research in this area has largely focused on combining natural language processing (NLP) with computer vision (CV), but these methods still suffer from several deficiencies in practical applications. First, most methods focus solely on parsing textual information, neglecting visual elements such as graphics and tables within the manuals. This results in an incomplete understanding of device components and operating procedures. Second, when aligning component information in the manuals with the actual equipment, there is a lack of effective visual cue mechanisms, resulting in inaccurate component recognition and alignment, which in turn affects the accuracy of operational planning. Furthermore, these methods often rely on predefined rules or templates for operational planning, lacking flexibility and intelligence, making them incapable of handling complex and diverse operational tasks. Furthermore, when robots perform specific operational actions, they often lack mechanisms to effectively integrate high-level task planning with low-level operational control, resulting in inadequate operational consistency and efficiency.

[0033] In response to the above problems, this application proposes a robot operation method based on equipment manuals. This method combines optical character recognition (OCR), multimodal large language model (MLLM) and visual prompt mechanism to achieve comprehensive parsing of text information and visual information in equipment manuals, and generate accurate operation planning information. By aligning the operable parts in the actual device with the operable parts in the manual, the accurate execution of the operation steps is ensured. This method not only improves the robot's adaptability to diversified electrical equipment, but also enhances its execution efficiency and accuracy in complex tasks, and has broad application prospects.

[0034] Specifically, to enable manual-based robotic operation planning, the applicant proposed the ManualPlan model, which is used for manual-based task planning. For tasks involving robot execution, ManualPlan can be integrated as a high-level planning module of the relevant operation model to achieve expansion.

[0035] See also Figure 1 , Figure 1This is a framework diagram of a ManualPlan model provided in an embodiment of the present application. The ManualPlan model includes, for example, a manual extraction module, an operation planning module, and a component alignment module. ManualPlan can perform advanced planning to control CAD (Computer Aided Design)-assisted basic actions or large models of open vocabulary operations, such as OmniManip, to use equipment. These models can serve as the basis for the CheckManual (check manual) dataset. Figure 1 In the example, the target device is an oven, the target task is "heating bread in the upper oven", the upper oven door and the upper oven temperature knob are component names, open, close, and rotate forward 30° are operation states, and component 1 and component 5 are component identifiers.

[0036] It should be noted that although some embodiments herein are described using robots operating home appliances as an example, this application is applicable not only to robots operating home appliances, but also to robots operating other appliances or devices other than appliances. Although some embodiments herein are described using manuals as an example, this application is applicable not only to robot operation planning based on manuals, but also to robot operation planning based on instructions other than manuals.

[0037] The following is a detailed description of the implementation of this application.

[0038] See also Figure 2 , Figure 2 It is a flowchart of a robot operation method provided in an embodiment of the present application.

[0039] An embodiment of the present application provides a robot operation method, which includes steps S101 to S103.

[0040] Step S101: Generate operation planning information for achieving the target task based on the text information and / or visual information in the device manual in the document format corresponding to the target device; the operation planning information is used to indicate the component name and operation status of the operable component in the target device corresponding to at least one operation step.

[0041] Step S102: After receiving the observation image containing the target device, align the operable components in the target device with the operable components in the device manual to obtain a first correspondence between the component identification of the operable components in the target device and the component name of the operable components in the device manual.

[0042] Step S103: Based on the operation planning information and the first corresponding relationship, the robot is controlled to execute a corresponding operation state for the operable component corresponding to the operation step to implement the operation step.

[0043] As an example, the ManualPlan model introduces a new framework for high-level task planning for robotic operations, which requires the use of an equipment manual. Specifically, the model processes the manual content, generates a detailed plan, and aligns the operable parts in the actual equipment with the operable parts referenced in the manual.

[0044] In some embodiments, the document format of the device manual may be a PDF format. In other embodiments, the document format of the device manual may be a DOC format.

[0045] In some embodiments, before generating the operation planning information, the method may also include: converting the device manual in the document format corresponding to the target device into one or more page images; performing text recognition on the page images to obtain the text information, and / or performing visual element detection on the page images to obtain the visual information.

[0046] In some embodiments, performing text recognition on the page image to obtain the text information may include performing text recognition on the page image to obtain the text information using an OCR (Optical Character Recognition) model.

[0047] In some embodiments, performing visual element detection on the page image to obtain the visual information may include: performing visual element detection on the page image to obtain the visual information using an MLLM multimodal large language model. MLLM stands for Multimodal Large Language Model.

[0048] In some embodiments, generating the operational planning information for achieving the target task based on the textual information and / or visual information in the device manual in the corresponding document format of the target device may include generating the operational planning information for achieving the target task based on the textual information and / or visual information in the device manual in the corresponding document format of the target device using an LLM (Large Language Model). LLM stands for Large Language Model.

[0049] As an example, assume that the device manuals in the dataset include electrical appliance manuals, and the electrical appliance manuals are in PDF format, which is consistent with the real manuals. In the case where the PDF file cannot be directly processed by a large language model (which can be simply referred to as a large language model), the manual PDF file can be first divided into page-level images (which can be simply referred to as page images). Then, an OCR optical character recognition model (for example, the Paddle OCR model) is used to extract text content from these page images to obtain text information. In addition, the MLLM multimodal large language model is used to detect visual elements on each page image, such as graphics and tables. In this way, textual information and visual information at the page granularity of the electrical appliance manual can be obtained.

[0050] Through the extracted manual information, the LLM large language model is called to plan a step-by-step appliance operation process for the target task (e.g., a given task). For example, the large language model can specify the component function name (i.e., component name) and operation status for each planning step (i.e., the planned operation step) in a dictionary format. Only the operable components mentioned in the manual can be used for planning. Sometimes, only part of the task is demonstrated in the manual, so the large language model can plan based on common sense of appliance use and the manual content. In practical applications, difficult predictions (i.e., when the operation planning information fails to be generated) can trigger a regeneration request to regenerate the operation planning information for achieving the target task.

[0051] In order to achieve alignment of the actual device (i.e., target device) with the components in the device specification, the applicant adopted a visual cue mechanism called Set-of-Mark (SoM) to align the operable components of the actual device in the field of view with the operable components described in the device specification. SoM uses a segmentation model to divide the observed image containing the target device into different regions and assigns a digital mark to each region, which can significantly improve the visual positioning ability of MLLM. Specifically, a detection model (e.g., an open vocabulary object detection model Grounding-DINO) is first used to predict the bounding boxes of the operable components, and these bounding boxes are used to prompt a segmentation model (e.g., Segment Anything segmentation model) to generate masks (i.e., masks) of the operable components. These masks use different display parameters, for example, different colors can be assigned to different masks with transparency, and then such masks are superimposed on the observed image. Afterwards, the component identification is marked on the mask of each operable component, for example, at the geometric center of the mask. Then, the observed image and the device image in the device manual are provided to the MLLM, which is required to identify the mapping relationship (i.e., the first correspondence relationship) between the mask ID (i.e., the component identification on the mask) and the component function name (i.e., the component name). In this way, the component alignment between the observed image and the device manual is achieved (i.e., the component alignment between the operational components of the actual device in the observed image and the operational components in the device manual).

[0052] For robotic manipulation tasks based on device specifications and structural models (e.g., CAD models), even if the structural model is available, the component function names (i.e., component names) in the device specifications still need to be aligned with the component model identifiers of the operable components on the structural model. Therefore, a similar SoM-based approach can be used with the MLLM to achieve this alignment, resulting in a second correspondence between component model identifiers and component names.

[0053] In robot manipulation tasks based purely on the manual, the structural model is not available, while in robot manipulation tasks based on the device manual-structural model-actual device alignment, component alignment has been achieved. For the latter, this alignment is achieved through two sequentially interchangeable alignment steps, namely, aligning the operable components in the structural model with the operable components in the device manual (obtaining a second correspondence), and aligning the operable components of the actual device in the observed image with the operable components in the structural model (obtaining a third correspondence). The component alignment process will be described in detail below, including cases where the structural model is available and cases where the structural model is not available. As an example, the reason why the structural model is unavailable is, for example, that the structural model has not been received, or that the received structural model does not meet the usage requirements.

[0054] See also Figure 3 , Figure 3 This is an alignment flow chart of a ManualPlan model provided in an embodiment of the present application (the structural model is available). Figure 3 In the example, the target device is a coffee machine, the target task is "prepare a small cup of coffee", the brew button (i.e., brew button), cup size inbox (i.e., cup size selector), power button, and top cover are the component names in the device manual (e.g., device manual), part 0, part 1, part 2, and part 3 are the component model identifiers in the structural model (e.g., CAD model), and open, close, press once, and press twice are the operation states. It can be seen that the ManualPlan model correctly plans tasks based on the given device manual (e.g., device manual) and achieves component alignment between the device manual and the structural model. After achieving component alignment between the device manual, the structural model, and the actual device, in the final planning, each step uses the component model identifier as the target and specifies a specific operation action. It should be noted that after the component alignment between the device manual, the structural model and the actual device is achieved, any two of the component identification (of the operable component of the target device in the observation image), the component model identification (of the operable component in the structural model), and the component name (of the operable component in the device manual) all satisfy a one-to-one correspondence. Therefore, the robot operation control based on component identification is similar to the robot operation control based on component model identification, and this application does not limit this.

[0055] In some embodiments based on the device specification and the structural model, the aligning of the operable components in the target device with the operable components in the device specification to obtain a first correspondence between the component identification of the operable components in the target device and the component name of the operable components in the device specification may include: when the corresponding structural model of the target device is available, aligning the operable components in the structural model with the operable components in the device specification to obtain a second correspondence between the component model identification of the operable components in the structural model and the component name of the operable components in the device specification; aligning the target device with the structural model according to the object posture of the target device in the observed image to obtain a third correspondence between the component identification of the operable components in the target device and the component model identification of the operable components in the structural model; and determining the first correspondence between the component identification of the operable components in the target device and the component name of the operable components in the device specification based on the second correspondence and the third correspondence.

[0056] In some embodiments, the structural model may include a three-dimensional structural model and / or a two-dimensional structural model.

[0057] In some embodiments, the structural model may be a CAD model.

[0058] In some embodiments, the visual information may include a device image, which may include component names. For example, a graphic, diagram, or image in the visual information may be referred to as a device image. Since the visual information is obtained by detecting visual elements from a corresponding page image of a device manual, and device manuals typically use dotted and line annotations (or other visual annotation methods) to indicate the names of multiple device components for easier reading, the device image may, for example, include component names.

[0059] In some embodiments, aligning the operable components in the structural model with the operable components in the device manual to obtain a second correspondence between the component model identifier of the operable component in the structural model and the component name of the operable component in the device manual may include: marking the component model identifier of the operable component on the corresponding operable component in the rendered image of the structural model; inputting the rendered image with the component model identifier and the device image containing the component name into the MLLM multimodal large language model to identify the second correspondence between the component model identifier and the component name.

[0060] In some embodiments, marking the component model identifier of the operable component on the corresponding operable component in the rendered image of the structural model may include: detecting the bounding box of the operable component in the rendered image using a detection model; using the bounding box as input to a segmentation model to generate a mask of the operable component; superimposing the mask of the operable component on the corresponding operable component in the rendered image; and marking the component model identifier of the operable component on the mask of the corresponding operable component in the rendered image. Using the bounding box as input to the segmentation model to generate the mask of the operable component, for example, means inputting the bounding box into the segmentation model so that the segmentation model performs a segmentation operation to obtain the mask of the operable component.

[0061] In some embodiments, masks of different operable components may correspond to different display parameters. In some embodiments, the display parameters may include one or more of color, transparency, font style, font size, font weight, brightness, contrast, saturation, exposure, hue, brightness, grayscale, color temperature, and scaling.

[0062] In some embodiments, masks for different operable components may correspond to different colors and / or transparencies.

[0063] In some embodiments, in the rendered image, the component model identifier of the operable component may be located at the geometric center of the corresponding mask.

[0064] In some embodiments, aligning the operable component in the target device with the operable component in the device manual to obtain a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device manual may include: marking the component identification of the operable component on the corresponding operable component in the observation image; inputting the observation image with the component identification and the device image containing the component name into the MLLM multimodal large language model to identify the first correspondence between the component identification and the component name.

[0065] In some embodiments, marking the component identification of the operable component on the corresponding operable component in the observed image may include: using a detection model to detect the bounding box of the operable component in the observed image; using the bounding box as an input of a segmentation model to generate a mask of the operable component; superimposing the mask of the operable component on the corresponding operable component in the observed image; and marking the component identification of the operable component on the mask of the corresponding operable component in the observed image.

[0066] In some embodiments, in the observed image, the component identifier of the operable component may be located at the geometric center of the corresponding mask.

[0067] When performing operations on a target device, ManualPlan can be used as a high-level planner to design different robot operation strategies for cases where a structural model is available and when a structural model is not available.

[0068] See also Figure 4 , Figure 4 This is a robot operation diagram that combines ManualPlan and CAD models provided in the embodiment of this application (the structural model is available). Figure 4 In the example, the target device is a coffee machine, the target task is "prepare a cup of coffee", parts 1 to 8 are component model identifiers, the power button, valve stem button, volume button, clean button, and single cup button are component names, and pressing once is the operation state. ManualPlan uses the object pose predicted by FoundationPose (the basic pose model) and the CAD model to correctly align device components and plan tasks. Using this information, atomic actions such as pressing buttons and turning knobs are designed to perform corresponding operation actions according to the task plan.

[0069] In some embodiments, controlling the robot to execute the corresponding operation state for the operable component corresponding to the operation step based on the operation planning information and the first correspondence may include: determining the component model identification and operation state corresponding to one or more operation steps based on the operation planning information and the first correspondence when the corresponding structural model of the target device is available; designing the operation action of the robot to execute the corresponding operation state for the corresponding operable component according to the component model identification and operation state corresponding to the operation step and the target structural features of the corresponding operable component in the structural model; and controlling the robot to execute the operation action for the corresponding operation state for the operable component.

[0070] In the above embodiment, the operation planning information corresponds to component names and operation states. After achieving component alignment between the device specification, the structural model, and the actual device, the component names correspond to the component model identifiers. Therefore, the component model identifiers and operation states corresponding to the operation steps can be determined. If the structural model is available, the target structural features of the operable components in the structural model can be used to design the robot's operational actions (e.g., atomic actions).

[0071] In some embodiments, the target structural feature of the operable component may be determined according to the component type of the operable component.

[0072] In some embodiments, the component type may include one or more of a slider, a button, and a knob.

[0073] In some embodiments, the target structural feature may include one or more of a joint, a hinge, a door, a slider, and a surface.

[0074] For example, assume that the structural model is a CAD model. When the CAD model is available, the 6-DOF pose of the target device observed in the camera frame is estimated as the object pose of the target device in the observed image. Using this estimated object pose, the observed actual device can be aligned with its CAD model, with the positions and axis orientations of all joints available. As an example, corresponding primitive motions can be designed for different operable components to synthesize CAD-assisted manipulation strategies. For example, for doors and components with sliders (e.g., screens, sliders, lids), the 3D bounding box of the operable component is provided to constrain the robot's component-level grasping pose. The gripper pose in the motion sequence can then be calculated based on information about the joints and manipulation methods. The gripper is an example of a robot's end effector. For a button component, its surface geometric center can be used as the contact point, and the closed gripper can be controlled to push the button from the normal direction of the contact point. For a knob component, the open gripper can be controlled to approach the knob from the normal direction of its surface geometric center, and the gripper can be closed to grasp it. After the gripper successfully grasps the knob, it rotates by a planned angle around the normal direction of its surface geometric center.

[0075] See also Figure 5 , Figure 5 This is a schematic diagram of a robot operation combining ManualPlan and OmniManip provided in an embodiment of the present application (the structural model is not available). Figure 5 In this example, the target device is a microwave oven, the target task is "Inspect the interior of the microwave oven," "1" and "9" are component identifiers, the microwave oven front door and rotating tray are component names, and open and closed are the operational states. ManualPlan achieved accurate alignment of the device components and task planning without the aid of a CAD model. OmniManip further successfully executed the two-step operation.

[0076] In some embodiments, controlling the robot to execute the corresponding operation state for the operable component corresponding to the operation step based on the operation planning information and the first correspondence may include: determining the component identification and operation state corresponding to one or more operation steps based on the operation planning information and the first correspondence when the corresponding structural model of the target device is unavailable; processing the component identification and operation state corresponding to the operation step to obtain operation information in a language format; and using an open vocabulary operation model to process the operation information to control the robot to execute the operation action of the corresponding operation state for the operable component.

[0077] The above embodiment provides a robot manipulation strategy for when a structural model is unavailable. As an example, an open-vocabulary manipulation model (OMM) can be integrated into ManualPlan to complete manipulation tasks where a structural model is unavailable (e.g., without a CAD model). An example of an OMM is OmniManip. In this framework, the ManualPlan and OmniManip models collaborate similarly to the cerebrum and cerebellum in the human brain. ManualPlan reads a device specification (e.g., a manual), performs high-level planning, and aligns the observed operable components with those described in the manual. ManualPlan then locates the operable components and displays the corresponding manipulation information in a language format. This manipulation information is transmitted to OmniManip, for example, via RGB-D camera observation, to control the robot to perform low-level manipulation actions on the target device (e.g., a microwave oven). In RGB-D camera observation, RGB represents information about the three colors red, green, and blue, providing a color image; D represents depth information, providing distance data between the object and the camera. For example, an RGB-D camera is used to acquire RGB-D data of a target device (e.g., a microwave oven), including both a color image and a depth image. ManualPlan transmits RGB-D data and operation information to OmniManip. Based on this data and operation information, OmniManip controls the robot to perform specific, basic operations on the target device, such as pressing a button or turning a knob. In this way, the robot can intelligently perform specific operations on complex equipment based on environmental perception and operating status.

[0078] See also Figure 6 , Figure 6 This is a structural block diagram of a robot provided in an embodiment of the present application.

[0079] An embodiment of the present application further provides a robot, comprising a control module, wherein the control module is configured to execute any one of the above methods to implement one or more operation steps.

[0080] In some embodiments, the robot may be a humanoid robot (also known as a bipedal robot, a humanoid robot), a quadrupedal robot, or a wheeled robot with a robotic arm.

[0081] In some embodiments, the robot may be a multi-jointed robot.

[0082] In some embodiments, the robot may include a robot body, a left arm, a right arm, a head, and a chassis. The left arm and the right arm may be collectively referred to as a robotic arm. However, the number of robotic arms may also be one or more, and the above embodiment does not limit this.

[0083] In some embodiments, the robot may include one or more of an image sensor, a laser sensor, an odometer, an IMU, an angle encoder, a torque sensor, and a PIR sensor.

[0084] In some embodiments, the image sensor may include a camera (or a camera module). As an example, a camera may be provided at the distal joint of the left arm, the distal joint of the right arm, the head, and the chassis, respectively, that is, a left-hand camera, a right-hand camera, a head camera, and a chassis camera.

[0085] In some embodiments, the laser sensor may include a 2D laser sensor and / or a 3D laser sensor. Taking a laser radar as an example, the laser radar may be installed on the head of the robot. However, the above embodiment is not limited to this, and the laser radar may also be installed in other parts of the robot.

[0086] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0087] An embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, implements any of the above methods.

[0088] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0089] See also Figure 7 , Figure 7 This is a structural block diagram of a computer device provided in an embodiment of the present application.

[0090] An embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0091] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0092] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.

[0093] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.

[0094] The processor 120 is configured to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solutions of the embodiments of the present application are implemented through software or firmware, the computer program for implementing the solutions of the embodiments of the present application may be stored in the processor 120 and executed by the processor 120.

[0095] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.

[0096] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor.

[0097] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 120 or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor 120. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0098] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input and output system (BIOS), application software, etc.

[0099] An operating system manages the hardware and / or software resources of a computer device and is the core and cornerstone of the computer. It handles basic tasks such as managing and allocating memory, prioritizing the supply and demand of system resources, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide an interface for users to interact with the system.

[0100] The BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display the processor temperature and execute functions such as adjusting temperature protection strategies.

[0101] Application software, also known as an application program, is software written for a specific user purpose. It is a major category of computer software. For example, application software might be a program used for power control, temperature management, and other purposes.

[0102] It should be understood that the specific examples in this specification are only intended to help those skilled in the art better understand the implementation methods of the present application, rather than to limit the scope of protection of the present application.

[0103] It can be understood that in the various implementations of this specification, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0104] It can be understood that the various implementation methods described in this specification can be implemented individually or in combination, and this application is not limited to this.

[0105] Unless otherwise indicated, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art in the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more of the relevant listed items. The singular forms "a", "above", and "the" used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0106] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0107] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described embodiments may refer to the corresponding processes in other embodiments and will not be repeated here.

[0108] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0109] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the technical solutions of this application.

[0110] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0111] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0112] The above are only specific embodiments of this specification, but the scope of protection of this application is not limited to them. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope disclosed in this specification should be included in the scope of protection of this specification. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A robot operation method, characterized in that: The method comprises: generating, based on textual information and / or visual information in a device manual in a document format corresponding to the target device, operation plan information for achieving the target task; the operation plan information indicating a component name and an operation status of an operable component in the target device corresponding to at least one operation step; After receiving an observation image including the target device, aligning an operable component in the target device with an operable component in the device manual to obtain a first correspondence between a component identifier of the operable component in the target device and a component name of the operable component in the device manual; Based on the operation planning information and the first corresponding relationship, controlling the robot to execute a corresponding operation state for an operable component corresponding to the operation step to implement the operation step; Based on the operation planning information and the first corresponding relationship, controlling the robot to execute a corresponding operation state for an operable component corresponding to the operation step includes: When a structural model corresponding to the target device is available, determining component model identifiers and operation states corresponding to one or more operation steps based on the operation plan information and the first corresponding relationship; Designing an operation action for the robot to perform a corresponding operation state on the corresponding operable component according to the component model identifier and operation state corresponding to the operation step and the target structural feature of the corresponding operable component in the structural model; The robot is controlled to perform an operation action in a corresponding operation state on the operable component.

2. The robot operation method according to claim 1, characterized in that: Before generating the operation plan information, the method further includes: Converting the device manual in a document format corresponding to the target device into one or more page images; Text recognition is performed on the page image to obtain the text information, and / or visual element detection is performed on the page image to obtain the visual information.

3. The robot operation method according to claim 2, characterized in that: The performing text recognition on the page image to obtain the text information includes: performing text recognition on the page image using an OCR optical character recognition model to obtain the text information; and / or, The performing visual element detection on the page image to obtain the visual information includes: performing visual element detection on the page image using an MLLM multimodal large language model to obtain the visual information.

4. The robot operation method according to claim 1, characterized in that: The aligning the operable component in the target device with the operable component in the device specification to obtain a first correspondence between the component identifier of the operable component in the target device and the component name of the operable component in the device specification includes: When a structural model corresponding to the target device is available, aligning the operable components in the structural model with the operable components in the device specification to obtain a second correspondence between component model identifiers of the operable components in the structural model and component names of the operable components in the device specification; aligning the target device with the structural model according to the object posture of the target device in the observed image, so as to obtain a third correspondence between the component identifier of the operable component in the target device and the component model identifier of the operable component in the structural model; Based on the second correspondence and the third correspondence, a first correspondence between the component identifier of the operable component in the target device and the component name of the operable component in the device manual is determined.

5. The robot operation method according to claim 1, characterized in that: The visual information includes a device image, the device image includes a component name, and aligning the operable component in the target device with the operable component in the device manual to obtain a first correspondence between the component identifier of the operable component in the target device and the component name of the operable component in the device manual includes: marking a component identification of the operable component on the corresponding operable component in the observed image; The observation image with the part identification and the device image containing the part name are input into the MLLM multimodal large language model to identify the first correspondence between the part identification and the part name.

6. The robot operation method according to claim 5, characterized in that: The step of marking the component identification of the operable component on the corresponding operable component in the observation image includes: detecting a bounding box of the operable component in the observed image using a detection model; Using the bounding box as input to a segmentation model to generate a mask of the operable component; superimposing the mask of the operable component on the corresponding operable component in the observed image; The component identification of the operable component is marked on the mask of the corresponding operable component in the observation image.

7. The robot operation method according to claim 6, characterized in that: Masks of different operable components correspond to different colors and / or transparencies; and / or, In the observed image, the component identifier of the operable component is located at the geometric center of the corresponding mask.

8. The robot operation method according to claim 1, characterized in that: The target structural feature of the operable component is determined according to the component type of the operable component.

9. The robot operation method according to claim 1, characterized in that: The controlling the robot to execute a corresponding operation state for an operable component corresponding to the operation step based on the operation planning information and the first corresponding relationship includes: In a case where the structural model corresponding to the target device is unavailable, determining component identifiers and operation states corresponding to one or more operation steps based on the operation plan information and the first correspondence; Processing the component identification and operation status corresponding to the operation step to obtain operation information in a language format; The operation information is processed using an open vocabulary operation model to control the robot to perform an operation action of a corresponding operation state on the operable component.

10. A robot, characterized in that: The robot comprises a control module, and the control module is used to execute the method according to any one of claims 1 to 9 to implement one or more operation steps.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images