Robot operation method, robot, storage medium, device and program product

Abstract

The invention provides a robot operation method, a robot, a storage medium, equipment and a program product, and the method comprises the steps: generating operation planning information used for achieving a target task according to text information and / or visual information in an equipment specification of a document format corresponding to target equipment; after an observation image containing the target equipment is received, an operable component in the target equipment is aligned with an operable component in the equipment specification, so that a first corresponding relation between a component identifier of the operable component in the target equipment and a component name of the operable component in the equipment specification is obtained; and based on the operation planning information and the first corresponding relationship, controlling the robot to execute a corresponding operation state for an operable part corresponding to the operation step so as to realize the operation step. The adaptability and universality of the robot operation equipment can be improved.

Description

Technical Field

[0001] The present application relates to the field of robot operation technology, and in particular to a robot operation method, a robot, a storage medium, a device and a program product. Background Art

[0002] With the rapid development of intelligent manufacturing and service robot technology, robots are increasingly used in daily life and industrial production. In the field of equipment operation and maintenance, robots can automatically perform complex operation tasks, improve efficiency and reduce human operation errors. However, related robot operation methods rely on pre-programmed action sequences or instant decision-making based on sensors. These methods usually require special programming for each specific device, resulting in insufficient adaptability and versatility when facing diverse and complex equipment.

[0003] Based on this, the present application provides a robot operation method, a robot, a storage medium, a device and a program product to improve the relevant technology. Summary of the invention

[0004] The purpose of this application is to provide a robot operation method, a robot, a storage medium, a device and a program product to improve the adaptability and versatility of the robot operation device.

[0005] The purpose of this application is achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a robot operation method, the method comprising: generating operation planning information for achieving a target task based on text information and / or visual information in a device manual in a document format corresponding to the target device; the operation planning information is used to indicate a component name and an operation status of an operable component in the target device corresponding to at least one operation step; after receiving an observation image containing the target device, aligning the operable component in the target device with the operable component in the device manual to obtain a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device manual; based on the operation planning information and the first correspondence, controlling the robot to execute a corresponding operation status for the operable component corresponding to the operation step to achieve the operation step.

[0007] In some embodiments, before generating the operation planning information, the method also includes: converting the device manual in the document format corresponding to the target device into one or more page images; performing text recognition on the page images to obtain the text information, and / or performing visual element detection on the page images to obtain the visual information.

[0008] In some embodiments, performing text recognition on the page image to obtain the text information includes: using an OCR optical character recognition model to perform text recognition on the page image to obtain the text information; and / or, performing visual element detection on the page image to obtain the visual information includes: using an MLLM multimodal large language model to perform visual element detection on the page image to obtain the visual information.

[0009] In some embodiments, the aligning of the operable components in the target device with the operable components in the device manual to obtain a first correspondence between component identifications of the operable components in the target device and component names of the operable components in the device manual includes: when a corresponding structural model of the target device is available, aligning the operable components in the structural model with the operable components in the device manual to obtain a second correspondence between component model identifications of the operable components in the structural model and component names of the operable components in the device manual; aligning the target device with the structural model through the object posture of the target device in the observed image to obtain a third correspondence between component identifications of the operable components in the target device and component model identifications of the operable components in the structural model; and determining a first correspondence between component identifications of the operable components in the target device and component names of the operable components in the device manual based on the second correspondence and the third correspondence.

[0010] In some embodiments, the visual information includes a device image, the device image includes a component name, and the aligning the operable component in the target device with the operable component in the device manual to obtain a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device manual includes: marking the component identification of the operable component on the corresponding operable component in the observation image; inputting the observation image with the component identification and the device image including the component name into the MLLM multimodal large language model to identify the first correspondence between the component identification and the component name.

[0011] In some embodiments, marking the component identification of the operable component on the corresponding operable component in the observed image includes: using a detection model to detect the bounding box of the operable component in the observed image; using the bounding box as an input of a segmentation model to generate a mask of the operable component; superimposing the mask of the operable component on the corresponding operable component in the observed image; marking the component identification of the operable component on the mask of the corresponding operable component in the observed image.

[0012] In some embodiments, masks of different operable components correspond to different colors and / or transparencies; and / or, in the observed image, the component identifier of the operable component is located at the geometric center of the corresponding mask.

[0013] In some embodiments, based on the operation planning information and the first correspondence, controlling the robot to execute a corresponding operation state for the operable component corresponding to the operation step includes: when the corresponding structural model of the target device is available, determining the component model identifier and operation state corresponding to one or more operation steps based on the operation planning information and the first correspondence; designing the operation action of the robot to execute the corresponding operation state for the corresponding operable component according to the component model identifier and operation state corresponding to the operation step and the target structural features of the corresponding operable component in the structural model; and controlling the robot to execute the operation action for the corresponding operation state for the operable component.

[0014] In some embodiments, the target structural feature of the operable component is determined according to the component type of the operable component.

[0015] In some embodiments, based on the operation planning information and the first correspondence, the robot is controlled to execute the corresponding operation state for the operable components corresponding to the operation steps, including: when the corresponding structural model of the target device is not available, based on the operation planning information and the first correspondence, determining the component identification and operation state corresponding to one or more operation steps; processing the component identification and operation state corresponding to the operation steps to obtain operation information in a language format; using an open vocabulary operation model to process the operation information to control the robot to execute the operation action of the corresponding operation state for the operable components.

[0016] In a second aspect, the present application provides a robot, comprising a control module, wherein the control module is used to execute any of the above methods to implement one or more operation steps.

[0017] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements any of the above methods when executed by a processor.

[0018] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.

[0019] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program implements any one of the above methods when executed by a processor.

[0020] The present application provides a robot operation method, a robot, a storage medium, a device and a program product. First, based on the text information and visual information (such as charts, diagrams, etc.) contained in the device manual of the target device, an operation planning information for achieving the target task is generated, and the planning information indicates the component name of the operable component corresponding to each operation step and its expected operation state. After receiving the observation image containing the target device, the operable component in the actual device is aligned with the operable component described in the device manual, and a first corresponding relationship between the component identification and the component name is established. Subsequently, based on the operation planning information and the first corresponding relationship, the robot is controlled to perform the operation action of the corresponding operation state for the operable component, thereby realizing the operation steps of the task planning. The present application ensures the consistency between the actual device and the instruction manual information through a precise component alignment mechanism, and improves the accuracy and reliability of the operation planning. Secondly, based on the effective integration of high-level task planning and low-level operation control, the robot can flexibly respond to complex and changeable operation tasks, and enhances the adaptability and versatility of the robot operation. In addition, this method supports efficient operation in different scenarios with and without the assistance of structural models, broadens the application scope of robots, achieves a significant improvement in the intelligence and automation level of robot operations, and improves the efficiency and accuracy of robot operations on various equipment such as household appliances. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present application is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Figure 1 It is a framework diagram of a ManualPlan model provided in an embodiment of the present application.

[0023] Figure 2 It is a flow chart of a robot operation method provided in an embodiment of the present application.

[0024] Figure 3 It is an alignment flow chart of a ManualPlan model provided in an embodiment of the present application (the structural model is available).

[0025] Figure 4 It is a robot operation schematic diagram (structural model is available) that combines ManualPlan and CAD model provided in an embodiment of the present application.

[0026] Figure 5It is a robot operation schematic diagram combining ManualPlan and OmniManip provided in an embodiment of the present application (the structural model is not available).

[0027] Figure 6 It is a structural block diagram of a robot provided in an embodiment of the present application.

[0028] Figure 7 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the implementation mode of this application. Obviously, the described implementation mode is only a part of the implementation mode of this application, not all the implementation modes. Based on the implementation mode in this application, all other implementation modes obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0030] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0031] With the rapid development of intelligent manufacturing and service robot technology, robots are increasingly used in daily life and industrial production. In the field of equipment operation and maintenance, robots can automatically perform complex operation tasks, improve efficiency and reduce human operation errors. However, related robot operation methods rely on pre-programmed action sequences or instant decision-making based on sensors. These methods usually require special programming for each specific device, resulting in insufficient adaptability and versatility when facing diverse and complex equipment. In addition, related methods lack efficient parsing and understanding capabilities when processing text and visual information in equipment manuals, making it difficult to automatically generate operation plans for specific tasks.

[0032] As an important document to guide users to use the equipment correctly, the equipment manual contains a wealth of operating steps, component information and safety precautions. However, how to convert these human-readable manual contents into robot-executable operating instructions is still an urgent problem to be solved. The research of related technologies in this regard is mostly focused on the combination of natural language processing (NLP) and computer vision (CV), but in practical applications, there are still the following shortcomings. First, most methods only focus on the analysis of text information, ignoring the visual elements such as graphics and tables contained in the manual, resulting in an incomplete understanding of equipment components and operating steps. Secondly, when matching the component information in the manual with the actual equipment, there is a lack of effective visual prompt mechanism, resulting in low accuracy of component identification and alignment, which in turn affects the correctness of operation planning. In addition, when performing operation planning, related methods often rely on predefined rules or templates, lack flexibility and intelligence, and are difficult to cope with complex and changeable operation tasks. In addition, when robots perform specific operation actions, they often lack a mechanism to effectively combine high-level task planning with low-level operation control, resulting in insufficient consistency and efficiency of operation.

[0033] In response to the above problems, this application proposes a robot operation method based on equipment manuals. This method combines optical character recognition (OCR), multimodal large language model (MLLM) and visual prompt mechanism to achieve comprehensive parsing of text information and visual information in the equipment manual, and generate accurate operation planning information. By aligning the operable parts in the actual device with the operable parts in the manual, the accurate execution of the operation steps is ensured. This method not only improves the robot's adaptability to diversified electrical equipment, but also enhances its execution efficiency and accuracy in complex tasks, and has broad application prospects.

[0034] Specifically, in order to implement robot operation planning based on the manual, the applicant proposed the ManualPlan model, which is used for manual-based task planning. For tasks involving robot execution, as an example, ManualPlan can be integrated as a high-level planning module of the relevant operation model to achieve expansion.

[0035] See also Figure 1 , Figure 1This is a framework diagram of a ManualPlan model provided in an embodiment of the present application. The ManualPlan model includes, for example, a manual extraction module, an operation planning module, and a component alignment module. ManualPlan can perform advanced planning to control the large model OmniManip of basic actions assisted by CAD (Computer Aided Design) or open vocabulary operations to use the device. These models can be used as the basic model of the CheckManual (check manual) data set. Figure 1 In the example, the target device is an oven, the target task is “heating bread in the upper oven”, the upper oven door and the upper oven temperature knob are the component names, open, close, and rotate forward 30° are the operating states, and component 1 and component 5 are the component identifiers.

[0036] It should be noted that although some embodiments of this document are described using robots operating home appliances as an example, this application is not only applicable to robots operating home appliances, but also to robots operating other appliances or devices other than home appliances. Although some embodiments of this document are described using manuals as an example, this application is not only applicable to robot operation planning based on manuals, but also to robot operation planning based on instructions other than manuals.

[0037] The implementation methods of the present application are described in detail below.

[0038] See also Figure 2 , Figure 2 It is a flow chart of a robot operation method provided in an embodiment of the present application.

[0039] An embodiment of the present application provides a robot operation method, which includes steps S101 to S103.

[0040] Step S101: Generate operation planning information for achieving a target task based on text information and / or visual information in a device manual in a document format corresponding to the target device; the operation planning information is used to indicate the component name and operation status of an operable component in the target device corresponding to at least one operation step.

[0041] Step S102: after receiving the observation image containing the target device, align the operable components in the target device with the operable components in the device specification to obtain a first correspondence between the component identification of the operable components in the target device and the component name of the operable components in the device specification.

[0042] Step S103: Based on the operation planning information and the first corresponding relationship, the robot is controlled to execute a corresponding operation state for the operable components corresponding to the operation step to implement the operation step.

[0043] As an example, the ManualPlan model introduces a new framework for high-level task planning for robotic operations, which requires the use of equipment manuals. Specifically, the model processes the manual content, generates a detailed plan, and aligns the operable parts in the actual equipment with the operable parts referenced in the manual.

[0044] In some embodiments, the document format of the device manual may be in PDF format. In other embodiments, the document format of the device manual may be in DOC format.

[0045] In some embodiments, before generating the operation planning information, the method may also include: converting the device manual in the document format corresponding to the target device into one or more page images; performing text recognition on the page images to obtain the text information, and / or performing visual element detection on the page images to obtain the visual information.

[0046] In some embodiments, performing text recognition on the page image to obtain the text information may include: performing text recognition on the page image using an OCR optical character recognition model to obtain the text information. The full name of OCR is Optical Character Recognition.

[0047] In some embodiments, performing visual element detection on the page image to obtain the visual information may include: performing visual element detection on the page image to obtain the visual information using a MLLM multimodal large language model, where MLLM stands for Multimodal Large Language Model.

[0048] In some embodiments, generating the operation planning information for achieving the target task according to the text information and / or visual information in the device manual in the document format corresponding to the target device may include: generating the operation planning information for achieving the target task according to the text information and / or visual information in the device manual in the document format corresponding to the target device using an LLM large language model. The full name of LLM is Large Language Model.

[0049] As an example, assume that the device manuals in the dataset include electrical appliance manuals, and the electrical appliance manuals are in PDF format, which is consistent with the real manuals. In the case where the PDF file cannot be directly processed by a large language model (which can be referred to as a large language model), the manual PDF file can be first divided into page-level images (which can be referred to as page images). Then, an OCR optical character recognition model (for example, the Paddle OCR model) is used to extract text content from these page images to obtain text information. In addition, the MLLM multimodal large language model is used to detect visual elements such as graphics and tables on each page image. In this way, text information and visual information at the page granularity of the electrical appliance manual can be obtained.

[0050] Through the extracted manual information, the LLM large language model is called to plan a step-by-step appliance operation process for a target task (e.g., a given task). For example, the large language model can specify the component function name (i.e., component name) and the operation status for each planning step (i.e., the planned operation step) in a dictionary format. Only the operable components mentioned in the manual can be used for planning. Sometimes, only some tasks are demonstrated in the manual, so the large language model can plan based on common sense of appliance use and the manual content. In practical applications, difficult predictions (i.e., in the case of failure to generate operation planning information) can trigger a regeneration request to regenerate the operation planning information for achieving the target task.

[0051] In order to achieve the alignment of the actual device (i.e., the target device) with the components of the device specification, the applicant adopts a visual cue mechanism called Set-of-Mark (SoM) to align the operable components of the actual device in the field of view with the operable components described in the device specification. SoM uses a segmentation model to divide the observed image containing the target device into different regions and assigns a digital mark to each region, which can significantly improve the visual positioning ability of MLLM. Specifically, the bounding boxes of the operable components are first predicted using a detection model (e.g., an open vocabulary object detection model Grounding-DINO), and these bounding boxes are used to prompt the segmentation model (e.g., Segment Anything segmentation model) to generate masks (i.e., masks) of the operable components. These masks use different display parameters, for example, different colors can be assigned to different masks with transparency, and then such masks are superimposed on the observed image. Afterwards, the component identification is marked on the mask of each operable component, for example, at the geometric center of the mask. Then, the observed image and the device image in the device manual are provided to the MLLM, and the MLLM is required to identify the mapping relationship (i.e., the first correspondence relationship) between the mask ID (i.e., the component identification on the mask) and the component function name (i.e., the component name). In this way, the component alignment between the observed image and the device manual is achieved (i.e., the component alignment between the operable components of the actual device in the observed image and the operable components in the device manual).

[0052] For robot manipulation tasks based on device specifications and structural models (e.g., CAD models), even though the structural model is available, the component function names (i.e., component names) of the device specifications still need to be aligned with the component model identifiers of the operable components on the structural model. Therefore, a similar SoM-based approach can be used to achieve this alignment with MLLM, obtaining a second correspondence between component model identifiers and component names.

[0053] In the robot operation task purely based on the specification, the structural model is not available, while in the robot operation task of device specification-structural model-actual device alignment, component alignment has been achieved. For the latter, this alignment is achieved through two alignment steps that can be interchanged in sequence, namely, aligning the operable components in the structural model with the operable components in the device specification (obtaining a second corresponding relationship), and aligning the operable components of the actual device in the observed image with the operable components in the structural model (obtaining a third corresponding relationship). The component alignment process will be specifically introduced below, including the case where the structural model is available and the case where the structural model is not available. As an example, the reason why the structural model is unavailable is, for example, that the structural model has not been received, or that the received structural model does not meet the usage requirements.

[0054] See also Figure 3 , Figure 3 This is an alignment flow chart of a ManualPlan model provided in an embodiment of the present application (the structural model is available). Figure 3 In the example, the target device is a coffee machine, the target task is "prepare a small cup of coffee", the brew button (i.e., brew button), cup size inbox (i.e., cup size selector), power button, and top cover are the component names in the device manual (e.g., device manual), part 0, part 1, part 2, and part 3 are the component model identifiers in the structural model (e.g., CAD model), and open, close, press once, and press twice are the operation states. It can be seen that the ManualPlan model correctly plans tasks according to the given device manual (e.g., device manual) and realizes the component alignment between the device manual and the structural model. After realizing the component alignment between the device manual, the structural model, and the actual device, in the final planning, each step uses the component model identifier as the target and specifies the specific operation action. It should be noted that after the component alignment between the device manual, the structural model and the actual device is achieved, any two of the component identification (of the operable component of the target device in the observation image), the component model identification (of the operable component in the structural model), and the component name (of the operable component in the device manual) all satisfy a one-to-one correspondence. Therefore, the robot operation control based on the component identification is similar to the robot operation control based on the component model identification, and this application does not limit this.

[0055] In some embodiments based on the device specification and the structural model, the aligning of the operable components in the target device with the operable components in the device specification to obtain a first correspondence between component identifications of the operable components in the target device and component names of the operable components in the device specification may include: when the corresponding structural model of the target device is available, aligning the operable components in the structural model with the operable components in the device specification to obtain a second correspondence between component model identifications of the operable components in the structural model and component names of the operable components in the device specification; aligning the target device with the structural model through the object posture of the target device in the observed image to obtain a third correspondence between component identifications of the operable components in the target device and component model identifications of the operable components in the structural model; and determining a first correspondence between component identifications of the operable components in the target device and component names of the operable components in the device specification based on the second correspondence and the third correspondence.

[0056] In some embodiments, the structural model may include a three-dimensional structural model and / or a two-dimensional structural model.

[0057] In some embodiments, the structural model may be a CAD model.

[0058] In some embodiments, the visual information may include a device image, and the device image may include component names. As an example, a graphic, diagram, or image in the visual information may be referred to as a device image. Since the visual information is obtained by performing visual element detection on a page image corresponding to the device manual, the device manual will mostly use a dotted line annotation method (or other visual annotation methods) to mark multiple component names of the device for the convenience of human reading, so the device image, for example, includes component names.

[0059] In some embodiments, aligning the operable components in the structural model with the operable components in the device specification to obtain a second correspondence between the component model identifier of the operable component in the structural model and the component name of the operable component in the device specification may include: marking the component model identifier of the operable component on the corresponding operable component in the rendered image of the structural model; inputting the rendered image with the component model identifier and the device image containing the component name into the MLLM multimodal large language model to identify the second correspondence between the component model identifier and the component name.

[0060] In some embodiments, marking the component model identifier of the operable component on the corresponding operable component in the rendered image of the structural model may include: using a detection model to detect the bounding box of the operable component in the rendered image; using the bounding box as an input of a segmentation model to generate a mask of the operable component; superimposing the mask of the operable component on the corresponding operable component in the rendered image; marking the component model identifier of the operable component on the mask of the corresponding operable component in the rendered image. Wherein, using the bounding box as an input of a segmentation model to generate the mask of the operable component, for example, means inputting the bounding box into the segmentation model so that the segmentation model performs a segmentation operation to obtain the mask of the operable component.

[0061] In some embodiments, masks of different operable components may correspond to different display parameters. In some embodiments, the display parameters may include one or more of color, transparency, font style, font size, font weight, brightness, contrast, saturation, exposure, hue, brightness, grayscale, color temperature, and scaling.

[0062] In some embodiments, masks for different operable components may correspond to different colors and / or transparencies.

[0063] In some embodiments, in the rendered image, the component model identifier of the operable component may be located at the geometric center of the corresponding mask.

[0064] In some embodiments, aligning the operable components in the target device with the operable components in the device manual to obtain a first correspondence between component identifications of the operable components in the target device and component names of the operable components in the device manual may include: marking the component identifications of the operable components on the corresponding operable components in the observed image; and inputting the observed image with the component identifications and the device image containing the component names into the MLLM multimodal large language model to identify the first correspondence between the component identifications and the component names.

[0065] In some embodiments, marking the component identification of the operable component on the corresponding operable component in the observed image may include: using a detection model to detect the bounding box of the operable component in the observed image; using the bounding box as an input of a segmentation model to generate a mask of the operable component; superimposing the mask of the operable component on the corresponding operable component in the observed image; marking the component identification of the operable component on the mask of the corresponding operable component in the observed image.

[0066] In some embodiments, in the observed image, the component identification of the operable component may be located at the geometric center of the corresponding mask.

[0067] When performing operations on the target device, ManualPlan can be used as a high-level planner to design different robot operation strategies for the cases where the structural model is available and the cases where the structural model is not available.

[0068] See also Figure 4 , Figure 4 This is a robot operation schematic diagram combining ManualPlan and CAD model provided in the embodiment of the present application (the structural model is available). Figure 4 In the example, the target device is a coffee machine, the target task is "prepare a cup of coffee", parts 1 to 8 are component model identifiers, the power button, valve stem button, volume button, clean button, and single cup button are component names, and pressing once is the operation state. ManualPlan performs correct device component alignment and task planning based on the object pose predicted by FoundationPose (foundation pose model) and CAD model. Use this information to design atomic actions, such as pressing buttons and turning knobs, to perform corresponding operation actions according to the task plan.

[0069] In some embodiments, controlling the robot to execute a corresponding operation state for the operable component corresponding to the operation step based on the operation planning information and the first correspondence may include: determining the component model identifier and operation state corresponding to one or more operation steps based on the operation planning information and the first correspondence when the corresponding structural model of the target device is available; designing the operation action for the robot to execute the corresponding operation state for the corresponding operable component according to the component model identifier and operation state corresponding to the operation step and the target structural features of the corresponding operable component in the structural model; and controlling the robot to execute the operation action for the corresponding operation state for the operable component.

[0070] In the above embodiment, in the operation planning information, the operation steps correspond to the component names and operation states. After the component alignment between the device specification-structure model-actual device is achieved, the component names correspond to the component model identifiers, so the component model identifiers and operation states corresponding to the operation steps can be determined. When the structure model is available, the target structural features of the operable components in the structure model can be used to design the robot's operation actions (e.g., atomic actions).

[0071] In some embodiments, the target structural feature of the operable component may be determined according to the component type of the operable component.

[0072] In some embodiments, the component types may include one or more of sliders, buttons, and knobs.

[0073] In some embodiments, the target structural feature may include one or more of a joint, a hinge, a door, a slider, and a surface.

[0074] As an example, assume that the structural model adopts a CAD model. When the CAD model is available, the 6-DOF pose of the target device observed in the camera frame is estimated as the object pose of the target device in the observed image. With this estimated object pose, the observed actual device can be aligned with its CAD model, and the positions and axis directions of all joints become available. As an example, corresponding basic actions can be designed for different operable parts to synthesize CAD-assisted operation strategies. For example, for doors and parts with sliders (e.g., screens, sliders, covers), the 3D bounding box of the operable parts is provided to constrain the robot's part-level grasping pose. Then, the gripper pose in the motion sequence can be calculated based on the relevant information of the joints and the operation method. Among them, the gripper is an example of the end effector of the robot. For the button part, its surface geometric center can be used as the contact point, and the closed gripper can be controlled to push the button from the normal direction of the contact point. For the knob part, the open gripper can be controlled to approach the knob from the normal direction of its surface geometric center, and the gripper can be closed for grasping. After the gripper successfully grasps the knob, the gripper will rotate the planned angle around the normal of its surface geometric center.

[0075] See also Figure 5 , Figure 5 This is a robot operation diagram combining ManualPlan and OmniManip provided in an embodiment of the present application (the structural model is not available). Figure 5 In the example, the target device is a microwave oven, the target task is "check the inside of the microwave oven", "1" and "9" are part identifiers, the front door and rotatable tray are part names, and open and closed are operation states. ManualPlan achieves correct alignment of equipment parts and task planning without the assistance of CAD models. OmniManip further successfully executes two-step operation actions.

[0076] In some embodiments, controlling the robot to execute corresponding operation states for the operable components corresponding to the operation steps based on the operation planning information and the first correspondence may include: when the corresponding structural model of the target device is not available, determining the component identification and operation states corresponding to one or more operation steps based on the operation planning information and the first correspondence; processing the component identification and operation states corresponding to the operation steps to obtain operation information in a language format; and using an open vocabulary operation model to process the operation information to control the robot to execute operation actions for the corresponding operation states for the operable components.

[0077] The above embodiment provides a robot operation strategy when a structural model is unavailable. As an example, an open-vocabulary manipulation model can be integrated into ManualPlan to complete the operation task when the structural model is unavailable (for example, there is no CAD model). Among them, the open-vocabulary manipulation model is, for example, OmniManip. In this framework, the cooperation of ManualPlan and OmniManip models is similar to the cerebrum and cerebellum in the human brain. ManualPlan reads the equipment manual (for example, the equipment manual), performs high-level planning, and aligns the observed operable parts with the operable parts described in the manual. Then, ManualPlan locates the operable parts and displays the corresponding operation information in a language format, and the operation information is transmitted to OmniManip, for example, through camera RGB-D observation, to control the robot to perform low-level operation actions on the target device (for example, a microwave oven). In camera RGB-D observation, RGB represents information of three colors: red, green, and blue, providing a color image; D represents depth information, providing distance data between the object and the camera. For example, an RGB-D camera is used to obtain RGB-D data of a target device (such as a microwave oven), for example, including a color image and a depth image. ManualPlan transmits RGB-D data and operation information to OmniManip, which controls the robot to perform specific, basic operations on the target device, such as pressing a button or turning a knob, based on the received RGB-D data and operation information. In this way, the robot can intelligently perform specific operations on complex equipment based on environmental perception and operation status.

[0078] See also Figure 6 , Figure 6 It is a structural block diagram of a robot provided in an embodiment of the present application.

[0079] An embodiment of the present application also provides a robot, comprising a control module, wherein the control module is used to execute any of the above methods to implement one or more operation steps.

[0080] In some embodiments, the robot may be a humanoid robot (also known as a bipedal robot, a humanoid robot), a quadrupedal robot, or a wheeled robot with a robotic arm.

[0081] In some embodiments, the robot may be a multi-jointed robot.

[0082] In some embodiments, the robot may include a robot body, a left arm, a right arm, a head, and a chassis. The left arm and the right arm may be collectively referred to as a robotic arm. However, the number of robotic arms may also be one or more, and the above embodiment does not limit this.

[0083] In some embodiments, the robot may include one or more of an image sensor, a laser sensor, an odometer, an IMU, an angle encoder, a torque sensor, and a PIR sensor.

[0084] In some embodiments, the image sensor may include a camera (or a camera module). As an example, cameras may be provided at the end joint of the left arm, the end joint of the right arm, the head, and the chassis, respectively, that is, a left hand camera, a right hand camera, a head camera, and a chassis camera.

[0085] In some embodiments, the laser sensor may include a 2D laser sensor and / or a 3D laser sensor. Taking a laser radar as an example, the laser radar may be set on the head of the robot, however, the above embodiment is not limited to this, and the laser radar may also be set on other parts of the robot.

[0086] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0087] An embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0088] The computer program product may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present application is not limited thereto, and the computer program product may be any combination of one or more computer readable media.

[0089] See also Figure 7 , Figure 7 It is a structural block diagram of a computer device provided in an embodiment of the present application.

[0090] An embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0091] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0092] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.

[0093] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.

[0094] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solution of the embodiment of the present application is implemented by software or firmware, the computer program for implementing the solution of the embodiment of the present application can be stored in the processor 120 and executed by the processor 120.

[0095] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.

[0096] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor, etc.

[0097] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 120 or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor 120 for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0098] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input output system (BIOS), application software, etc.

[0099] The operating system is used to manage the hardware and / or software resources of computer devices and is the core and cornerstone of computer devices. The operating system needs to handle basic tasks such as managing and configuring memory, determining the priority of system resource supply and demand, controlling input and output devices, operating networks, and managing file systems. In order to facilitate user operations, most operating systems will provide an operating interface for users to interact with the system.

[0100] BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and application programs. In some implementations, BIOS can also monitor and display processor temperature and execute functions such as adjusting temperature protection strategies.

[0101] Application software is also called application program, which can be understood as software written for a specific application purpose of the user. It is one of the main categories of computer software. For example, application software can be a program used to achieve power control, temperature management, etc.

[0102] It should be understood that the specific examples in this specification are only intended to help those skilled in the art to better understand the implementation methods of the present application, rather than to limit the scope of protection of the present application.

[0103] It is understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0104] It can be understood that the various implementation modes described in this specification can be implemented individually or in combination, and this application is not limited to this.

[0105] Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0106] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each embodiment described above can refer to the corresponding process in other embodiments and will not be repeated here.

[0108] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0110] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0111] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0112] The above are only specific implementations of this specification, but the protection scope of this application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this specification, which should be included in the protection scope of this specification. Therefore, the protection scope of this application should be based on the protection scope of the claims.

Claims

1. A robot operation method, characterized in that: The method comprises: Generate operation planning information for achieving the target task according to text information and / or visual information in the device manual in the document format corresponding to the target device; the operation planning information is used to indicate the component name and operation status of the operable component in the target device corresponding to at least one operation step; After receiving the observed image including the target device, aligning the operable component in the target device with the operable component in the device specification to obtain a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device specification; Based on the operation planning information and the first corresponding relationship, the robot is controlled to execute corresponding operation states for the operable components corresponding to the operation steps to implement the operation steps.

2. The robot operation method according to claim 1, characterized in that: Before generating the operation plan information, the method further includes: Converting a device manual in a document format corresponding to the target device into one or more page images; Text recognition is performed on the page image to obtain the text information, and / or visual element detection is performed on the page image to obtain the visual information.

3. The robot operation method according to claim 2, characterized in that: The performing text recognition on the page image to obtain the text information includes: performing text recognition on the page image using an OCR optical character recognition model to obtain the text information; and / or, The performing visual element detection on the page image to obtain the visual information includes: performing visual element detection on the page image using an MLLM multimodal large language model to obtain the visual information.

4. The robot operation method according to claim 1, characterized in that: The step of aligning the operable component in the target device with the operable component in the device specification to obtain a first correspondence between the component identifier of the operable component in the target device and the component name of the operable component in the device specification includes: When a structural model corresponding to the target device is available, aligning the operable components in the structural model with the operable components in the device specification to obtain a second correspondence between component model identifiers of the operable components in the structural model and component names of the operable components in the device specification; Aligning the target device with the structural model according to the object posture of the target device in the observed image, so as to obtain a third corresponding relationship between a component identifier of an operable component in the target device and a component model identifier of the operable component in the structural model; Based on the second correspondence and the third correspondence, a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device manual is determined.

5. The robot operation method according to claim 1, characterized in that: The visual information includes a device image, the device image includes a component name, and the aligning the operable component in the target device with the operable component in the device specification to obtain a first correspondence between the component identification of the operable component in the target device and the component name of the operable component in the device specification includes: marking a component identification of the operable component on a corresponding operable component in the observed image; The observed image with the part identification and the device image containing the part name are input into the MLLM multimodal large language model to identify the first correspondence between the part identification and the part name.

6. The robot operation method according to claim 5, characterized in that: The step of marking the component identification of the operable component on the corresponding operable component in the observed image comprises: detecting a bounding box of an operable component in the observed image using a detection model; Using the bounding box as an input to a segmentation model to generate a mask of the operable component; superimposing the mask of the operable component on the corresponding operable component in the observed image; The component identification of the operable component is marked on the mask of the corresponding operable component in the observed image.

7. The robot operation method according to claim 6, characterized in that: Masks of different operable components correspond to different colors and / or transparencies; and / or, In the observed image, the component identification of the operable component is located at the geometric center of the corresponding mask.

8. The robot operation method according to claim 1, characterized in that: The controlling the robot to execute a corresponding operation state for an operable component corresponding to the operation step based on the operation planning information and the first corresponding relationship includes: When the structural model corresponding to the target device is available, determining component model identifiers and operation states corresponding to one or more operation steps based on the operation plan information and the first corresponding relationship; According to the component model identifier and the operation state corresponding to the operation step and the target structural features of the corresponding operable component in the structural model, designing the robot to perform an operation action of the corresponding operation state on the corresponding operable component; The robot is controlled to perform an operation action of a corresponding operation state on the operable component.

9. The robot operation method according to claim 8, characterized in that: The target structural feature of the operable component is determined according to the component type of the operable component.

10. The robot operation method according to claim 1, characterized in that: The controlling the robot to execute a corresponding operation state for an operable component corresponding to the operation step based on the operation planning information and the first corresponding relationship includes: In the case where the structural model corresponding to the target device is unavailable, determining component identifications and operation states corresponding to one or more operation steps based on the operation plan information and the first corresponding relationship; Processing the component identification and operation status corresponding to the operation step to obtain operation information in a language format; The operation information is processed using an open vocabulary operation model to control the robot to perform an operation action of a corresponding operation state on the operable component.

11. A robot, characterized in that: The robot comprises a control module, and the control module is used to execute the method of any one of claims 1 to 10 to achieve one or more operation steps.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 10 when executing the computer program.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images