A device-controlled robot based on reading the instruction manual and its control method

By obtaining the equipment manual and using a multimodal model to calibrate buttons and a large language model to decompose instructions, the problem of low adaptability of the robot in controlling the equipment was solved, enabling rapid adaptation to the operation of different equipment and reducing the setup cost of new equipment.

CN118418121BActive Publication Date: 2026-11-17HANGZHOU DEXIO ROBOTS CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410514728.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-26
Publication Date
2026-11-17
Estimated Expiration
2044-04-26

AI Technical Summary

Technical Problem

Existing equipment has low adaptability to control robots, and new equipment has high setup costs and is difficult to quickly adapt to the control requirements of different equipment.

Method used

By obtaining the equipment manual, using a multimodal model to calibrate the buttons, segmenting and recognizing button functions, combining a large language model to decompose control instructions, and using visual matching technology to operate the new equipment buttons.

Benefits of technology

It reduces the setup cost of new equipment and improves the versatility of the robot, enabling it to quickly adapt to the operation of different devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118418121B_ABST
    Figure CN118418121B_ABST
Patent Text Reader

Abstract

This application belongs to the field of equipment control robots, and specifically relates to an equipment control robot based on reading an instruction manual and its control method. The method includes: obtaining an instruction manual and calibrating the buttons in the manual; obtaining control commands and decomposing them into a sequence of button operations on the device to be controlled; obtaining position information and moving to the device based on the position information; obtaining an image of the device to be controlled and visually matching the buttons in the instruction manual with the buttons in the image to obtain a button matching result; and sequentially operating the buttons on the device to be controlled according to the button operation sequence and the button matching result. This instruction manual-based equipment control robot control method reduces the setup cost of new equipment, enables the equipment control robot to quickly meet the control needs of different devices, and greatly improves the versatility of the equipment control robot.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of equipment control robots, and specifically relates to an equipment control robot based on reading the instruction manual and its control method. Background Technology

[0002] Equipment control robots typically include a robotic arm, which allows users to control the equipment by operating buttons on the device.

[0003] In the existing technology, before a device control robot can operate a new device, it is necessary to pre-enter the location of the buttons on the device and pre-set the control program. Because the process is cumbersome and the setup cost of the new device is high, a device control robot can usually only adapt to one type of device, resulting in low adaptability.

[0004] Therefore, it is desirable to have a technical solution to overcome or at least mitigate one of the aforementioned defects of the prior art. Summary of the Invention

[0005] The purpose of this application is to provide a device-controlled robot and its control method based on reading the instruction manual, so as to solve the problem of low adaptability of the device-controlled robot in the prior art.

[0006] The technical solution of this application is:

[0007] The first aspect of this application provides a device-controlled robot control method based on reading a user manual, comprising:

[0008] Obtain the device instruction manual and calibrate the buttons in the instruction manual;

[0009] Acquire control commands and decompose the control commands into a sequence of button operations for the device to be controlled;

[0010] Obtain location information and move to the device to be controlled based on the location information;

[0011] Acquire an image of the device to be operated, and visually match the buttons in the device manual with the buttons in the image of the device to be operated to obtain the button matching result;

[0012] The buttons on the device to be controlled are operated sequentially according to the button operation sequence and the button matching result.

[0013] In at least one embodiment of this application, calibrating the buttons in the device manual includes:

[0014] The operating interface in the device manual was located using a multimodal model;

[0015] The buttons in the user interface are segmented using a segmentation model;

[0016] The function of each button is identified through a multimodal model, and all buttons and their corresponding functions are saved to the database.

[0017] In at least one embodiment of this application, the multimodal model adopts the GPT model.

[0018] In at least one embodiment of this application, the segmentation model adopts the Segment Anything Model.

[0019] In at least one embodiment of this application, the step of decomposing the control command into a sequence of button operations on the device to be controlled includes:

[0020] The control instructions are decomposed into a sequence of button operations on the device to be controlled using a large language model, wherein the large language model adopts the GPT model.

[0021] A second aspect of this application provides a device control robot based on reading an instruction manual. Based on the device control robot control method described above, the device control robot includes: a controller, a movement module, an image acquisition module, and an execution module, wherein...

[0022] The controller includes:

[0023] The equipment manual calibration unit is used to acquire the equipment manual and calibrate the buttons in the equipment manual.

[0024] A control instruction decomposition unit is used to acquire control instructions and decompose the control instructions into a sequence of button operation sequences for the device to be controlled.

[0025] A mobile control unit is used to acquire location information and control the mobile module to move to the device to be operated based on the location information;

[0026] A button matching unit is used to control the image acquisition module to acquire an image of the device to be operated, and to visually match the buttons in the device manual with the buttons in the image of the device to be operated to obtain a button matching result.

[0027] An operation control unit is used to control the execution module to operate the buttons on the device to be controlled sequentially according to the button operation sequence and the button matching result.

[0028] In at least one embodiment of this application, the mobile module includes a mobile chassis and a mounting bracket disposed on the mobile chassis.

[0029] In at least one embodiment of this application, the image acquisition module is a camera mounted on the mounting bracket.

[0030] In at least one embodiment of this application, the execution module is a robotic arm mounted on the mounting bracket.

[0031] The invention has at least the following beneficial technical effects:

[0032] The device control method for operating a robot based on reading the instruction manual disclosed in this application first uploads and calibrates the instruction manual before operating a new device. Then, the control commands are decomposed into a sequence of button operations for the device to be operated. After receiving the location information of the device to be operated, the robot moves to the device, acquires an image of the device, which contains multiple buttons, and uses visual matching technology to match the buttons in the instruction manual with the buttons in the image of the device to be operated, obtaining the button matching results. Finally, according to the button operation sequence decomposed from the control commands and the button matching results between the instruction manual and the image of the device to be operated, the buttons on the device to be operated are operated sequentially to complete the operation of the device. Attached Figure Description

[0033] Figure 1 This is a flowchart of a device control robot control method based on reading the instruction manual, according to one embodiment of this application;

[0034] Figure 2 This is a device instruction manual calibration flowchart of one embodiment of this application;

[0035] Figure 3 This is a schematic diagram of the device control robot architecture according to one embodiment of this application.

[0036] in:

[0037] 100-Controller; 101-Equipment instruction calibration unit; 102-Control command decomposition unit; 103-Motion control unit; 104-Button matching unit; 105-Operation control unit; 200-Motion module; 300-Image acquisition module; 400-Execution module. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0039] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the scope of protection of this application.

[0040] The following is in conjunction with the appendix Figures 1 to 3 This application will be described in further detail.

[0041] The first aspect of this application provides a device control method for operating a robot based on reading a user manual, such as... Figure 1 As shown, it includes the following steps:

[0042] Obtain the equipment manual and calibrate the buttons in the manual;

[0043] Acquire control commands and break them down into button operation sequences for the device to be controlled;

[0044] Obtain location information and move to the device to be controlled based on the location information;

[0045] Acquire an image of the device to be controlled, and visually match the buttons in the device manual with the buttons in the image of the device to be controlled to obtain the button matching results;

[0046] The buttons on the device to be controlled are operated sequentially according to the button operation sequence and button matching results.

[0047] The device control robot method based on reading the instruction manual in this application requires that, when the device control robot needs to learn to operate a new device, the device instruction manual must first be uploaded, and the buttons in the instruction manual must be calibrated. In a preferred embodiment of this application, such as... Figure 2 As shown, the process of calibrating the buttons in the equipment manual is as follows:

[0048] The operating interface in the equipment manual was located using a multimodal model;

[0049] The buttons in the user interface are segmented using a segmentation model;

[0050] The function of each button is identified through a multimodal model, and all buttons and their corresponding functions are saved to the database.

[0051] Understandably, the multimodal model can use the GPT model or other multimodal models that achieve the desired effect. The segmentation model can use the Segment Anything Model (SAM) or other similar models. When calibrating the equipment, a user-friendly interface can be provided to assist the user in the calibration process.

[0052] In a preferred embodiment of this application, the multimodal model used is gpt-4v. The training process of gpt-4v adds processing of image data. It uses a multimodal pre-training method to train text and image data together, enabling it to understand the relationship between text and images and generate text related to images. gpt-4v can quickly locate the operating interface in the device manual, then the Segment Anything Model (SAM) is used to segment the buttons from the interface, and finally gpt-4v quickly identifies the function of each button before saving each button and its corresponding function.

[0053] In a preferred embodiment of this application, the control instructions are decomposed into a sequence of button operations on the device to be controlled by a large language model. The large language model can be a GPT model, such as GPT-3.5V or GPT-4V.

[0054] The device control method for operating a robot based on reading the instruction manual disclosed in this application first uploads and calibrates the instruction manual before operating a new device. Then, the control commands are decomposed into a sequence of button operations for the device to be operated. After receiving the location information of the device to be operated, the robot moves to the device, acquires an image of the device, which contains multiple buttons, and uses visual matching technology to match the buttons in the instruction manual with the buttons in the image of the device to be operated, obtaining the button matching results. Finally, according to the button operation sequence decomposed from the control commands and the button matching results between the instruction manual and the image of the device to be operated, the buttons on the device to be operated are operated sequentially to complete the operation of the device.

[0055] Based on the aforementioned method for controlling a robot by reading an instruction manual, a second aspect of this application provides a robot for controlling a robot by reading an instruction manual, such as... Figure 3 As shown, the device controlling the robot includes: a controller 100, a movement module 200, an image acquisition module 300, and an execution module 400.

[0056] Specifically, the controller 100 includes a device instruction calibration unit 101, a control command decomposition unit 102, a movement control unit 103, a button matching unit 104, and an operation control unit 105.

[0057] The device manual calibration unit 101 is used to acquire the device manual and calibrate the buttons in the manual. The device manual calibration unit 101 of the controller 100 is equipped with a device manual upload interface. The upload interface includes a field for uploading device manual files, which can be in PDF or PNG format. The upload interface also includes auxiliary information such as the device name. After calibration, all buttons in the device manual and their corresponding functions are obtained.

[0058] The control instruction decomposition unit 102 is used to acquire control instructions and decompose the control instructions into a sequence of button operation on the device to be controlled.

[0059] The motion control unit 103 is used to acquire location information and control the motion module 200 to move to the device to be operated based on the location information.

[0060] The button matching unit 104 is used to control the image acquisition module 300 to acquire the image of the device to be operated, and to visually match the buttons in the device manual with the buttons in the image of the device to be operated to obtain the button matching result.

[0061] The operation control unit 105 is used to control the execution module 400 to operate the buttons on the device to be controlled in sequence according to the button operation sequence and button matching result.

[0062] The device control robot based on the instruction manual of this application integrates multiple data processing models in the controller 100 to process image and text data. Furthermore, the controller 100 controls the movement module 200 via the movement control unit 103 to move the device control robot, controls the image acquisition module 300 via the button matching unit 104 to acquire images of the device to be controlled, and controls the execution module 400 via the operation control unit to operate the buttons on the device to be controlled. In a preferred embodiment of this application, the movement module 200 of the device control robot includes a mobile chassis and a mounting frame mounted on the mobile chassis. The image acquisition module 300 is a camera mounted on the mounting frame, and the execution module 400 is a robotic arm mounted on the mounting frame.

[0063] The device control robot based on reading the instruction manual of this application only needs to upload and calibrate the instruction manual of the device before it needs to operate a new device. After receiving the control command and the location information of the device to be operated, the device control robot can automatically move to the device to be operated and operate the buttons on the device in sequence to complete the operation of the device.

[0064] The device control robot based on reading the instruction manual proposed in this application can reduce the setup cost of new equipment, enable the device control robot to quickly meet the control needs of different equipment, and greatly improve the versatility of the device control robot.

[0065] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for controlling a robot based on reading an instruction manual, characterized in that, include: Obtain the device instruction manual and calibrate the buttons in the instruction manual; Acquire control commands and decompose the control commands into a sequence of button operations for the device to be controlled; Obtain location information and move to the device to be controlled based on the location information; Acquire an image of the device to be operated, and visually match the buttons in the device manual with the buttons in the image of the device to be operated to obtain the button matching result; The buttons on the device to be controlled are operated sequentially according to the button operation sequence and the button matching result; The calibration of the buttons in the device manual includes: The operating interface in the device manual was located using a multimodal model; The buttons in the user interface are segmented using a segmentation model; The function of each button is identified through a multimodal model, and all buttons and their corresponding functions are saved to the database.

2. The device control method for operating a robot based on reading the instruction manual according to claim 1, characterized in that, The multimodal model adopts the GPT model.

3. The device control method for operating a robot based on reading the instruction manual according to claim 1, characterized in that, The segmentation model adopted is the Segment Anything Model.

4. The device control method for operating a robot based on reading the instruction manual according to claim 1, characterized in that, The step of decomposing the control command into a sequence of button operations for the device to be controlled includes: The control instructions are decomposed into a sequence of button operations on the device to be controlled using a large language model, wherein the large language model adopts the GPT model.

5. A device control robot based on reading an instruction manual, comprising the device control method for reading an instruction manual as described in any one of claims 1 to 4, characterized in that, The device controlling the robot includes: a controller, a movement module, an image acquisition module, and an execution module, wherein, The controller includes: The equipment manual calibration unit is used to acquire the equipment manual and calibrate the buttons in the equipment manual. A control instruction decomposition unit is used to acquire control instructions and decompose the control instructions into a sequence of button operation sequences for the device to be controlled. A mobile control unit is used to acquire location information and control the mobile module to move to the device to be operated based on the location information; A button matching unit is used to control the image acquisition module to acquire an image of the device to be operated, and to visually match the buttons in the device manual with the buttons in the image of the device to be operated to obtain a button matching result. An operation control unit is used to control the execution module to operate the buttons on the device to be controlled sequentially according to the button operation sequence and the button matching result.

6. The device-controlled robot based on reading the instruction manual according to claim 5, characterized in that, The mobile module includes a mobile chassis and a mounting bracket mounted on the mobile chassis.

7. The device-controlled robot based on reading the instruction manual according to claim 6, characterized in that, The image acquisition module is a camera mounted on the mounting bracket.

8. The device-controlled robot based on reading the instruction manual according to claim 7, characterized in that, The execution module is a robotic arm mounted on the mounting bracket.

Citation Information

Patent Citations

  • Intelligent service robot for autonomously operating box type elevator

    CN114505840A

  • Product specification element extraction method, device, equipment, medium and program product

    CN115862025A