Intelligent mechanical arm operation method and system based on multi-mode large visual language model

Abstract

The invention discloses a mechanical arm operation method based on a multi-mode large visual language model, which comprises the following steps of: firstly, acquiring image data of a mechanical arm operation scene through a visual sensor, and acquiring a user instruction in combination with a voice recognition module or a text input module; and inputting the natural language instruction and the visual scene data into a multi-modal large visual language model for multi-modal task analysis. The model identifies the category, position and attribute of a target object in a scene, and generates a specific operation sequence in combination with an input instruction. And then, according to the operation sequence generated through analysis, the system directly drives the mechanical arm to execute corresponding tasks including specific actions such as grabbing, moving and placing of the target object, and in the task execution process, the mechanical arm sequentially completes the actions according to the operation sequence generated in advance. The problems that in the prior art, a mechanical arm is inaccurate in complex task instruction analysis, poor in operation flexibility and adaptability and poor in real-time response capacity in a dynamic environment are effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent robotic arm operation, and more specifically, relates to an intelligent robotic arm operation method and system based on a multimodal large visual language model. Background Art

[0002] With the rapid development of artificial intelligence and robotics, the demand for human-machine collaboration in modern industry and other application fields is increasing. Traditional robotic systems often rely on fixed control algorithms and manual programming, lacking flexibility and intelligence.

[0003] Specifically, existing robot systems can generally be divided into two categories: (1) Robots based on control technology, which mainly rely on control algorithms to perform tasks. Such robots usually work according to predetermined programs and are divided into open-loop control and closed-loop control. Open-loop control robots do not rely on feedback mechanisms, and there is no real-time environmental data input during the execution process. They are suitable for scenarios where tasks are simple and do not require precise adjustments; closed-loop control systems adjust the behavior of robots through real-time feedback, and are common in application fields that require precision and flexibility, such as industrial automation or precision operations; (2) Robots based on perception technology, which rely on various sensors to obtain environmental information, including vision, sound, touch, etc., and then respond accordingly; vision-oriented robots process visual data through image recognition technology, while hearing-oriented robots interact with the environment through voice recognition technology; different perception capabilities enable robots to respond flexibly in a variety of application scenarios and make decisions or actions;

[0004] However, both of the above robots have some drawbacks that cannot be ignored:

[0005] (1) They all rely on predetermined programs or preset rules and cannot understand complex or natural language instructions. This limits the depth of interaction between robots and humans, making them inadequate in unstructured tasks;

[0006] (2) Control-based robots lack perception technology, and perception-based robots are usually unable to process information integrated from multiple modalities. As a result, when faced with complex dynamic environments, they are prone to mission failure due to insufficient data or modality limitations.

[0007] (3) Both rely on fixed rules and preset models for path planning and task execution and lack dynamic adjustment capabilities. Therefore, when faced with real-time changing environments or task requirements, planning efficiency is low and error-prone.

[0008] (4) Due to the lack of language processing technology, both robots are unable to understand human semantic content and have difficulty adapting to contextual changes or ambiguous expressions in human-computer interaction. Therefore, this interaction mode not only affects the user experience, but also limits the wide application of robots. Summary of the invention

[0009] In view of the above defects or improvement needs of the prior art, the present invention provides an intelligent robotic arm operation method and system based on a multimodal large visual language model, the purpose of which is to solve the technical problem that the existing robotic arm operation method based on preset rules or traditional algorithms is difficult to efficiently parse complex task instructions and adapt to diversified scenarios due to the lack of natural language understanding ability and multimodal information comprehensive processing ability, resulting in insufficient adaptability to environmental changes. The existing methods also have problems such as insufficient operation fluency in dynamic environments, slow response to environmental changes, and difficulty in meeting the real-time interaction requirements in complex scenarios. The existing system also relies on manually configured multi-step process operation logic, low task switching efficiency and complex implementation, and difficulty in meeting flexibility and generalization requirements.

[0010] To achieve the above object, according to one aspect of the present invention, a method for operating an intelligent robotic arm based on a multimodal large visual language model is provided, comprising the following steps:

[0011] (1) Initialize the current operating environment of the robot arm to obtain the operating environment of the robot arm after initialization.

[0012] (2) receiving the task instruction T input by the user according to the operation environment of the robot arm after initialization in step (1) input , task instruction T input Included are the names of each starting point object and each end point object that the user needs to operate.

[0013] (3) The task instruction T obtained in step (2) input Parse to obtain the parsing result, which includes the subtask sequence set S subtasks And its corresponding operation function call sequence set F call .

[0014] (4) The operation function call sequence set F obtained according to step (3) call , drive the robot arm to move to the operating environment of the robot arm after initialization according to step (1), start the camera in the visual module, and use the camera to obtain and save the image I in the operating environment after a preset delay, and record the saving time and saving path of the image I.

[0015] (5) According to step (3), the subtask sequence set S is obtained subtasks, and use the multimodal large visual language model LVLM to process the image I obtained in step (4) to obtain the task parsing result S task , task analysis result S task Including the upper left corner coordinates and lower right corner coordinates of each starting point object and each end point object that the user needs to operate.

[0016] (6) Use the subtask sequence set S obtained in step (3) subtasks And the operation function call sequence set F call The task analysis result S obtained in step (5) task The upper left corner coordinates and the lower right corner coordinates of the starting point object and the end point object are processed to obtain the center point pixel coordinates of each starting point object and the end point object and the visual annotation result of the image I;

[0017] (7) According to the subtask sequence set S obtained in step (3) subtasks , perform hand-eye calibration on the pixel coordinates of the center points of all the starting and ending objects obtained in step (6) to obtain the robot arm coordinates (X start_mc , Y start_mc ), with the center point of each end point object as the robot arm coordinate (X end_mc , Y end_mc ).

[0018] (8) The operation function call sequence set F obtained according to step (3) call , set the motion mode of the robot arm to interpolation mode to obtain the robot arm after startup.

[0019] (9) The center point robot coordinates (X) of each starting point object obtained in step (7) start_mc , Y start_mc ), and the center point of each end point object's robot coordinates (X end_mc , Y end_mc ), and use the robotic arm started in step (8) to move the starting object to the end object.

[0020] Preferably, the robotic arm is a 6-DOF robotic arm;

[0021] Step (1) is specifically as follows: first, the camera resolution in the visual module of the robotic arm is initialized to 640*480 pixels, the field of view angle is initialized to 60 degrees, the focal length is initialized to 1.7 mm, and the user's command input method is set to keyboard input and voice input.

[0022] Then, initialize two sets of fixed calibration points, which are as follows according to the operating environment:

[0023] The lower left corner of the pixel coordinates is cali_1 im=[130,290];

[0024] The upper right corner of the pixel coordinates is cali_2 im =[640,0];

[0025] The lower left corner calibration point of the robot arm coordinates is cali_1 mc =[-21.8,-197.4];

[0026] The upper right corner calibration point of the robot arm coordinates is cali_2 mc =[215,-59.1].

[0027] Afterwards, the six joints and their motion ranges [J1, J2, J3, J4, J5, J6] of the robot are initialized as follows:

[0028] The base joint has a range of motion J1 of -180° to +180°;

[0029] Shoulder joint, its range of motion J2 is -270°~+90°;

[0030] Elbow joint, its range of motion J3 is -150°~+150°;

[0031] The first wrist joint has a range of motion of J4 from -260° to +80°;

[0032] The second wrist joint has a range of motion of J5 from -168° to +168°;

[0033] The range of motion of the terminal joint J6 is -174° to +174°.

[0034] Finally, set the upper limit of the robot arm's operating speed to 15.

[0035] Preferably, step (3) comprises the following sub-steps:

[0036] (3-1) The task instruction T obtained in step (2) input Integrate with the preset operation target prompt word SYSTEM_PROMPT to obtain the input instruction INPUT:

[0037] INPUT=SYSTEM_PROMPT+T input

[0038] (3-2) Input the input instruction INPUT obtained in step (3-1) into the large language model LLM for task instruction parsing to obtain the subtask sequence set S subtasks And the corresponding operation function call sequence set F call :

[0039] Ssubtasks ,F call =f LLM (INPUT)

[0040] where f LLM () represents the text instruction parsing function of LLM, and the subtask sequence set S subtasks In the table, all subtasks are arranged in the order of execution, and the operation function call sequence set F call Each operation function call sequence in corresponds to a subtask sequence set S subtasks The subtasks in .

[0041] Preferably, step (5) comprises the following sub-steps:

[0042] (5-1) Use the operation function call sequence set F obtained in step (3) call , perform Base64 encoding on the image I obtained in step (4) to obtain the image string I base64 ;

[0043] (5-2) The image string I obtained in step (5-1) base64 The input instruction INPUT obtained in step (3) is input into LVLM to obtain the task analysis result S task :

[0044] S task =f LVLM (INPUT,I base64 )

[0045] where f LVLM () represents the image and text instruction parsing function of LVLM.

[0046] Preferably, step (6) comprises the following sub-steps:

[0047] (6-1) Using the operation function call sequence set F obtained in step (3) call , for the robot operation planning set S obtained in step (5) task The upper left corner coordinate (X start_left ,Y start_left ) and the lower right corner coordinate (X start_right ,Y start_right ) are normalized and scaled to obtain the upper left corner pixel coordinates and lower right corner pixel coordinates of the starting point object, and the robot operation planning set S obtained in step (5) is task The upper left corner coordinate (X end_left ,Y end_left ) and the lower right corner coordinate (X end_right ,X end_right) are normalized and scaled to obtain the upper left corner pixel coordinates and the lower right corner pixel coordinates of the endpoint object.

[0048] (6-2) The manipulator operation planning set S obtained according to step (6-1) task The pixel coordinates of the upper left corner and the lower right corner of each starting point object are used to obtain the pixel coordinates of the center point of the starting point object (X start ,Y start ), and the manipulator operation planning set S obtained according to step (6-1) task The pixel coordinates of the upper left corner and the lower right corner of each end point object are used to obtain the pixel coordinates of the center point of the end point object (X end ,Y end );

[0049] (6-3) Determine whether the user needs to perform visual annotation. If so, proceed to step (6-4), otherwise the process ends.

[0050] (6-4) Label each starting object and each end point object on image I according to the upper left corner pixel coordinates and the lower right corner pixel coordinates of each starting point object and each end point object obtained in step (6-1), and the center point pixel coordinates of each starting point object and each end point object obtained in step (6-2) to obtain a visual labeling result, including a bounding box labeling result, a center point labeling result, and an object name labeling result.

[0051] Preferably, the pixel coordinates of the upper left corner of the starting point object are equal to:

[0052]

[0053] Where factor represents the scaling factor, which is fixed to 999, L represents the length of the resolution of image I, and W represents the width of the resolution of image I.

[0054] The pixel coordinates of the upper left corner of the starting object are equal to:

[0055]

[0056] The pixel coordinates of the upper left corner of the endpoint object are equal to:

[0057]

[0058] The pixel coordinates of the lower right corner of the endpoint object are equal to:

[0059]

[0060] Get the pixel coordinates of the center point of the starting object (X start ,Y start ) is calculated using the following formula:

[0061]

[0062] Get the pixel coordinates of the center point of the end object (X end ,Y end ) is calculated using the following formula:

[0063]

[0064] Preferably, step (7) comprises the following sub-steps:

[0065] (7-1) The operation function call sequence set F obtained according to step (3) call , and the pixel coordinate range [X Cali_im ,Y Cali_im ] and the calibrated robot arm coordinate range [X Cali_mc ,Y Cali_mc ];

[0066] (7-2) According to the calibrated pixel coordinate range obtained in step (7-1), linear interpolation is performed on the pixel coordinates of the center point of each starting point object and each end point object obtained in step (6) to obtain the robot arm coordinates (X start_mc , Y start_mc ) and the center point of each end point object’s robot coordinates (X end_mc , Y end_mc ).

[0067] Preferably, step (7-1) is to obtain the calibrated pixel coordinate range using the following formula:

[0068] [X Cali_im ,Y Cali_im ]=[[cali_1 im [1],cali_2 im [1]],[cali_1 im [2],cali_2 im [2]]]

[0069] Among them, cali_1 im [1] represents the lower left corner calibration point cali_1 of the pixel coordinates im The first element in cali_1 im [2] represents the lower left corner calibration point cali_1 of the pixel coordinates im The second element in cali_2 im [1] indicates the upper right corner calibration point cali_2 of the pixel coordinates im The first element in cali_2im [2] indicates the upper right corner calibration point cali_2 of the pixel coordinates im The second element within.

[0070] Step (7-1) is to use the following formula to obtain the calibrated robot arm coordinate range:

[0071] [X Cali_mc ,Y Cali_mc ]=[[cali_1 mc [1],cali_2 mc [1]],[cali_1 mc [2],cali_2 mc [2]]]

[0072] where cali_1 mc [1] indicates the lower left corner calibration point cali_1 of the robot arm coordinates mc The first element in cali_1 mc [2] indicates the lower left corner calibration point cali_1 of the robot arm coordinates mc The second element in cali_2 mc [1] indicates the upper right corner calibration point cali_2 of the robot arm coordinates mc The first element in cali_2 mc [2] indicates the upper right corner calibration point cali_2 of the robot arm coordinates mc The second element within.

[0073] In step (7-2), the center point robot coordinates (X start_mc , Y start_mc ) is calculated using the following formula:

[0074]

[0075] In step (7-2), the center point of each end point object is obtained by the robot arm coordinates (X end_mc , Y end_mc ) is calculated using the following formula:

[0076]

[0077] Preferably, step (9) comprises the following sub-steps:

[0078] (9-1) Drive the robot arm to move to the center point of each starting point object above the robot arm coordinates and at a safe height Height safe and get the position of the robot arm:

[0079] [X start_mc ,Y start_mc,Height safe ,0,180,90]

[0080] The safety height safe Fixed setting is 220 mm.

[0081] (9-2) According to the posture of the robotic arm obtained in step (9-1), turn on the suction pump to drive the robotic arm from a safe height to safe Move to grab height Height start The position of the robot arm is obtained as follows:

[0082] [X start_mc ,Y start_mc ,Height start ,0,180,90]

[0083] The grab height Height start Fixed setting is 90 mm.

[0084] (9-3) According to the posture of the robotic arm obtained in step (9-2), drive the robotic arm to move to a safe height. safe , and the posture of the robotic arm at this time is obtained as:

[0085] [X end_mc ,Y end_mc ,Height safe ,0,180,90]

[0086] (9-4) According to the posture of the robotic arm obtained in step (9-3), move the robotic arm vertically downward to the target placement height Height end , turn off the suction pump, open the air release valve to release the residual negative pressure, then close the air release valve and after the preset delay, drive the robot arm back to the zero position [0,0,0,0,0,0], where the target placement height Height end Fixed setting is 100 mm.

[0087] According to another aspect of the present invention, there is provided an intelligent robotic arm operating system based on a multimodal large visual language model, comprising:

[0088] The first module is used to initialize the operating environment of the current robotic arm to obtain the operating environment of the robotic arm after initialization.

[0089] The second module is used to receive the task instruction T input by the user according to the operating environment of the robot arm after the first module is initialized. input , task instruction T input Included are the names of each starting point object and each end point object that the user needs to operate.

[0090] The third module is used to execute the task instruction T obtained by the second module. input Parse to obtain the parsing result, which includes the subtask sequence set S subtasks And its corresponding operation function call sequence set F call .

[0091] The fourth module is used to call the sequence set F of the operation function obtained in the third module. call , drive the robot arm to move to the operating environment of the robot arm after initialization according to the first module, start the camera in the visual module, and use the camera to obtain and save the image I in the operating environment after a preset delay, and record the saving time and saving path of the image I.

[0092] The fifth module is used to obtain the subtask sequence set S according to the third module. subtasks , and use the multimodal large visual language model LVLM to process the image I obtained by the fourth module to obtain the task parsing result S task , task analysis result S task Including the upper left corner coordinates and lower right corner coordinates of each starting point object and each end point object that the user needs to operate.

[0093] The sixth module is used to use the subtask sequence set S obtained in the third module subtasks And the operation function call sequence set F call The task analysis result S obtained from the fifth module task The upper left corner coordinates and the lower right corner coordinates of the starting point object and the end point object are processed to obtain the center point pixel coordinates of each starting point object and the end point object and the visual annotation result of the image I;

[0094] The seventh module is used to obtain the subtask sequence set S according to the third module. subtasks , perform hand-eye calibration on the pixel coordinates of the center points of all the starting and ending objects obtained by the sixth module to obtain the robot arm coordinates (X start_mc , Y start_mc ), with the center point of each end point object as the robot arm coordinate (X end_mc , Y end_mc ).

[0095] The eighth module is used to call the sequence set F of the operation functions obtained in the third module. call , set the motion mode of the robot arm to interpolation mode to obtain the robot arm after startup.

[0096] The ninth module is used to calculate the center point robot coordinates (X start_mc , Y start_mc), and the center point of each end point object's robot coordinates (X end_mc , Y end_mc ), and use the robotic arm started by the eighth module to move the starting object to the end object.

[0097] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0098] (1) Since the present invention adopts step (3), it can understand complex natural language instructions through the text instruction parsing ability of the large language model and convert them into executable specific task objectives, eliminating the dependence on fixed rules, thereby solving the technical problem of limited task instruction understanding of existing robots;

[0099] (2) Since the present invention adopts step (5), it can simultaneously process multimodal information such as vision and language through the parsing ability of multimodal data such as images and text instructions of the multimodal large visual language model, so as to have a more comprehensive understanding and dynamic response to the environment, thereby solving the technical problem of the single environmental perception ability of existing robots;

[0100] (3) The present invention adopts steps (5) to (7), which converts the real-time pixel information obtained by parsing the multimodal large visual language model into operation coordinates through hand-eye calibration, and combines the analysis of text instructions to enable the robot arm to dynamically adjust the path planning and execution strategy, thereby solving the technical problem of poor dynamic adaptability of task planning of existing robots due to fixed rule settings;

[0101] (4) Since the present invention adopts steps (3) to (5), it can combine context, task objectives and environmental information by utilizing a large model and a multimodal large visual language model to understand and adapt to user instructions, so that the robotic arm is closer to user needs and the interaction efficiency and experience are significantly improved. Therefore, it can solve the technical problem of insufficient human-computer interaction caused by the lack of language processing technology in existing robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 It is a flow chart of the intelligent robotic arm operation method based on the multimodal large visual language model of the present invention. DETAILED DESCRIPTION

[0103] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0104] The present invention proposes a method and system for manipulating a robot arm based on a multimodal large visual language model. The method first collects image data of the robot arm operation scene through a visual sensor, and obtains user instructions in combination with a speech recognition module or a text input module, and inputs natural language instructions and visual scene data into the multimodal large visual language model for multimodal task analysis. The model identifies the category, position and attributes of the target object in the scene, and generates a specific operation sequence in combination with the input instruction. Subsequently, the system directly drives the robot arm to perform the corresponding task according to the operation sequence generated by the analysis, including specific actions such as grabbing, moving and placing the target object. During the task execution process, the robot arm completes the action in sequence according to the pre-generated operation sequence. If an abnormal situation occurs (such as target position offset or environmental change), the operation sequence can be updated by re-collecting environmental information and calling the multimodal large visual language model. Through this method, the present invention effectively solves the problems of inaccurate analysis of complex task instructions, insufficient operation flexibility and adaptability, and poor real-time response ability in dynamic environments in the prior art, and significantly improves the intelligence and stability of the robot arm task execution.

[0105] In order to make the technical solutions and advantages of the present invention more clearly understood, the whole set of robot arm operation methods are further described in detail below in conjunction with the accompanying drawings.

[0106] like Figure 1 As shown, the present invention provides an intelligent robotic arm operation method based on a multimodal large visual language model, comprising the following steps:

[0107] (1) Initialize the current operating environment of the robot arm to obtain the operating environment of the robot arm after initialization.

[0108] Specifically, the robot arm used in the present invention is a 6-degree-of-freedom (6-DoF) elephant myCobot 280PI robot arm.

[0109] Specifically, this step is as follows: first, the camera resolution in the visual module of the robotic arm is initialized to 640*480 pixels, the field of view angle is initialized to 60 degrees, the focal length is initialized to 1.7 mm, and the user's command input method is set to keyboard input and voice input.

[0110] Then, initialize two sets of fixed calibration points, which are as follows according to the operating environment:

[0111] The lower left corner of the pixel coordinates is cali_1 im =[130,290];

[0112] The upper right corner of the pixel coordinates is cali_2im =[640,0];

[0113] The lower left corner calibration point of the robot arm coordinates is cali_1 mc =[-21.8,-197.4];

[0114] The upper right corner calibration point of the robot arm coordinates is cali_2 mc =[215,-59.1].

[0115] Afterwards, the six joints and their motion ranges [J1, J2, J3, J4, J5, J6] of the robot are initialized as follows:

[0116] The base joint has a range of motion J1 of -180° to +180°;

[0117] Shoulder joint, its range of motion J2 is -270°~+90°;

[0118] Elbow joint, its range of motion J3 is -150°~+150°;

[0119] The first wrist joint has a range of motion of J4 from -260° to +80°;

[0120] The second wrist joint has a range of motion of J5 from -168° to +168°;

[0121] The range of motion of the terminal joint J6 is -174° to +174°.

[0122] Then, set the upper limit value of the robot arm's operation speed (such as grabbing operation, putting down operation, etc.) to 15.

[0123] (2) receiving the task instruction T input by the user according to the operation environment of the robot arm after initialization in step (1) input , task instruction T input Included are the names of each starting point object and each end point object that the user needs to operate.

[0124] Specifically, in this step, the user gives the task instruction T according to the instruction input method set in step (1). input Keyboard input is when a user inputs a natural language task statement through a keyboard, and voice input is when a user uses automatic speech recognition technology (ASR) to convert the user's voice command into text.

[0125] For the task instruction T input For example, taking the task instruction "Please put the red block on the green block" as an example, T inputThe starting point object that needs to be operated is the "red square" and the end point object is the "green square".

[0126] (3) The task instruction T obtained in step (2) input Parse to obtain the parsing result, which includes the subtask sequence set S subtasks And its corresponding operation function call sequence set F call .

[0127] Specifically, this step includes the following sub-steps:

[0128] (3-1) The task instruction T obtained in step (2) input Integrate with the preset operation target prompt word SYSTEM_PROMPT to obtain the input instruction INPUT:

[0129] INPUT=SYSTEM_PROMPT+T input

[0130] (3-2) Input the input instruction INPUT obtained in step (3-1) into the large language model (LLM) for task instruction parsing to obtain the subtask sequence set S subtasks And the corresponding operation function call sequence set F call :

[0131] S subtasks ,F call =f LLM (INPUT)

[0132] where f LLM () represents the text instruction parsing function of LLM, and the subtask sequence set S subtasks In the table, all subtasks are arranged in the order of execution, and the operation function call sequence set F call Each operation function call sequence in corresponds to a subtask sequence set S subtasks The subtasks in .

[0133] The advantage of this step is that the efficient language parsing capability of LLM can be utilized to achieve effective semantic understanding of the user's task instructions.

[0134] (4) The operation function call sequence set F obtained according to step (3) call , drive the robot arm to move to the operating environment of the robot arm after initialization according to step (1) (i.e., the lower left corner vertex and the upper right corner vertex of the pixel coordinates), start the camera in the visual module, and use the camera to obtain and save the image I in the operating environment after a preset delay (preferably 0.3 seconds), and record the saving time and saving path of the image I.

[0135] (5) According to step (3), the subtask sequence set S is obtained subtasks , and use the multimodal large vision language model (LVLM) to process the image I obtained in step (4) to obtain the task parsing result S task , task analysis result S task Including the upper left corner coordinates and lower right corner coordinates of each starting point object and each end point object that the user needs to operate.

[0136] Specifically, this step includes the following sub-steps:

[0137] (5-1) Use the operation function call sequence set F obtained in step (3) call , perform Base64 encoding on the image I obtained in step (4) to obtain the image string I base64 ;

[0138] (5-2) The image string I obtained in step (5-1) base64 The input instruction INPUT obtained in step (3) is input into LVLM to obtain the task analysis result S task :

[0139] S task =f LVLM (INPUT,I base64 )

[0140] where f LVLM () represents the image and text instruction parsing function of LVLM.

[0141] The advantage of this step is that we can use LVLM's ability to parse multimodal data to achieve a more comprehensive understanding and dynamic response to the instructions combined with the environment.

[0142] (6) Use the subtask sequence set S obtained in step (3) subtasks And the operation function call sequence set F call The task analysis result S obtained in step (5) task The upper left corner coordinates and the lower right corner coordinates of the starting point object and the end point object are processed to obtain the center point pixel coordinates of each starting point object and the end point object and the visual annotation result of the image I;

[0143] Specifically, this step includes the following sub-steps:

[0144] (6-1) Using the operation function call sequence set F obtained in step (3) call , for the robot operation planning set S obtained in step (5) taskThe upper left corner coordinate (X start_left ,Y start_left ) and the lower right corner coordinate (X start_right ,Y start_right ) are normalized and scaled to obtain the upper left corner pixel coordinates and lower right corner pixel coordinates of the starting point object, and the robot operation planning set S obtained in step (5) is task The upper left corner coordinate (X end_left ,Y end_left ) and the lower right corner coordinate (X end_right ,Y end_right ) are normalized and scaled to obtain the upper left corner pixel coordinates and the lower right corner pixel coordinates of the endpoint object.

[0145] The pixel coordinates of the upper left corner of the starting object are equal to:

[0146]

[0147] Where factor represents the scaling factor, which is fixed to 999, L represents the length of the resolution of image I, and W represents the width of the resolution of image I.

[0148] The pixel coordinates of the upper left corner of the starting object are equal to:

[0149]

[0150] The pixel coordinates of the upper left corner of the endpoint object are equal to:

[0151]

[0152] The pixel coordinates of the lower right corner of the endpoint object are equal to:

[0153]

[0154] (6-2) The manipulator operation planning set S obtained according to step (6-1) task The pixel coordinates of the upper left corner and the lower right corner of each starting point object are used to obtain the pixel coordinates of the center point of the starting point object (X start ,Y start ), and the manipulator operation planning set S obtained according to step (6-1) task The pixel coordinates of the upper left corner and the lower right corner of each end point object are used to obtain the pixel coordinates of the center point of the end point object (X end ,Y end );

[0155] Specifically, in this step, the pixel coordinates of the center point of the starting point object (X start ,Y start ) is calculated using the following formula:

[0156]

[0157] In this step, the pixel coordinates of the center point of the end point object (X end ,Y end ) is calculated using the following formula:

[0158]

[0159] (6-3) Determine whether the user needs to perform visual annotation. If so, proceed to step (6-4), otherwise the process ends.

[0160] (6-4) Label each starting object and each end point object on image I according to the upper left corner pixel coordinates and the lower right corner pixel coordinates of each starting point object and each end point object obtained in step (6-1), and the center point pixel coordinates of each starting point object and each end point object obtained in step (6-2) to obtain a visual labeling result, including a bounding box labeling result, a center point labeling result, and an object name labeling result.

[0161] Specifically, the bounding box annotation is based on the upper left corner pixel coordinates and lower right corner pixel coordinates of each starting point object and the upper left corner pixel coordinates and lower right corner pixel coordinates of each end point object obtained in step (6-1), and each starting point object is annotated with a red rectangular box and each end point object is annotated with a blue rectangular box.

[0162] The center point marking is the pixel coordinates of the center point of each starting object and end point object obtained in step (6-2). A dot is drawn at the center point of the starting object and the end point object, where the center point of the starting object is a red dot and the center point of the end point object is a blue dot.

[0163] The object name is labeled by the name of each starting object and end point object, and the object name is labeled above the labeled bounding box of the starting object and the end point object, where the name of the starting object is labeled in red text and the name of the end point object is labeled in blue text.

[0164] (7) According to the subtask sequence set S obtained in step (3) subtasks , perform hand-eye calibration on the pixel coordinates of the center points of all the starting and ending objects obtained in step (6) to obtain the robot arm coordinates (X start_mc , Y start_mc ), with the center point of each end point object as the robot arm coordinate (X end_mc , Y end_mc ).

[0165] Specifically, this step includes the following sub-steps:

[0166] (7-1) The operation function call sequence set F obtained according to step (3) call , and the pixel coordinate range [X Cali_im ,Y Cali_im ] and the calibrated robot arm coordinate range [X Cali_mc ,Y Cali_mc ];

[0167] Specifically, this step uses the following formula to obtain the calibrated pixel coordinate range:

[0168] [X Cali_im ,Y Cail_im ]=[[cali_1 im [1],cali_2 im [1]],[cali_1 im [2],cali_2 im [2]]]

[0169] Among them, cali_1 im [1] represents the lower left corner calibration point cali_1 of the pixel coordinates im The first element in cali_1 im [2] represents the lower left corner calibration point cali_1 of the pixel coordinates im The second element in cali_2 im [1] indicates the upper right corner calibration point cali_2 of the pixel coordinates im The first element in cali_2 im [2] indicates the upper right corner calibration point cali_2 of the pixel coordinates im The second element within.

[0170] This step uses the following formula to obtain the calibrated robot arm coordinate range:

[0171] [X Cail_mc ,Y Cali_mc ]=[[cali_1 mc [1],cali_2 mc [1]],[cali_1 mc [2],cali_2 mc [2]]]

[0172] where cali_1 mc [1] indicates the lower left corner calibration point cali_1 of the robot arm coordinates mc The first element in cali_1 mc [2] indicates the lower left corner calibration point cali_1 of the robot arm coordinates mcThe second element in cali_2 mc [1] indicates the upper right corner calibration point cali_2 of the robot arm coordinates mc The first element in cali_2 mc [2] indicates the upper right corner calibration point cali_2 of the robot arm coordinates mc The second element within.

[0173] (7-2) According to the calibrated pixel coordinate range obtained in step (7-1), linear interpolation is performed on the pixel coordinates of the center point of each starting point object and each end point object obtained in step (6) to obtain the robot arm coordinates (X start_mc , Y start_mc ) and the center point of each end point object’s robot coordinates (X end_mc , Y end_mc )

[0174] Specifically, obtain the center point robot coordinates (X start_mc , Y start_mc ) is calculated using the following formula:

[0175]

[0176] Get the center point of each end point object. end_mc , Y end_mc ) is calculated using the following formula:

[0177]

[0178]

[0179] The advantage of the above steps (5) to (7) is that by converting the analysis content of the LVLM into robot arm coordinates through hand-eye calibration, it is possible to perceive complex environments and perform flexible adaptation operations.

[0180] (8) The operation function call sequence set F obtained according to step (3) call , set the motion mode of the robot arm to interpolation mode to obtain the robot arm after startup.

[0181] (9) The center point robot coordinates (X) of each starting point object obtained in step (7) start_mc , Y start_mc ), and the center point of each end point object's robot coordinates (X end_mc , Y end_mc ), and use the robotic arm started in step (8) to move the starting object to the end object.

[0182] Specifically, this step includes the following sub-steps:

[0183] (9-1) Drive the robot arm to move to the center point of each starting point object above the robot arm coordinates and at a safe height Height safe and get the position of the robot arm:

[0184] [X start_mc ,Y start_mc ,Height safe ,0,180,90]

[0185] The safety height safe Fixed setting is 220 mm.

[0186] (9-2) According to the posture of the robotic arm obtained in step (9-1), turn on the suction pump to drive the robotic arm from a safe height to safe Move to grab height Height start The position of the robot arm is obtained as follows:

[0187] [X start_mc ,Y start_mc ,Height start ,0,180,90]

[0188] The grab height Height start Fixed setting is 90 mm.

[0189] (9-3) According to the posture of the robotic arm obtained in step (9-2), drive the robotic arm to move to a safe height. safe , and the posture of the robotic arm at this time is obtained as:

[0190] [X end_mc ,Y end_mc ,Height safe ,0,180,90]

[0191] (9-4) According to the posture of the robotic arm obtained in step (9-3), move the robotic arm vertically downward to the target placement height Height end , turn off the suction pump, open the air release valve, release the residual negative pressure (the release time is preferably 0.2 seconds), then close the air release valve and after a preset delay (the preset delay is preferably 0.05 seconds), drive the robotic arm to return to the zero position [0,0,0,0,0,0].

[0192] The target placement height is Height end Fixed setting is 100 mm.

[0193] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent robotic arm operation method based on a multimodal large visual language model, characterized in that: The following steps are involved: (1) Initialize the current operating environment of the robot arm to obtain the operating environment of the robot arm after initialization. (2) receiving the task instruction T input by the user according to the operation environment of the robot arm after initialization in step (1) input , task instruction T input Included are the names of each starting point object and each end point object that the user needs to operate. (3) The task instruction t obtained in step (2) input Parse to obtain the parsing result, which includes the subtask sequence set S subtasks And its corresponding operation function call sequence set F call . (4) The operation function call sequence set F obtained according to step (3) call , drive the robot arm to move to the operating environment of the robot arm after initialization according to step (1), start the camera in the visual module, and use the camera to obtain and save the image I in the operating environment after a preset delay, and record the saving time and saving path of the image I. (5) The subtask sequence set S obtained according to step (3) subtasks , and use the multimodal large visual language model LVLM to process the image I obtained in step (4) to obtain the task parsing result S task , task analysis result S task Including the upper left corner coordinates and lower right corner coordinates of each starting point object and each end point object that the user needs to operate. (6) Use the subtask sequence set S obtained in step (3) subtasks And the operation function call sequence set F call The task analysis result S obtained in step (5) task The upper left corner coordinates and the lower right corner coordinates of the starting point object and the end point object are processed to obtain the center point pixel coordinates of each starting point object and the end point object and the visual annotation result of the image I; (7) According to the subtask sequence set S obtained in step (3) subtasks , perform hand-eye calibration on the pixel coordinates of the center points of all the starting and ending objects obtained in step (6) to obtain the robot arm coordinates (X start_mc , Y start_mc ), with the center point of each end point object as the robot arm coordinate (X end_mc , Y end_mc ). (8) The operation function call sequence set F obtained according to step (3) call , set the motion mode of the robot arm to interpolation mode to obtain the robot arm after startup. (9) The center point robot coordinates (X) of each starting point object obtained in step (7) start_mc , Y start_mc ), and the center point of each end point object's robot coordinates (X end_mc , Y end_mc ), and use the robotic arm started in step (8) to move the starting object to the end object.

2. The intelligent robotic arm operation method based on a multimodal large visual language model according to claim 1 is characterized in that: The robot arm is a 6-DOF robot; Step (1) is specifically as follows: first, the camera resolution in the visual module of the robotic arm is initialized to 640*480 pixels, the field of view angle is initialized to 60 degrees, the focal length is initialized to 1.7 mm, and the user's command input method is set to keyboard input and voice input. Then, initialize two sets of fixed calibration points, which are as follows according to the operating environment: The lower left corner of the pixel coordinates is the calibration point cail_1 im =[130,290]; The upper right corner of the pixel coordinates is cali_2 im =[640,0]; The lower left corner calibration point of the robot arm coordinates is cali_1 me =[-21.8,-197.4]; The upper right corner calibration point of the robot arm coordinates is cali_2 mc =[215,-59.1]. Afterwards, the six joints and their motion ranges [J1, J2, J3, J4, J5, J6] of the robot are initialized as follows: The base joint has a range of motion J1 of -180° to +180°; Shoulder joint, its range of motion J2 is -270°~+90°; Elbow joint, its range of motion J3 is -150°~+150°; The first wrist joint has a range of motion of J4 from -260° to +80°; The second wrist joint has a range of motion of J5 from -168° to +168°; The range of motion of the terminal joint J6 is -174° to +174°. Finally, set the upper limit of the robot arm's operating speed to 15.

3. The intelligent robotic arm operation method based on a multimodal large visual language model according to claim 1 or 2, characterized in that: Step (3) includes the following sub-steps: (3-1) The task instruction T obtained in step (2) input Integrate with the preset operation target prompt word SYSTEM_PROMPT to obtain the input instruction INPUT: INPUT=SYSTEM_PROMPT+T input (3-2) Input the input instruction INPUT obtained in step (3-1) into the large language model LLM for task instruction parsing to obtain the subtask sequence set S subtasks And the corresponding operation function call sequence set F call : S subtasks ,F call =f LLM (INPUT) where f LLM () represents the text instruction parsing function of LLM, and the subtask sequence set S subtasks In the table, all subtasks are arranged in the order of execution, and the operation function call sequence set F call Each operation function call sequence in corresponds to a subtask sequence set S subtasks The subtasks in .

4. The intelligent robotic arm operation method based on a multimodal large visual language model according to any one of claims 1 to 3, characterized in that: Step (5) includes the following sub-steps: (5-1) Use the operation function call sequence set F obtained in step (3) call , perform Base64 encoding on the image I obtained in step (4) to obtain the image string I base64 ; (5-2) The image string I obtained in step (5-1) base64 The input instruction INPUT obtained in step (3) is input into LVLM to obtain the task analysis result S task : S task =f LVLM (INPUT,I base64 ) where f LVLM () represents the image and text instruction parsing function of LVLM.

5. The intelligent robotic arm operation method based on a multimodal large visual language model according to claim 4 is characterized in that: Step (6) includes the following sub-steps: (6-1) Using the operation function call sequence set F obtained in step (3) call , for the robot operation planning set S obtained in step (5) task The upper left corner coordinate (X start_left ,Y start_left ) and the lower right corner coordinate (X start_right ,Y start_right ) are normalized and scaled to obtain the upper left corner pixel coordinates and lower right corner pixel coordinates of the starting point object, and the robot operation planning set S obtained in step (5) is task The upper left corner coordinate (X end_left ,Y end_left ) and the lower right corner coordinate (X end_right ,Y end_right ) are normalized and scaled to obtain the upper left corner pixel coordinates and the lower right corner pixel coordinates of the endpoint object. (6-2) The manipulator operation planning set S obtained according to step (6-1) task The pixel coordinates of the upper left corner and the lower right corner of each starting point object are used to obtain the pixel coordinates of the center point of the starting point object (X start ,Y start ), and the manipulator operation planning set S obtained according to step (6-1) task The pixel coordinates of the upper left corner and the lower right corner of each end point object are used to obtain the pixel coordinates of the center point of the end point object (X end ,Y end ); (6-3) Determine whether the user needs to perform visual annotation. If so, proceed to step (6-4), otherwise the process ends. (6-4) Label each starting object and each end point object on image I according to the upper left corner pixel coordinates and the lower right corner pixel coordinates of each starting point object and each end point object obtained in step (6-1), and the center point pixel coordinates of each starting point object and each end point object obtained in step (6-2) to obtain a visual labeling result, including a bounding box labeling result, a center point labeling result, and an object name labeling result.

6. The intelligent robotic arm operation method based on a multimodal large visual language model according to claim 5 is characterized in that: The pixel coordinates of the upper left corner of the starting object are equal to: Where factor represents the scaling factor, which is fixed to 999, L represents the length of the resolution of image I, and W represents the width of the resolution of image I. The pixel coordinates of the upper left corner of the starting object are equal to: The pixel coordinates of the upper left corner of the endpoint object are equal to: The pixel coordinates of the lower right corner of the endpoint object are equal to: Get the pixel coordinates of the center point of the starting point object (X start ,Y start ) is calculated using the following formula: Get the pixel coordinates of the center point of the end object (X end ,Y end ) is calculated using the following formula:

7. The intelligent robotic arm operation method based on a multimodal large visual language model according to claim 6 is characterized in that: Step (7) includes the following sub-steps: (7-1) The operation function call sequence set F obtained according to step (3) call , and the pixel coordinate range [X Cali_im ,Y Cali_im ] and the calibrated robot arm coordinate range [X Cali_mc ,Y Cali_mc ]; (7-2) According to the calibrated pixel coordinate range obtained in step (7-1), linear interpolation is performed on the pixel coordinates of the center point of each starting point object and each end point object obtained in step (6) to obtain the robot arm coordinates (X start_mc , Y start_mc ) and the center point of each end point object’s robot coordinates (X end_mc , Y end_mc ).

8. The intelligent robotic arm operation method based on a multimodal large visual language model according to claim 7 is characterized in that: Step (7-1) is to obtain the calibrated pixel coordinate range using the following formula: [X] Cali_im ,Y Cali_im ]=[[cali_1 im [1],cali_2 im [1]],[cali_1 im [2],cail_2 im [2]]] Among them, cail_1 im [1] represents the lower left corner calibration point cail_1 of the pixel coordinates im The first element in cali_1 im [2] represents the lower left corner calibration point cali_1 of the pixel coordinates im The second element in cali_2 im [1] indicates the upper right corner calibration point cali_2 of the pixel coordinates im The first element in cali_2 im [2] indicates the upper right corner calibration point cali_2 of the pixel coordinates im The second element within. Step (7-1) is to use the following formula to obtain the calibrated robot arm coordinate range: [X Cali_mc ,Y Cali_mc ]=[[cali_1 mc [1],cali_2 mc [1]],[cali_1 mc [2],cali_2 mc [2]]] where cali_1 mc [1] indicates the lower left corner calibration point cali_1 of the robot arm coordinates mc The first element in cali_1 mc [2] indicates the lower left corner calibration point cail_1 of the robot arm coordinates mc The second element in cali_2 mc [1] indicates the upper right corner calibration point cali_2 of the robot arm coordinates mc The first element in cali_2 mc [2] indicates the upper right corner calibration point cali_2 of the robot arm coordinates mc The second element within. In step (7-2), the center point robot coordinates (X start_mc , Y start_mc ) is calculated using the following formula: In step (7-2), the center point of each end point object is obtained by the robot arm coordinates (X end_mc , Y end_mc ) is calculated using the following formula:

9. The intelligent robotic arm operation method based on a multimodal large visual language model according to claim 8, characterized in that: Step (9) includes the following sub-steps: (9-1) Drive the robot arm to move to the center point of each starting point object above the robot arm coordinates and at a safe height Height safe and get the position of the robot arm: [X start_mc ,Y start_mc ,Height safe ,0,180,90] The safety height safe Fixed setting is 220 mm. (9-2) According to the posture of the robotic arm obtained in step (9-1), turn on the suction pump to drive the robotic arm from a safe height to safe Move to grab height Height start The position of the robot arm is obtained as follows: [X start_mc ,Y start_mc ,Height start ,0,180,90] The grab height Height start Fixed setting is 90 mm. (9-3) According to the posture of the robotic arm obtained in step (9-2), drive the robotic arm to move to a safe height. safe , and the posture of the robotic arm at this time is obtained as: [X end_mc ,Y end_mc ,Height safe ,0,180,90] (9-4) According to the posture of the robotic arm obtained in step (9-3), move the robotic arm vertically downward to the target placement height Height end , turn off the suction pump, open the air release valve to release the residual negative pressure, then close the air release valve and after the preset delay, drive the robot arm back to the zero position [0,0,0,0,0,0], where the target placement height Height end Fixed setting is 100 mm.

10. An intelligent robotic arm operating system based on a multimodal large visual language model, characterized in that: include: The first module is used to initialize the operating environment of the current robotic arm to obtain the operating environment of the robotic arm after initialization. The second module is used to receive the task instruction T input by the user according to the operating environment of the robot arm after the first module is initialized. input , task instruction T input Included are the names of each starting point object and each end point object that the user needs to operate. The third module is used to execute the task instruction T obtained by the second module. input Parse to obtain the parsing result, which includes the subtask sequence set S subtasks And its corresponding operation function call sequence set F call . The fourth module is used to call the sequence set F of the operation function obtained in the third module. call , drive the robot arm to move to the operating environment of the robot arm after initialization according to the first module, start the camera in the visual module, and use the camera to obtain and save the image I in the operating environment after a preset delay, and record the saving time and saving path of the image I. The fifth module is used to obtain the subtask sequence set S according to the third module. subtasks , and use the multimodal large visual language model LVLM to process the image I obtained by the fourth module to obtain the task parsing result S task , task analysis result S task Including the upper left corner coordinates and lower right corner coordinates of each starting point object and each end point object that the user needs to operate. The sixth module is used to use the subtask sequence set S obtained in the third module subtasks And the operation function call sequence set F call The task analysis result S obtained from the fifth module task The upper left corner coordinates and the lower right corner coordinates of the starting point object and the end point object are processed to obtain the center point pixel coordinates of each starting point object and the end point object and the visual annotation result of the image I; The seventh module is used to obtain the subtask sequence set S according to the third module. subtasks , perform hand-eye calibration on the pixel coordinates of the center points of all the starting and ending objects obtained by the sixth module to obtain the robot arm coordinates (X start_mc , Y start_mc ), with the center point of each end point object as the robot arm coordinate (X end_mc , Y end_mc ). The eighth module is used to call the sequence set F of the operation functions obtained in the third module. call , set the motion mode of the robot arm to interpolation mode to obtain the robot arm after startup. The ninth module is used to calculate the center point robot coordinates (X start_mc , Y start_mc ), and the center point of each end point object's robot coordinates (X end_mc , Y end_mc ), and use the robotic arm started by the eighth module to move the starting object to the end object.

Images