Measurement and control instrument system intelligent connection method and device based on multi-modal large model

Abstract

The invention discloses an intelligent connection method and device for a measurement and control instrument system based on a multi-modal large model, and belongs to the technical field of intelligent control. By utilizing the powerful information processing capability of a multi-modal large model and a man-machine interaction interface which is easy to operate, through a target detection algorithm based on deep learning and the like, positions of various operations such as grabbing, plugging, moving and the like of the multi-modal large model are accurately positioned; the intelligent agent executing action arrangement and the action skill knowledge base help the mechanical arm to complete complex tasks. The multi-mode large model can receive various information such as pictures, texts and voices, process the information and then call proper actions for combination, and control over the mechanical arm is completed. According to the method, the dependence on the environment is effectively reduced, the operation method of the mechanical arm is simplified, and meanwhile the real-time processing capacity is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and more specifically to an intelligent connection method and device for a measurement and control instrument system based on a multi-modal large model. Background Art

[0002] Artificial intelligence is a branch of computer science and a science and technology that aims to enable machines to simulate human intelligence. It includes the ability to learn (acquire information and regularize it to reach conclusions), reason (use rules to reach approximate or definite conclusions), and self-correction. The goal is to create intelligent systems that can perform complex tasks. Artificial intelligence is a broad field that includes multiple branches, such as machine learning, natural language processing, computer vision, and robotics.

[0003] Multimodal large models are an important development direction in the field of artificial intelligence, with broad application prospects and huge potential. Multimodal large models can process and understand multiple types of data such as text, images, audio, and video. They fuse information from different modalities through attention mechanisms, embedding mapping, and cross-modal alignment, allowing the model to establish associations between multimodal data. Multimodal large models have brought new breakthroughs in the perception and cognitive capabilities of artificial intelligence systems, and have shown broad application prospects in the fields of intelligent question answering, image description generation, and video understanding.

[0004] Deep learning is a subfield of machine learning that focuses on using artificial neural networks to analyze and learn complex patterns and features. In recent years, deep learning has shown amazing accuracy in image recognition, object detection, semantic segmentation, and other fields. The research on deep learning mainly involves three types of methods: 1) Convolutional Neural Networks (CNNs); 2) Autoencoder Neural Networks, including autoencoders and sparse coding; 3) Deep Belief Networks, which are pre-trained in the form of multi-layer autoencoder neural networks and then further optimize the neural network weights by combining discriminant information.

[0005] However, when operating the robotic arm, there are currently problems such as strong environmental dependence, poor adaptability, high system complexity and cost, low real-time processing capability, and high technical requirements for users. Therefore, how to provide an intelligent connection method and device for a measurement and control instrument system based on a multimodal large model is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides an intelligent connection method and device for a measurement and control instrument system based on a multimodal large model, which fully utilizes the advantages of the multimodal large model's simple human-computer interaction mode and powerful complex scene processing capabilities to solve the problems in the prior art.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A method for intelligently connecting a measurement and control instrument system based on a multi-modal large model, wherein the measurement and control instrument system includes a mechanical arm for operating the instrument, comprising the following steps:

[0009] Step 1: Receive input instructions through the multimodal big model, and according to the different types of input instructions, match the corresponding recognition components to convert them into a form that can be read by the big model;

[0010] Step 2: Obtain the end position picture taken when the robot arm is in the original position, and the start position picture taken when the robot arm moves to the overlooking state;

[0011] Step 3: The multimodal large model calls the deep learning algorithm to detect and locate the starting point and end point target positions in the end point position image and the starting point position image, and outputs the coordinates of the starting point and end point target positions;

[0012] Step 4: The modal large model autonomously calls actions to combine and complete all operations according to the starting point, end point target position coordinates and input instructions;

[0013] Step 5: After the multimodal large model completes all operations, it outputs a feedback string value, which is converted into an audio file output using a speech synthesis algorithm to achieve human-computer dialogue communication.

[0014] Furthermore, the input instructions include at least multiple modes such as voice, text, image and video.

[0015] Furthermore, in step 2, an RGB-D camera is connected to the robotic arm to collect images and infrared distance data within the field of view of the depth camera.

[0016] Furthermore, in step 3: various types of target detection algorithms, semantic segmentation algorithms, template matching algorithms and all deep learning algorithms used for visual positioning are used to locate the target object and output the target position coordinate data.

[0017] Furthermore, in step 4: the multimodal large model uses the data of the target image database, function library and action skill knowledge base collected in advance to autonomously call actions to combine and complete all operations.

[0018] Furthermore, in step 5: after the multimodal large model executes all operations, it outputs a feedback whose value is a string, and uses different synthesis algorithms to convert the string into files in various forms such as voice, text, and pictures for output, thereby realizing human-computer interaction.

[0019] A connection device for an intelligent connection method of a measurement and control instrument system based on a multi-modal large model, wherein the measurement and control instrument system includes a mechanical arm for operating the measurement and control instrument, including:

[0020] The instruction splitting and processing module receives input instructions through the multimodal large model, and matches the corresponding recognition component according to the different types of input instructions to convert them into a form that can be read by the large model;

[0021] The input module obtains the end position picture taken when the robot arm is in the original position, and the start position picture taken when the robot arm moves to the overlooking state;

[0022] Detection module: The multimodal large model calls the deep learning algorithm to detect and locate the starting and ending target positions in the end point position image and the starting point position image, and outputs the starting and ending target position coordinates;

[0023] The intelligent agent action arrangement and execution module, the modal large model autonomously calls the action to combine and complete all operations according to the starting point, end point target position coordinates and input instructions;

[0024] In the human-computer interaction module, after the multimodal large model executes all operations, it outputs a feedback value as a string, which is converted into an audio file output using a speech synthesis algorithm to achieve human-computer dialogue communication.

[0025] It can be seen from the above technical solutions that the present invention provides a method and device for intelligent connection of a measurement and control instrument system based on a multi-modal large model, which has the following beneficial effects compared with the prior art:

[0026] The present invention utilizes the powerful information processing capability of the multimodal large model and the easy-to-operate human-computer interaction interface. After receiving the input command, the command is parsed, the deep learning algorithm is used for positioning, the required function and auxiliary prompt words are called to select appropriate actions from the action skill knowledge base for combination to complete the complex task, and finally feedback is given through the interactive interface. The present invention effectively reduces the disadvantage of the traditional method of strong dependence on the environment, improves the real-time processing capability and simplifies the operation method of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0028] Figure 1 A schematic diagram of the steps of an intelligent connection method for a measurement and control instrument system based on a multi-modal large model;

[0029] Figure 2 It is a structural schematic diagram of an intelligent connection device for a measurement and control instrument system based on a multi-modal large model;

[0030] Figure 3 This is a flowchart of the operation of the robotic arm described in Example 3;

[0031] Figure 4 This is a flowchart of the operation of the robotic arm described in Example 4;

[0032] Figure 5 This is a flowchart of the operation of the robotic arm described in Example 5;

[0033] Figure 6 This is a flowchart of the robot arm operation described in Example 6. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Embodiment 1:

[0036] The embodiment of the present invention discloses an intelligent connection method of a measurement and control instrument system based on a multi-modal large model, wherein the measurement and control instrument system includes a mechanical arm for operating the measurement and control instrument, see Figure 1 , including the following steps:

[0037] Step 1: Receive input instructions through the multimodal big model, and according to the different types of input instructions, match the corresponding recognition components to convert them into a form that can be read by the big model; the input instructions include at least multiple modes such as voice, text, image and video. Taking voice input as an example, after receiving the voice instruction, the multimodal big model uses the voice recognition component to convert it into text.

[0038] Step 2: Obtain the end position picture taken when the robot arm is in its original position, and the start position picture taken when the robot arm moves to a top-down state, and input the pictures into the multimodal vision model for visual understanding.

[0039] Specifically, an RGB-D camera can be connected to a robotic arm to collect images and infrared distance data within the field of view of the depth camera.

[0040] Step 3: The multimodal large model calls the deep learning algorithm to detect and locate the starting point and end point target positions in the end point position image and the starting point position image, and outputs the coordinates of the starting point and end point target positions.

[0041] Various types of target detection algorithms, semantic segmentation algorithms, template matching algorithms, and all deep learning algorithms used for visual positioning are used to locate the target object and output the target position coordinate data.

[0042] Step 4: The modal model uses the previously collected target image database, function library, and action skill knowledge base data to autonomously call actions and combine them to complete all operations based on the starting and ending target position coordinates and input instructions. Through detailed prompt words, the agent can flexibly and accurately select appropriate actions from the action skill knowledge base to combine and complete complex tasks.

[0043] Step 5: After the multimodal large model completes all operations, it outputs a feedback string value, which is converted into a file output in the form of speech, text, and pictures using a speech synthesis algorithm to achieve human-computer dialogue communication.

[0044] Embodiment 2:

[0045] The embodiment of the present invention also discloses an intelligent connection device for a measurement and control instrument system based on a multi-modal large model, wherein the measurement and control instrument system includes a mechanical arm for operating the measurement and control instrument, see Figure 2 ,include:

[0046] The instruction splitting and processing module receives input instructions through the multimodal large model, and matches the corresponding recognition component according to the different types of input instructions to convert them into a form that can be read by the large model;

[0047] The input module obtains the end position picture taken when the robot arm is in the original position, and the start position picture taken when the robot arm moves to the overlooking state;

[0048] Detection module: The multimodal large model calls the deep learning algorithm to detect and locate the starting and ending target positions in the end point position image and the starting point position image, and outputs the starting and ending target position coordinates;

[0049] The intelligent agent action arrangement and execution module, the modal large model autonomously calls the action to combine and complete all operations according to the starting point, end point target position coordinates and input instructions;

[0050] In the human-computer interaction module, after the multimodal large model executes all operations, it outputs a feedback value as a string, which is converted into an audio file output using a speech synthesis algorithm to achieve human-computer dialogue communication.

[0051] Furthermore, the application scenarios of the above-mentioned solutions of the present invention are introduced through the following embodiments.

[0052] Embodiment three:

[0053] A method for intelligently connecting a measurement and control instrument system based on a multimodal large model, wherein the measurement and control instrument system includes a mechanical arm for operating the measurement and control instrument, and the multimodal large model obtains voice instructions: identifies a BNC probe, grabs it and inserts it into the BNC interface of an oscilloscope RIGOL DS1102E, and returns to its original position after completion; when the mechanical arm is in the original position, takes a picture containing the oscilloscope RIGOL The image of DS1102E is taken, and the NanoDet target detection algorithm is called to obtain the position of the BNC interface, and the position coordinates of the interface are recorded, which are the end point coordinates; the robot arm moves to a top-down state, obtains the image within the field of view of the depth camera, and calls the NanoDet target detection algorithm again to detect the position of the BNC probe, and records the position coordinates, which are the starting point coordinates; the robot arm performs a grabbing operation based on the position of the BNC probe located by the detection algorithm; after grabbing, the DSNet semantic segmentation algorithm is called to obtain the precise position of the track on the BNC probe, and the infrared ranging function of the depth camera is used to calculate the length of the track position from the probe boundary, and the robot arm rotates the track to the center of the probe; after all information is processed, the intelligent agent that performs the intelligent choreography action independently decides to call the required function to perform the operation, and selects appropriate actions from the action skill knowledge base through detailed prompt words to combine and complete the entire insertion task; the multimodal large model outputs voice feedback: the operation has been completed. The robot arm operation flow chart in this scenario is as follows Figure 3 shown.

[0054] Embodiment 4:

[0055] A method for intelligently connecting a measurement and control instrument system based on a multimodal large model, wherein the measurement and control instrument system includes a robotic arm for operating the measurement and control instrument, and the multimodal large model obtains voice instructions: identifies a banana plug, grabs it and inserts it into the banana jack of a precision source meter module (SMU) B2901B, and returns to its original position after completion; when the robotic arm is in its original position, takes a picture containing the SMU Take a picture of B2901B, call the NanoDet target detection algorithm to get the position of the banana jack, and record the position coordinates of the jack, which are the end coordinates; the robot moves to a top-down state, gets a picture within the field of view of the depth camera, and calls the NanoDet target detection algorithm again to detect the position of the banana plug, and records the position coordinates, which are the starting point coordinates; the robot performs a grabbing operation based on the banana plug position located by the detection algorithm; after grabbing, call the DSNet semantic segmentation algorithm to get the precise position of the track on the banana jack; after processing all the information, the intelligent agent that performs the intelligent choreography action independently decides to call the required function to perform the operation, and selects appropriate actions from the action skill knowledge base through detailed prompt words to combine and complete the entire insertion task; the multimodal large model outputs voice feedback: the operation has been completed. The robot arm operation flow chart in this scenario is as follows Figure 4 shown.

[0056] Embodiment five:

[0057] A method for intelligent connection of a measurement and control instrument system based on a multimodal large model, wherein the measurement and control instrument system includes a robotic arm for operating the measurement and control instrument, and the multimodal large model obtains voice instructions: identify the DB9 female connector, grab it and insert it into the Multi interface of the industrial computer, and return to the original position after completion; when the robotic arm is in the original position, take a picture containing the industrial computer, call the NanoDet target detection algorithm and combine it with template matching to obtain the position of the Multi interface, and record the position coordinates of the socket, which are the end point coordinates; the robotic arm moves to a top-down state, obtains a picture within the field of view of the depth camera, and calls the NanoDet target detection algorithm again to detect the position of the DB9 female connector, and records the position coordinates, which are the starting point coordinates; the robotic arm performs a grabbing operation based on the position of the DB9 female connector located by the detection algorithm; after all information is processed, the intelligent agent that performs the intelligent choreography action independently decides to call the required function to perform the operation, and selects appropriate actions from the action skill knowledge base through detailed prompt words to combine and complete the entire insertion task; the multimodal large model outputs voice feedback: the operation has been completed. The robotic arm operation flow chart in this scenario is shown in the following figure. Figure 5 shown.

[0058] Embodiment six:

[0059] A method for intelligent connection of a measurement and control instrument system based on a multimodal large model, wherein the measurement and control instrument system includes a robotic arm for operating the measurement and control instrument, and the multimodal large model obtains voice instructions: identify the ground resistance test probe, grab it and insert it into the safety terminal of the programmable ground resistance tester, and return to the original position after completion; when the robotic arm is in the original position, take a picture containing the programmable ground resistance tester, call the NanoDet target detection algorithm to obtain the position of the safety terminal, and record the position coordinates of the terminal, which are the end point coordinates; the robotic arm moves to a top-down state, obtains a picture within the field of view of the depth camera, and calls the NanoDet target detection algorithm again to detect the position of the ground resistance test probe, and records the position coordinates, which are the starting point coordinates; the robotic arm performs a grabbing operation according to the banana plug position located by the detection algorithm; after all information is processed, the intelligent agent that performs the intelligent choreographed action independently decides to call the required function to perform the operation, and selects appropriate actions from the action skill knowledge base through detailed prompt words to combine and complete the entire insertion task; the multimodal large model outputs voice feedback: the operation has been completed. The robotic arm operation flow chart in this scenario is as follows Figure 6 shown.

[0060] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0061] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent connection method for a measurement and control instrument system based on a multi-modal large model, wherein the measurement and control instrument system includes a mechanical arm for operating the measurement and control instrument, characterized in that: The following steps are involved: Step 1: Receive input instructions through the multimodal big model, and according to the different types of input instructions, match the corresponding recognition components to convert them into a form that can be read by the big model; Step 2: Obtain the end position picture taken when the robot arm is in the original position, and the start position picture taken when the robot arm moves to the overlooking state; Step 3: The multimodal large model calls the deep learning algorithm to detect and locate the starting point and end point target positions in the end point position image and the starting point position image, and outputs the coordinates of the starting point and end point target positions; Step 4: The modal large model autonomously calls actions to combine and complete all operations according to the starting point, end point target position coordinates and input instructions; Step 5: After the multimodal large model completes all operations, it outputs a feedback string value, which is converted into an audio file output using a speech synthesis algorithm to achieve human-computer dialogue communication.

2. According to claim 1, a method for intelligent connection of a measurement and control instrument system based on a multi-modal large model is characterized in that: The input instructions include at least multiple modes such as voice, text, image and video.

3. According to claim 1, a method for intelligent connection of measurement and control instrument systems based on a multi-modal large model is characterized in that: In step 2, the RGB-D camera is connected to the robotic arm to collect images and infrared distance data within the field of view of the depth camera.

4. The intelligent connection method of a measurement and control instrument system based on a multi-modal large model according to claim 1 is characterized in that: In step 3: various types of target detection algorithms, semantic segmentation algorithms, template matching algorithms and all deep learning algorithms used for visual positioning are used to locate the target object and output the target position coordinate data.

5. The intelligent connection method of a measurement and control instrument system based on a multi-modal large model according to claim 1 is characterized in that: In step 4: the multimodal large model uses the data of the target image database, function library and action skill knowledge base collected in advance to autonomously call actions to combine and complete all operations.

6. The intelligent connection method of a measurement and control instrument system based on a multi-modal large model according to claim 1 is characterized in that: In step 5: after the multimodal large model executes all operations, it outputs a feedback whose value is a string, and uses different synthesis algorithms to convert the string into files in various forms such as voice, text, and pictures for output, thereby realizing human-computer interaction.

7. A connection device using the intelligent connection method of a measurement and control instrument system based on a multi-modal large model according to any one of claims 1 to 6, wherein the measurement and control instrument system includes a mechanical arm for operating the measurement and control instrument, characterized in that: include: The instruction splitting and processing module receives input instructions through the multimodal large model, and matches the corresponding recognition component according to the different types of input instructions to convert them into a form that can be read by the large model; The input module obtains the end position picture taken when the robot arm is in the original position, and the start position picture taken when the robot arm moves to the overlooking state; Detection module: The multimodal large model calls the deep learning algorithm to detect and locate the starting point and end point target positions in the end point position image and the starting point position image, and outputs the coordinates of the starting point and end point target positions; The intelligent agent action arrangement and execution module, the modal large model autonomously calls the action to combine and complete all operations according to the starting point, end point target position coordinates and input instructions; In the human-computer interaction module, after the multimodal large model executes all operations, it outputs a feedback value as a string, which is converted into an audio file output using a speech synthesis algorithm to achieve human-computer dialogue communication.

Images