Mechanical arm control method and device based on large model, equipment and storage medium

Abstract

The embodiment of the invention provides a mechanical arm control method and device based on a large model, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps that in response to an operation task request, multi-modal information is obtained, and the multi-modal information comprises original visual information and language instruction information; performing visual encoding on the original visual information through a visual encoder to obtain a visual feature vector; performing semantic coding on the language instruction information through a language coder to obtain a semantic feature vector; performing feature fusion on the visual feature vector and the semantic feature vector to obtain a multi-modal fusion feature; and the multi-mode fusion features are decoded through an action decoder, and control instruction information used for controlling the mechanical arm is obtained. The method can be applied to business systems needing a large amount of data, such as financial science and technology and health medical treatment, and the control accuracy of the mechanical arm can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and is applied to the fields of financial technology and health care, and particularly relates to a large model-based mechanical arm control method and device, equipment and a storage medium. BACKGROUND

[0002] The mechanical arm can be used to grasp, carry and place objects, etc. For example, in the financial technology field, the mechanical arm can help the bank to count and sort paper money or coins, thereby improving work efficiency and reducing human error. For example, in the health care field, the mechanical arm can be used to grasp and operate surgical instruments to assist medical operations. In addition, the mechanical arm can also be used to sort, check and distribute medicines, thereby reducing the manpower for manually distributing medicines and improving the efficiency and accuracy of medicine distribution.

[0003] In the related art, the control of the mechanical arm mainly relies on a task planning model for task planning and a strategy control model for control execution. The task planning model and the strategy control model are trained separately, which not only consumes a large amount of time and computing resources, but also causes poor information transmission in the transition stage from task planning to actual control during task execution, resulting in inaccurate mechanical arm movements. Therefore, how to improve the accuracy of mechanical arm control has become a technical problem to be solved. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a large model-based mechanical arm control method and device, equipment and a storage medium, which aims to improve the accuracy of mechanical arm control.

[0005] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application provides a large model-based mechanical arm control method, which comprises:

[0006] In response to an operation task request, multi-modal information is obtained, the multi-modal information comprising original visual information and language instruction information, and the language instruction information being natural language;

[0007] A preset visual language action large model is obtained, the visual language action large model comprising a visual encoder, a language encoder and an action decoder;

[0008] The original visual information is visually encoded by the visual encoder to obtain a visual feature vector;

[0009] The language instruction information is semantically encoded by the language encoder to obtain a semantic feature vector;

[0010] The visual feature vector and the semantic feature vector are fused to obtain a multi-modal fusion feature;

[0011] The multi-modal fusion feature is decoded by the action decoder to obtain control instruction information, wherein the control instruction information is used for controlling the robot arm.

[0012] In some embodiments, the visual language action large model further comprises a large language model, and the feature fusion of the visual feature vector and the semantic feature vector obtains a visual semantic fusion vector, comprising:

[0013] The semantic feature vector is enhanced by the large language model to obtain an enhanced semantic representation;

[0014] The visual feature vector and the semantic feature vector are preliminarily fused to obtain a preliminary fusion vector;

[0015] The preliminary fusion vector and the enhanced semantic representation are spliced to obtain the multi-modal fusion feature.

[0016] In some embodiments, the preliminary fusion of the visual feature vector and the semantic feature vector obtains a preliminary fusion vector, comprising:

[0017] The visual feature vector is normalized to obtain a visual normalized vector, and the semantic feature vector is normalized to obtain a semantic normalized vector;

[0018] The visual normalized vector and the semantic normalized vector are dimensionally aligned to obtain visual aligned vectors and semantic aligned vectors with the same dimension;

[0019] The visual aligned vector and the semantic aligned vector are fused to obtain the preliminary fusion vector.

[0020] In some embodiments, the decoding of the multi-modal fusion feature by the action decoder to obtain the control instruction information comprises:

[0021] The multi-modal fusion feature is key information extracted by the action decoder to obtain preliminary key information;

[0022] The preliminary key information is enhanced by the action decoder to obtain enhanced key information;

[0023] The enhanced key information is action predicted by the action decoder to obtain action prediction parameters;

[0024] The action prediction parameters are action planned by the action decoder to obtain a time sequence of actions;

[0025] The action decoder is used for decoding the time sequence of actions to obtain control instruction information.

[0026] In some embodiments, the action decoder is trained by a reinforcement learning mechanism, and the training comprises the following steps:

[0027] obtaining sample visual information and sample semantic information; wherein the sample visual information and the sample semantic information are used as a current state in the reinforcement learning mechanism;

[0028] sampling actions based on the sample visual information and the sample semantic information to obtain sample action information; wherein the sample action information is used as a current action in the reinforcement learning mechanism;

[0029] performing reward feedback based on the sample action information to obtain sample reward information and an updated sample state; wherein the updated sample state is a next state of the current state;

[0030] training the action decoder according to the sample reward information and the updated sample state.

[0031] In some embodiments, the language encoder is used for performing semantic encoding on the language instruction information to obtain a semantic feature vector, and the semantic encoding comprises the following steps:

[0032] The language encoder is used for performing semantic splitting on the language instruction information to obtain a semantic splitting unit.

[0033] The language encoder is used for performing word segmentation processing on the semantic splitting unit to obtain a semantic word segmentation sequence.

[0034] The language encoder is used for performing upper and lower position awareness encoding on the semantic word segmentation sequence to obtain the semantic feature vector.

[0035] In some embodiments, the visual encoder comprises an enhancement layer, a feature extraction layer, a multi-scale feature fusion layer and a feature full connection layer, and the visual encoder is used for performing visual encoding on the original visual information to obtain a visual feature vector, and the visual encoding comprises the following steps:

[0036] The enhancement layer is used for performing data enhancement processing on the original visual information to obtain enhanced visual information.

[0037] The feature extraction layer is used for performing feature extraction on the enhanced visual information to obtain visual local features and visual global features.

[0038] The multi-scale feature fusion layer is used for fusing the visual local features and the visual global features to obtain visual fusion features.

[0039] Map the visual fusion feature to a target dimension through the characteristic full connection layer to obtain the visual feature vector.

[0040] To achieve the above object, a second aspect of the embodiment of the present application proposes a large model-based mechanical arm control device, which comprises:

[0041] A multi-modal data acquisition module is configured to acquire multi-modal information in response to an operation task request, wherein the multi-modal information comprises original visual information and language instruction information, and the language instruction information is natural language.

[0042] A large model acquisition module is configured to acquire a preset visual language action large model, wherein the visual language action large model comprises a visual encoder, a language encoder and an action decoder.

[0043] A visual encoding module is configured to perform visual encoding on the original visual information through the visual encoder to obtain a visual feature vector.

[0044] A language encoding module is configured to perform semantic encoding on the language instruction information through the language encoder to obtain a semantic feature vector.

[0045] A feature fusion module is configured to perform feature fusion on the visual feature vector and the semantic feature vector to obtain multi-modal fusion features.

[0046] An action decoding module is configured to decode the multi-modal fusion features through the action decoder to obtain control instruction information, wherein the control instruction information is used to control a mechanical arm.

[0047] To achieve the above object, a third aspect of the embodiment of the present application proposes an electronic device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method of the first aspect when executing the computer program.

[0048] To achieve the above object, a fourth aspect of the embodiment of the present application proposes a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0049] The method and device for controlling a robot arm based on a large model, equipment and a storage medium provided by the embodiments of the present application obtain multi-modal information including original visual information and language instruction information in response to an operation task request, encode the original visual information by a visual encoder of a visual language action large model to obtain a visual feature vector, and encode the language instruction information by a language encoder of the visual language action large model to obtain a semantic feature vector, thereby fusing the visual feature vector and the semantic feature vector, and obtaining multi-modal fusion features that can cooperate visual information and semantic information, so as to facilitate decoding of the multi-modal fusion features by an action decoder, thereby obtaining control instruction information for controlling the robot arm. In the embodiments of the present application, the visual encoder, the language encoder and the action decoder are integrated in the same large model and are deeply fused and cooperated with each other, thereby improving the accuracy of the robot arm control. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a flowchart of the method for controlling a robot arm based on a large model provided by the embodiments of the present application;

[0051] Figure 2 is a flowchart of step 103 in Figure 1

[0052] Figure 3 is a flowchart of step 105 in Figure 1

[0053] Figure 4 is a flowchart of step 302 in Figure 3

[0054] Figure 5 is a flowchart of step 106 in Figure 1

[0055] Figure 6 is a flowchart of the method for controlling a robot arm based on a large model provided by another embodiment of the present application;

[0056] Figure 7 is a structural schematic diagram of the device for controlling a robot arm based on a large model provided by the embodiments of the present application;

[0057] Figure 8 is a hardware structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] ​​​​It should be noted that although the functional modules are divided in the device schematic diagram, the logical order is shown in the flowchart, but in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.

[0061] First, the meanings of several terms involved in the present application are analyzed:

[0062] Artificial intelligence (AI): is a new technical science that studies, develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0063] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.), NLP is a branch of artificial intelligence and is an interdisciplinary subject of computer science and linguistics, and is also commonly known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0064] Large Language Model (LLM): LLM is a deep learning model trained on a large amount of text data, enabling it to generate natural language text or understand the meaning of language text. LLM can be trained on a large dataset to learn language patterns and structures, enabling it to understand, generate, and reason text, and achieving strong general modeling and generalization capabilities. The core technology principle of LLM is to learn the patterns and structures of natural language through large-scale unsupervised training, simulating human language cognition and generation process to some extent. LLM can not only perform simple language tasks such as spelling check and grammar correction, but also handle complex tasks such as text summarization, machine translation, sentiment analysis, dialogue generation, and content recommendation. Through pre-training on large-scale datasets, LLM can be applied in finance, healthcare, and education.

[0065] Visual Encoder: Visual encoder can be used to convert images / videos (such as image pixel matrix, video frame sequence, etc.) into machine-understandable feature vectors, enabling machines to understand visual information such as color, texture, and object position in images. Visual encoder can be used in multi-modal general intelligence and self-supervised training models.

[0066] Language Encoder: Language encoder is a deep learning encoder used to process text data and is a core component of NLP. Language encoder can be used to convert text (such as sentences, paragraphs, etc.) into high-order feature representation, enabling the encoder to extract contextual features of text layer by layer, and finally convert the input sequence into hidden layer representation. Language encoder can be used for text classification, sentiment analysis, etc.

[0067] Vision-Language-Action (VLA): VLA is a large model that extends the multi-modal large model in embodied intelligence. VLA can form an end-to-end decision system by integrating visual perception, language understanding, and action control. VLA establishes a mapping between visual signals, language instructions, and physical actions to realize closed-loop decision-making from environment understanding to behavior output. Compared with traditional decision-making models, VLA model can break through the limitations of single-modal processing.

[0068] Reinforcement Learning (RL): Reinforcement learning is a machine learning method that learns optimal decision-making strategies through trial and error and environmental feedback. Reinforcement learning enables agents to interact with the environment and adjust strategies through trying different actions and receiving rewards or penalties feedback without explicit guidance.

[0069] Embodied Intelligence: Embodied intelligence is a technology at the intersection of artificial intelligence and robotics, in which an intelligent agent achieves autonomous learning and evolution through dynamic interaction between the body and the environment. The core technology of embodied intelligence is to deeply integrate perception, action, and cognition. Embodied intelligence can be applied to intelligent manufacturing, smart home, intelligent medical care, intelligent service, and other fields.

[0070] Embodied intelligent robot: An embodied intelligent robot is a robot equipped with embodied intelligence technology. The embodied intelligent robot realizes the closed loop of perception, cognition, decision-making, and action through real-time interaction between the physical entity and the environment. The embodied intelligent robot mainly includes a perception module, a decision-making system, and an execution mechanism. The perception module is used to obtain environmental information through multi-modal sensors, including vision, touch, hearing, etc. The decision-making system forms dynamic adaptability by fusing machine learning and cognitive reasoning algorithms. The execution mechanism includes bionic joints, flexible driving, and other hardware systems.

[0071] The mechanical arm can be used to grasp, carry, and place objects, etc. For example, in the financial technology scene, the mechanical arm can help the bank to count, sort paper money or coins, thereby improving work efficiency and reducing manual errors. For example, in the health care scene, the mechanical arm can be used to grasp and operate surgical instruments to assist medical operations. In addition, the mechanical arm can also be used for sorting, checking, and delivering medicines, thereby reducing the manpower of manually delivering medicines and improving the efficiency and accuracy of medicine delivery.

[0072] In the related art, the control of the mechanical arm mainly depends on the task planning model for task planning and the strategy control model for control execution. The task planning model and the strategy control model are trained separately, which not only consumes a large amount of time and computing resources, but also because the cooperation between the two models is not naturally close, there is often poor information transmission in the transition stage from task planning to actual control during task execution, resulting in inaccurate mechanical arm movements and possible movement delays. For example, in a complex industrial assembly scene, the task planning model formulates a plan to grasp a part and assemble it, but when it is passed to the control strategy model, the control strategy model cannot accurately perform the grasping action due to the difference in understanding of the environment, affecting the assembly efficiency and quality.

[0073] Moreover, current technologies are mostly trained and designed for specific scenarios and specific objects. When the environment changes slightly, such as the change in the angle of the object placement, the change in the lighting conditions, or the face of new shapes and materials of objects, the mechanical arm often has difficulty in accurately grasping. The model is difficult to learn a general grasping mode and strategy, and cannot effectively transfer the experience learned in one scene to other scenes. For example, in a logistics warehouse, if the previously trained model is only for grasping regular-shaped packaging boxes, it cannot complete the grasping task when it encounters irregular-shaped packages.

[0074] In addition, in current embodied intelligence technology, the visual module is responsible for identifying the object, and the language module is responsible for understanding the task description, but there is a lack of deep integration between the visual module and the language module and the final action execution module. When the information obtained by the visual module and the language module is transmitted to the action module, the information loss is serious, which makes it difficult for the robot arm to accurately generate the execution action according to the visual and language information. For example, when receiving the instruction of "grab the blue cylindrical part", the visual module may correctly identify the blue cylinder, but when transmitting to the action execution module, the robot arm cannot grab the part with the appropriate posture and force due to the coordination problem.

[0075] Based on this, the embodiment of the application provides a large model-based robot arm control method and device, equipment and storage medium, aiming to improve the accuracy of the robot arm in grabbing the object.

[0076] The large model-based robot arm control method and device, equipment and storage medium provided by the embodiment of the application are specifically described through the following embodiments. First, the large model-based robot arm control method in the embodiment of the application is described.

[0077] The embodiment of the application can acquire and process related data based on artificial intelligence technology. Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Theory, method, technology and application system.

[0078] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. several major directions.

[0079] The method for controlling a robot arm based on a large model provided in the embodiments of the present application relates to the technical field of artificial intelligence, and is specifically applied to the financial technology and health care scenarios. The method for controlling a robot arm based on a large model provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can further be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system formed by multiple physical servers, and can further be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms; and the software can be an application for implementing the method for controlling a robot arm based on a large model, etc., but is not limited to the above forms.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0081] It should be noted that in each specific embodiment of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of a user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of a user, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the embodiments of the present application to function normally will be obtained.

[0082] Figure 1 is an optional flowchart of the method for controlling a robot arm based on a large model provided in the embodiments of the present application, Figure 1The method in the method can include but is not limited to steps 101-106.

[0083] Step 101, in response to an operation task request, obtaining multi-modal information including original visual information and language instruction information, the language instruction information being natural language;

[0084] Step 102, obtaining a preset visual language action large model, the visual language action large model including a visual encoder, a language encoder and an action decoder;

[0085] Step 103, the original visual information is visually encoded by the visual encoder to obtain a visual feature vector;

[0086] Step 104, the language instruction information is semantically encoded by the language encoder to obtain a semantic feature vector;

[0087] Step 105, the visual feature vector and the semantic feature vector are fused to obtain a multi-modal fusion feature;

[0088] Step 106, the multi-modal fusion feature is decoded by the action decoder to obtain control instruction information; wherein the control instruction information is used to control the manipulator.

[0089] The steps 101-106 shown in the embodiments of the present application, by responding to an operation task request, obtaining multi-modal information including original visual information and language instruction information, encoding the original visual information by the visual encoder of the visual language action large model to obtain a visual feature vector, and encoding the language instruction information by the language encoder of the visual language action large model to obtain a semantic feature vector, thereby fusing the visual feature vector and the semantic feature vector to obtain a multi-modal fusion feature, and decoding the multi-modal fusion feature by the action decoder to obtain control instruction information for controlling the manipulator, the visual encoder, the language encoder and the action decoder are integrated in the same large model in the embodiments of the present application and are deeply fused and cooperated among them, thereby improving the accuracy of the manipulator control.

[0090] In step 101 of some embodiments, an operation task request is received for requesting a robot arm to perform a related operation on a manipulation object, where the manipulation object can be an article, for example, in a financial technology scenario, the manipulation object can be a bill, a seal, etc., the bill can be a banknote, a check, a deposit certificate, a bill of exchange, etc., and the related operation can be, for example, grabbing the bill and classifying it into a storage compartment, and ensuring that the bill is wrinkle-free, undamaged, and accurately classified when grabbing and classifying; for example, in a health care scenario, the manipulation object can be a medicine, a medical device, etc., where the medicine can be an ampoule or a medicine box, and the medical device can be a scalpel, a hemostat, etc., and the related operation can be, for example, grabbing the medicine or delivering the medical device to a doctor, and ensuring aseptic operation, undamaged medicine, and accurate delivery (error ≤ 1 cm) when delivering to the doctor.

[0091] In some embodiments, the multi-modal information in step 101 includes at least visual information and text information, where the visual information is raw visual information, for example, in a financial technology scenario, the raw visual information can be an image of a bill, an image of a seal, and an image of a storage compartment, and for example, in a health care scenario, the raw visual information can be an image of a medicine, an image of a medical device, etc. The text information is natural language-based text, and the text information is language instruction information, for example, in a financial technology scenario, the language instruction information can be “grab the bill in the intelligent vault and classify it into the corresponding storage compartment, and ensure that the bill is wrinkle-free, undamaged, and accurately classified”; for example, in a health care scenario, the language instruction information can be “please deliver the scalpel to Dr. Wang, and ensure aseptic operation, undamaged medicine, and accurate delivery with an error of ≤ 1 cm”.

[0092] In step 102 of some embodiments, the visual language action large model is VLA, which includes a visual encoder, a language encoder, and an action decoder. In an application scenario, the structure of the visual language action large model can be represented as VLA = f(visual encoder, language encoder, action decoder), where f represents the fusion and calculation process inside the VLA model.

[0093] In step 103 of some embodiments, the visual encoder is used to encode the raw visual information to obtain a visual feature vector, which can be represented as V_{feat} = visual encoder(I_{input}), where the visual feature vector is V_{feat} and the raw visual information is I_{input}.

[0094] In an application scenario, the visual encoder in step 103 can adopt a deep residual network or a ViT (Vision Transformer). The deep residual network can extract object appearance features, and the ViT can capture spatial relationships. The deep residual network can be a ResNet-50, which is a deep convolutional neural network based on a residual network (ResNet) architecture. The ResNet-50 introduces residual blocks and skip connections to solve the gradient vanishing and degradation problems in deep networks and support deeper network levels. The ViT can be used to capture spatial relationships. The ViT is a model that applies the Transformer architecture to computer vision tasks. The core technical principle of the ViT is to divide an image into a patch sequence, process image data through a self-attention mechanism, and divide the image into multiple 16x16 pixel blocks (patches). Each patch is converted into a vector through linear embedding to form a sequence input into a Transformer encoder. Unlike traditional convolutional neural networks (CNNs), the ViT directly treats an image as a serialized input. The ViT directly converts a 2D image into a 1D sequence and uses a self-attention mechanism to process pixel relationships in the image to capture the relevance of different positions in the image. The ViT completely discards the CNN structure and only uses the Transformer to process image data, while the CNN relies on convolutional layers and pooling operations. The embodiment of the present application achieves stronger context modeling capability through the use of the fully connected layer of the ViT.

[0095] The visual encoder includes an enhancement layer, a feature extraction layer, a multi-scale feature fusion layer, and a feature full connection layer. Please refer to Figure 2 In step 103 of some embodiments, the following can be included but are not limited to:

[0096] Step 201: performing data enhancement processing on the original visual information through the enhancement layer to obtain enhanced visual information.

[0097] Step 202: performing feature extraction on the enhanced visual information through the feature extraction layer to obtain visual local features and visual global features.

[0098] Step 203: obtaining visual fusion features by fusing the visual local features and the visual global features through the multi-scale feature fusion layer.

[0099] Step 204: mapping the visual fusion features to a target dimension through the feature full connection layer to obtain a visual feature vector.

[0100] In some embodiments, before step 201 is performed, the large model based robot control method further includes: pre-processing the original visual information, such as normalization processing, size adjustment, channel alignment, etc. Specifically, for the RGB image in the original visual information, ImageNet mean and standard deviation can be used for normalization processing, and for the depth map in the original visual information, pixels in the distance range of 0-2m can be normalized to [0, 1], and pixels beyond the range are set to 0 to represent invalid areas. In an application scenario, the size can be adjusted to: scaling according to the model input resolution (for example, ResNet-50 is scaled to 224x224); channel alignment can be, for example: adjusting the channel number of the depth map to 3 through a 1x1 convolution, and splicing with the RGB image to realize 6-channel input.

[0101] In step 201 of some embodiments, the data augmentation processing of the original visual information can include at least one of the following: color enhancement, shape enhancement, material enhancement, layout enhancement, etc. For example, color enhancement can be: for color information (such as anti-counterfeit colors of financial bills, label colors of medical drugs, etc.), by adjusting brightness (±30%), contrast (±20%), saturation (±20%), etc. to generate variants, thereby avoiding the influence of light changes; shape enhancement can be, for example: for shape information (such as the edges of financial bills, the contours of surgical instruments, etc.), by randomly rotating (for example, -15° to 15°), scaling (for example, 0.8 to 1.2 times) to generate variants; material enhancement can be, for example: enhancing material information, including wrinkles and wear, such as the reflection of glass, the wrinkles of paper, etc. By adding Gaussian noise to simulate rough surfaces, blurring processing (such as simulating smooth surfaces), and reflection spots (such as simulating metal / glass reflection) to generate variants, for example, Gaussian blur can be added to metal instruments (such as needle holders) to simulate the reflection interference of surgical shadowless lamps; layout enhancement can be, for example: for environmental layout (such as the relative positions of multiple objects), by randomly cropping and randomly arranging objects (such as different placement positions of bills on the table surface) to generate variants, so as to learn the global layout rules.

[0102] In step 202 of some embodiments, local feature extraction and global feature extraction are performed on the enhanced visual information by a feature extraction layer to obtain visual local features and visual global features. The visual local features are features extracted from local regions of an image and can reflect detailed attributes of an object, such as local texture and detailed structure of the object, attributes of key parts, features of a small scale, etc. The visual local features can be, for example, micro-text on a bill, metal line concave-convex texture of a deposit edge, edge anti-counterfeit tooth pattern of a "financial special seal", a tear in the upper right corner of a check, sharpness of a scalpel blade, occlusion tooth pattern of a hemostat, etc. The visual global features are features extracted from an overall perspective of an image and can reflect macro attributes of a scene, such as overall distribution and relative position of objects, environmental attributes of a scene, etc. The overall distribution and relative position of objects can be, for example, arrangement of bills on a vault table, 3 stacked deposits on the left and 2 flat checks on the right, a red seal 10 cm above the checks, a magnetic stripe reader 5 cm to the right, etc. The environmental attributes of a scene can be, for example, illumination intensity of an operating room and layout of an instrument table, etc. The visual global features can also be, for example, 5 deposits of 10,000 yuan each, most of the bills being flat with only one bill in the lower right corner having obvious wrinkles, the instrument table being in the sterile area and the doctor's gloves not touching the non-sterile area, etc. The visual global features can also be, for example, a scalpel, a hemostat, and a suture needle arranged in order on the left side of the instrument tray, 2 scalpel blades, 3 hemostats, and 1 suture needle box in the surgical tray, an ampoule bottle not being opened, a medicine box not being damaged, and all metal instruments not having obvious rust, etc.

[0103] In step 203 of some embodiments, pyramid pooling is performed on the visual local features and the visual global features by a multi-scale feature fusion layer to convert multi-scale feature maps to the same size, compress the number of channels by convolution, and then perform multi-scale feature splicing to obtain visual fusion features. For example, the number of channels is compressed to 64 dimensions, 128 dimensions, 256 dimensions, etc. by a 1x1 convolution layer, and the dimensions of the visual fusion features obtained by splicing the multi-scale features are 64+128+256=448.

[0104] In step 204 of some embodiments, the visual fusion features are mapped to a target dimension by a feature full connection layer. The target dimension can be, for example, 512 dimensions. The feature full connection layer can be, for example, 2 layers. In an application scenario, 448-dimensional visual fusion features are input into a first layer of the feature full connection layer for processing. The first layer of the feature full connection layer outputs a 1024-dimensional feature vector to a second layer of the feature full connection layer for processing. The second layer of the feature full connection layer outputs a 512-dimensional visual feature vector. In addition, to ensure that the vectors are processed in a unified scale, step 204 further includes normalizing the visual feature vector, such as by L2 normalization.

[0105] The visual feature vector obtained through the steps 201 to 204 can contain key information such as the position, shape, color, material and environmental layout of the target object.

[0106] Taking the financial technology scene as an example, the visual encoder encodes the original visual information of the bills (checks, deposit certificates), seals and cash extracted from the bank monitoring image, so that the visual feature vector obtained can contain the position / shape (such as distinguishing between checks and deposit certificates), color / material (such as the color and texture of the anti-counterfeiting watermark), and environmental layout (such as the relative position of the bill stack state and the seal), so that the subsequent mechanical arm can be classified according to the instructions.

[0107] Taking the health care scene as an example, the visual encoder encodes the original visual information of surgical instruments (such as surgical knives and hemostatic forceps) and medicines (such as ampoule bottles and medicine boxes) extracted from the endoscope or monitoring image in the operating room, so that the visual feature vector obtained can contain the shape / material of the instrument (such as distinguishing between metal surgical knives and plastic forceps), the color / position of the medicine, and the environmental layout (such as the placement position of the instrument on the sterile table), wherein the color of the medicine can be used to distinguish the label color of different medicines, such as distinguishing between penicillin and cephalosporin, so that the subsequent mechanical arm can assist the doctor in delivering surgical instruments and medicines according to the instructions.

[0108] In step 104 of some embodiments, the language instruction information is encoded by the language encoder to obtain a semantic feature vector, which can be represented as: L_{sem}=language encoder(S_{input}), wherein the semantic feature vector is L_{sem}, and the language instruction information S_{input} is based on a natural language task description. The language instruction information may, for example, be "take the water cup on the table to the kitchen".

[0109] In an application scenario, the language encoder in step 104 can use BERT-base or RoBERTa (Robustly Optimized BERT Pretraining Approach). BERT-base is a natural language processing (NLP) model based on the Transformer architecture, which is the original version of BERT and can better understand the semantics of instructions. RoBERTa is an improved version of BERT and is a robust optimization method for BERT pre-training, which can handle complex sentence patterns.

[0110] In step 104 of some embodiments, the following can be included but are not limited to:

[0111] The language instruction information is semantically split by a language encoder to obtain a semantic split unit;

[0112] The semantic split unit is processed by the language encoder to obtain a semantic word segmentation sequence;

[0113] The semantic word segmentation sequence is encoded by the language encoder to obtain a semantic feature vector.

[0114] The core elements are identified by the semantic split of the language instruction information, and the core elements include, for example, an action element, an object element, and a constraint element. The action element can be, for example, grabbing, sorting, etc. The object element can be, for example, a check, a deposit certificate, a storage compartment, a surgical knife, a hemostat, a medicine, a medicine box, etc. The constraint element can be, for example, wrinkle-free, damage-free, accurate sorting, and delivery of the surgical knife to Dr. Wang to ensure sterility.

[0115] The semantic word segmentation sequence is obtained by processing the semantic split unit. For example, BERT-base can be used for word segmentation processing by using WordPiece. For example, RoBERTa can be used for word segmentation processing by using ByteLevelBPE, which can support rare words, such as the complete encoding of the medical term “ampoule bottle”, to avoid semantic loss caused by excessive splitting of subwords. For example, in the financial technology scenario, the semantic word segmentation sequence obtained by performing semantic split and word segmentation processing on the language instruction information “grab the bill in the intelligent vault and sort it into the corresponding storage compartment, ensuring that the bill is wrinkle-free, damage-free, and accurately sorted” includes “in”, “intelligent”, “vault”, “grab”, “bill”, “,”, “and”, “sort”, “into”, “corresponding”, “storage compartment”, “,”, “ensure”, “bill”, “wrinkle-free”, “,”, “damage-free”, “,”, and “accurately sorted”. For example, in the health care scenario, the semantic word segmentation sequence obtained by performing semantic split and word segmentation processing on the language instruction information “please deliver the surgical knife to Dr. Wang and ensure sterile operation, undamaged medicine, and accurate delivery error ≤1 cm” includes “please”, “deliver”, “surgical knife”, “to”, “Dr. Wang”, “,”, “and”, “ensure”, “sterile operation”, “,”, “undamaged medicine”, “,”, “accurate delivery”, “error”, “≤1 cm”.

[0116] The semantic word segmentation sequence is encoded by BERT-base or RoBERTa to obtain a semantic feature vector, which can achieve semantic enhancement and facilitate subsequent semantic understanding by the action encoder.

[0117] In some embodiments, the visual language action large model further comprises a large language model, which is an LLM, and the structure of the visual language action large model can be represented as VLA = f(visual encoder, language encoder, LLM, action decoder).

[0118] In an application scenario, the action encoder in step 105 can adopt a Transformer-based Seq2Seq or an MLP+MPC joint decoding. The joint decoding can output a joint trajectory sequence through the Transformer-based Seq2Seq, and can combine the dynamics constraint through the MLP+MPC joint decoding.

[0119] Referring to Figure 3 In step 105 of some embodiments, the following can be included but are not limited to:

[0120] Step 301: enhancing the semantic feature vector through the large language model to obtain an enhanced semantic representation;

[0121] Step 302: preliminarily fusing the visual feature vector and the semantic feature vector to obtain a preliminary fusion vector;

[0122] Step 303: concatenating the preliminary fusion vector and the enhanced semantic representation to obtain a multi-modal fusion feature.

[0123] In step 301 of some embodiments, the LLM, based on its pre-trained knowledge system, further understands and infers the semantic feature vector obtained based on the language encoder to obtain an enhanced semantic representation. In an application scenario, the semantic feature vector can be represented as L_{enhanced}.

[0124] Referring to Figure 4 In step 302 of some embodiments, the following can be included but are not limited to:

[0125] Step 401: normalizing the visual feature vector to obtain a visual normalized vector, and normalizing the semantic feature vector to obtain a semantic normalized vector;

[0126] Step 402: aligning the visual normalized vector and the semantic normalized vector in dimension to obtain a visual aligned vector and a semantic aligned vector with the same dimension;

[0127] Step 403: fusing the visual aligned vector and the semantic aligned vector to obtain a preliminary fusion vector.

[0128] In step 401 of some embodiments, by normalizing the visual feature vector, the light difference in the image can be eliminated, and by normalizing the semantic feature vector, the text length influence can be eliminated.

[0129] In step 402 of some embodiments, by dimensionally aligning the visual normalized vector and the semantic normalized vector, it is ensured that they have the same dimension. Specifically, a fully connected layer can be used to project to the same dimension, for example, both to 512 dimensions.

[0130] After normalization and dimension alignment, step 403 is performed for feature fusion, so as to obtain a preliminary fusion vector. Specifically, the visual aligned vector and the semantic aligned vector can be fused based on a multilayer perceptron (MLP). MLP is a kind of feedforward neural network, which usually includes an input layer, at least one hidden layer and an output layer, the neurons of each layer are fully connected with all neurons of the adjacent layer to pass signals through weight and bias parameters; the hidden layer of MLP introduces a nonlinear transformation through an activation function (such as Sigmoid, ReLU), so as to break through the limitation of single-layer perceptron that can only process linear classification. The hierarchical structure of MLP can be used to process complex nonlinear problems.

[0131] Through steps 401 to 403 of the embodiments of the present application, the visual feature vector is normalized and dimensionally aligned, so as to be preliminarily fused by a fusion layer (such as MLP), thereby facilitating the re-fusion in step 303.

[0132] In step 302 of the embodiments of the present application, the execution order of steps 401 and 402 is not limited, that is, step 401 can be executed first for normalization, and then step 402 can be executed for dimension alignment, or the dimension alignment can be performed first, and then the normalization can be performed, and the embodiments of the present application are described by taking the normalization first as an example.

[0133] In step 303 of some embodiments, by splicing the preliminary fusion vector and the enhanced semantic representation, the visual, semantic and enhanced semantic representation are deeply fused, so as to obtain a multi-modal fusion feature. Specifically, the preliminary fusion vector and the enhanced semantic representation can be fused and spliced based on a cross-attention mechanism, in an application scenario, the preliminary fusion vector can be represented as F_{fusion}, and the multi-modal fusion feature can be represented as I_{decoder}, which is used as the input of the action decoder in step 106, I_{decoder}=Concat(F_{fusion},L_{enhanced}), wherein Concat is used to represent the splicing operation.

[0134] Through steps 301 to 303 of the embodiment of the present application, the semantic feature vector is enhanced by the large language model, and the visual feature vector and the semantic feature vector are preliminarily fused, and then the preliminary fusion vector and the enhanced semantic representation are deeply fused, so as to obtain a multi-modal fusion feature containing interaction information between vision and semantics, so as to facilitate the action decoder in step 106 to generate control instruction information for controlling the action of the robot arm.

[0135] Through multi-modal information fusion optimization, the robot arm can more comprehensively and accurately understand task and environment information, thereby generating more accurate and appropriate grasping actions. For example, in a complex home scene, when receiving the task instruction "take the water cup on the table to the kitchen", the visual encoder in the VLA model can combine the visual recognition of the table, the water cup and the kitchen location, and the understanding of the meaning of the task instruction by the language encoder and the LLM, so that the action encoder can plan and execute accurate grasping and moving actions.

[0136] Please refer to Figure 5 In step 106 of some embodiments, it can include but is not limited to including:

[0137] Step 501, extracting key information based on the action decoder to the multi-modal fusion feature to obtain preliminary key information;

[0138] Step 502, enhancing the preliminary key information based on the action decoder to obtain enhanced key information;

[0139] Step 503, action prediction based on the action decoder to the enhanced key information to obtain action prediction parameters;

[0140] Step 504, action planning based on the action decoder to the action prediction parameters to obtain a time sequence of actions;

[0141] Step 505, format conversion based on the action decoder to the time sequence of actions to obtain control instruction information.

[0142] In step 501 of some embodiments, key information extraction can be performed on the multi-modal fusion features based on a Transformer layer and a fully connected layer. The preliminary key information obtained can include, but is not limited to, target object information, semantic constraint information, and environmental state information. The target object information can include, but is not limited to, 3D coordinates, shape types (such as rectangular / cylindrical, etc.), material hardness, etc. The semantic constraint information can include, but is not limited to, action types (such as gentle grasping, precise clamping, etc.), constraint thresholds (such as clamping force ≤ 3N, position error ≤ 2mm, sterile area boundary, etc.), etc. The environmental state information can include, but is not limited to, obstacle coordinates (such as card readers in a vault, instrument trays in an operating room), current joint angles of the robotic arm, etc.

[0143] In step 502 of some embodiments, the preliminary key information can be enhanced. The enhancement can include, but is not limited to, explicit enhancement of the extracted information. For example, the relative distance between the target object and the current position of the robotic arm can be calculated as the basis for motion planning. Prohibited conditions in the semantic constraints (such as prohibition of touching the blade, prohibition of wrinkling, etc.) can be converted into mask features (such as blade area coordinate mask), so as to avoid the action parameters falling into the forbidden zone.

[0144] In step 503 of some embodiments, the enhanced key information is used to perform hierarchical prediction of action parameters, thereby obtaining action prediction parameters. For example, the core parameters of the robotic arm grasping can be predicted by a multi-branch neural network, ensuring that each action prediction parameter is accurately matched with the object attributes and semantic constraints. In actual application scenarios, the grasping action involves five types of core action prediction parameters: position, pose, clamping force, motion speed, and motion duration.

[0145] In step 504 of some embodiments, action planning is performed based on the action prediction parameters, resulting in a time sequence action sequence representing actions in each stage. The time sequence action sequence can include a close action sequence, a contact action sequence, a clamp action sequence, and a lift action sequence. The close action sequence is used to represent a sequence of moving to a safe distance above the object in the close stage, for example. The contact action sequence is used to represent slowly descending to the surface of the object in the contact stage. The clamp action sequence is used to represent closing the end to a target force in the clamp stage, for example. The force can be set to 2.5 N for 0.5 seconds to ensure stability and no wrinkles in the financial technology scenario. For example, the clamp force for delivering a surgical knife in the health care scenario can be set to 4.5 N for 0.8 seconds. Since higher stability is required in the medical scenario, the duration is set to be longer. The lift action sequence is used to represent lifting the object to a safe height in the lift stage, for example. The speed of lifting the bill in the financial technology scenario is 0.2 m / s, and the position is (0.45 m, 0.34 m, 0.9 m). For example, the speed of delivering a surgical knife in the health care scenario is 0.15 m / s, and the position is (0.35 m, 0.63 m, 1.1 m).

[0146] Embodiments of the present application can use a bidirectional LSTM network (hidden layer dimension 512) to plan actions for action prediction parameters to ensure that the end position of the previous stage is the starting position of the next stage, and the deceleration process lasts for 0.2 s. In addition, each action prediction parameter is associated through attention mask, for example, only the force parameter is activated in the clamp stage, and the other stages (close stage, contact stage, and lift stage) are frozen.

[0147] In step 505 of some embodiments, each time sequence action sequence is converted into a format instruction for controlling the action of the robot arm, resulting in control instruction information. In some embodiments, the control instruction information can include, but is not limited to, at least one of the following: target state (joint angle / Cartesian coordinate), motion constraint (upper limit of speed, obstacle avoidance area), force control parameter (grasping force threshold), time sequence information (action stage identifier), and environment feedback (such as object pose change compensation amount).

[0148] Please refer to Figure 6 In some embodiments, the robot arm control method based on a large model can include, but is not limited to, the following:

[0149] In step 601, sample visual information and sample semantic information are obtained. The sample visual information and sample semantic information are used as the current state in the reinforcement learning mechanism.

[0150] In step 602, action sampling is performed based on the sample visual information and the sample semantic information to obtain sample action information; wherein the sample action information is taken as a current action in the reinforcement learning mechanism.

[0151] In step 603, reward feedback is performed based on the sample action information to obtain sample reward information and an updated sample state; wherein the updated sample state is a next state of the current state.

[0152] In step 604, the action decoder is trained according to the sample reward information and the updated sample state.

[0153] In step 601 of some embodiments, the sample visual information and the sample semantic information are taken as a current state S_t in the reinforcement learning mechanism. For example, in a financial technology scenario, the sample visual information can be an image of a bill, an image of a seal, and an image of a storage compartment. For example, in a health scenario, the sample visual information can be an image of a drug and an image of a medical device. The sample semantic information is a text based on natural language. For example, in a financial technology scenario, the sample semantic information can be "grab the bill in the intelligent safe and put it in the corresponding storage compartment, and ensure that the bill is wrinkle-free, damage-free, and accurately classified". For example, in a health care scenario, the sample semantic information can be "please pass the scalpel to Dr. Wang and ensure aseptic operation, undamaged drugs, and an error of ≤1 cm in accurate delivery".

[0154] In step 602 of some embodiments, action sampling is performed according to a policy network. Specifically, the sample visual information and the sample semantic information are taken as inputs of the policy network, and the policy network obtains an action A_t, i.e., sample action information A_t, by random sampling. For example, in a financial technology scenario, the action A_t can be the "grab" action in "grab the bill". For example, in a health care scenario, the action A_t can be the "pass to" action in "please pass the scalpel to Dr. Wang".

[0155] In an application scenario, the policy network is an Actor-Critic network, i.e., the policy network includes an Actor network and a Critic network. The Actor network is used to output an action distribution according to the sample visual information and the sample semantic information. The Critic network is used to perform state prediction according to the sample visual information, the sample semantic information, and the action distribution output by the Actor network to obtain a state value.

[0156] In step 603 of some embodiments, a new state S_{t+1}, i.e., an updated sample state S_{t+1}, is obtained by performing the action A_t, and sample reward information R_{t+1} is obtained by reward calculation, so that the action decoder is trained in step 604.

[0157] The embodiments of the present application learn a generalizable action policy by setting a diversified reward mechanism. In the training process, an action A_t is selected according to the current state S_t (including visual information, semantic information, etc.), the reward R_{t+1} and the new state S_{t+1} of the environment feedback are obtained after the action is executed, and the long-term cumulative reward is maximized to train the action decoder. For example, in the application scenario of the grasping task, the policy network outputs joint torque, if the end effector of the robot successfully touches the target object and there is no collision, the reward stacking task completion reward is triggered to accelerate the policy convergence.

[0158] The embodiments of the present application can be trained in a diversified simulation environment, which can include but is not limited to different object shapes, sizes, materials, different placement positions and angles, and different lighting, occlusion and other environmental factors. By training in a diversified simulation environment, the general grasping mode and the strategy for responding to different environmental changes are gradually mastered through continuous learning. When encountering a new scene or object, the action decision can be made according to the learned generalization strategy, so as to realize accurate grasping. For example, in the training process, how to adjust the grasping posture for objects with different center of gravity distribution to ensure stable grasping is learned, and when encountering a new object with a different center of gravity distribution, the posture can be adjusted based on the learned strategy.

[0159] The embodiments of the present application use the same VLA large model to complete task planning and control, eliminating the delay and information loss of inter-model cooperation, so that the robot arm can respond to task instructions faster and improve the task completion amount per unit time. In the application scenario of the industrial production line, the production efficiency can be significantly improved and the production cost can be reduced by using the embodiments of the present application. For example, a grasping and assembling task that originally takes 10 seconds can be shortened to 5 seconds by using the embodiments of the present application.

[0160] The VLA large model trained based on reinforcement learning in the embodiments of the present application has strong generalization ability, so that the robot arm can accurately complete the grasping task in various complex and variable environments, greatly expanding the application range of embodied intelligence in practical scenarios, such as handling various shapes and packaging goods in logistics sorting; in home services, different home layouts and object placement situations can be handled. Taking a logistics warehouse as an example, a conventional robot arm can only grasp a limited number of goods, while the robot arm in the embodiments of the present application can grasp common types of goods, greatly improving the degree of logistics automation.

[0161] The multi-modal information fusion in the embodiments of the present application makes the mechanical arm more accurate in understanding the task and the environment, and the grasping action is more accurate, thereby reducing the failure in the grasping process and improving the success rate of task execution. In the medical surgery auxiliary scene, the mechanical arm can more accurately grasp and operate the surgical instruments, reducing the risk of surgery; in electronic manufacturing, the grasping and assembly precision of precision parts can be improved, thereby improving the product quality.

[0162] Please refer to Figure 7 The embodiments of the present application also provide a mechanical arm control device based on a large model, which comprises:

[0163] A multi-modal data acquisition module is configured to acquire multi-modal information in response to an operation task request, wherein the multi-modal information comprises original visual information and language instruction information, and the language instruction information is natural language.

[0164] A large model acquisition module is configured to acquire a preset visual language action large model, wherein the visual language action large model comprises a visual encoder, a language encoder and an action decoder.

[0165] A visual encoding module is configured to perform visual encoding on the original visual information through the visual encoder to obtain a visual feature vector.

[0166] A language encoding module is configured to perform semantic encoding on the language instruction information through the language encoder to obtain a semantic feature vector.

[0167] A feature fusion module is configured to perform feature fusion on the visual feature vector and the semantic feature vector to obtain multi-modal fusion features.

[0168] An action decoding module is configured to decode the multi-modal fusion features through the action decoder to obtain control instruction information, wherein the control instruction information is used for controlling the mechanical arm.

[0169] In some embodiments, the visual encoding module can be specifically configured to implement:

[0170] Perform data enhancement processing on the original visual information through an enhancement layer to obtain enhanced visual information;

[0171] Perform feature extraction on the enhanced visual information through a feature extraction layer to obtain visual local features and visual global features;

[0172] Fuse the visual local features and the visual global features through a multi-scale feature fusion layer to obtain visual fusion features;

[0173] Map the visual fusion features to a target dimension through a feature full connection layer to obtain the visual feature vector.

[0174] Specifically, the visual encoding module can be configured to implement the steps 201-204.

[0175] In some embodiments, the feature fusion module can be configured to implement:

[0176] performing enhancement processing on the semantic feature vector by the large language model to obtain an enhanced semantic representation;

[0177] fusing the visual feature vector and the semantic feature vector to obtain a preliminary fusion vector;

[0178] concatenating the preliminary fusion vector and the enhanced semantic representation to obtain a multi-modal fusion feature.

[0179] Specifically, the feature fusion module can be configured to implement the steps 301-303.

[0180] In some embodiments, the feature fusion module can be configured to implement:

[0181] performing normalization processing on the visual feature vector to obtain a visual normalized vector, and performing normalization processing on the semantic feature vector to obtain a semantic normalized vector;

[0182] aligning the visual normalized vector and the semantic normalized vector in dimension to obtain a visual aligned vector and a semantic aligned vector with the same dimension;

[0183] performing feature fusion on the visual aligned vector and the semantic aligned vector to obtain a preliminary fusion vector.

[0184] Specifically, the feature fusion module can be configured to implement the steps 401-403.

[0185] In some embodiments, the action decoding module can be configured to implement:

[0186] performing key information extraction on the multi-modal fusion feature based on the action decoder to obtain preliminary key information;

[0187] performing enhancement processing on the preliminary key information based on the action decoder to obtain enhanced key information;

[0188] performing action prediction on the enhanced key information based on the action decoder to obtain an action prediction parameter;

[0189] performing action planning on the action prediction parameter based on the action decoder to obtain a time-series action sequence;

[0190] performing format conversion on the time-series action sequence based on the action decoder to obtain control instruction information.

[0191] Specifically, the action decoding module can be configured to implement the steps 501-505.

[0192] In some embodiments, the large model based robotic arm control device can also be used to implement:

[0193] Obtain sample visual information and sample semantic information; wherein the sample visual information and the sample semantic information are used as the current state in the reinforcement learning mechanism;

[0194] Sample actions based on sample visual information and sample semantic information to obtain sample action information; wherein the sample action information is used as the current action in the reinforcement learning mechanism;

[0195] Reward feedback based on sample action information to obtain sample reward information and updated sample state; wherein the updated sample state is the next state of the current state;

[0196] Train the action decoder according to the sample reward information and the updated sample state.

[0197] Specifically, the large model based robotic arm control device can be used to implement the above steps 601 to 604.

[0198] The specific implementation of the large model based robotic arm control device is basically the same as the above described specific embodiments of the large model based robotic arm control method, and will not be repeated here.

[0199] The embodiments of the present application also provide an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above described large model based robotic arm control method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0200] Please refer to Figure 8 , Figure 8 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0201] The processor 801 can be implemented in the form of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application;

[0202] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 802 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 802 and are called and executed by the processor 801 to implement the large model-based mechanical arm control method of the embodiments of the present application;

[0203] The input / output interface 803 is configured to realize information input and output.

[0204] The communication interface 804 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0205] The bus 805 is configured to transmit information between various components (for example, the processor 801, the memory 802, the input / output interface 803, and the communication interface 804) of the device.

[0206] The processor 801, the memory 802, the input / output interface 803, and the communication interface 804 are connected to each other through the bus 805 to realize the communication connection between them in the device.

[0207] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the large model-based mechanical arm control method.

[0208] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0209] The method and device for controlling a robot arm based on a large model, the equipment and the storage medium provided by the embodiments of the present application encode original visual information through a visual encoder of a visual language action large model to obtain a visual feature vector, and encode language instruction information through a language encoder of the visual language action large model to obtain a semantic feature vector, thereby fusing the visual feature vector and the semantic feature vector to obtain a multi-modal fusion feature, and decoding the multi-modal fusion feature through an action decoder to obtain control instruction information for controlling the robot arm. In the embodiments of the present application, the visual encoder, the language encoder and the action decoder of the same VLA large model are used, and the visual encoder, the language encoder and the action decoder are fused and cooperated through depth to improve the accuracy of the robot arm control. The VLA large model trained based on reinforcement learning in the embodiments of the present application has strong generalization ability, so that the robot arm can accurately complete the grasping task in various complex and variable environments, greatly expanding the application range of embodied intelligence in actual scenes, and the multi-modal information fusion makes the robot arm more accurately understand the task and the environment, and the grasping action is more accurate, thereby reducing the errors in the grasping process and improving the success rate of task execution.

[0210] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0211] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps or different steps.

[0212] The device embodiments described above are only schematic, and the units illustrated as separate components can or can not be physically separated, that is, they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0213] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functions of the modules / units in the system and the equipment can be implemented as software, firmware, hardware and their appropriate combinations.

[0214] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a recited step or its integral sub-steps or additional steps whether or not readily ascertainable from the description or the like. Further, the words "a" or "an", as used herein in the disclosure and elsewhere, are used indiscriminately and are to be interpreted in the same way, i.e. as meaning "one or more".

[0215] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or", used to describe the relationship between associated objects, means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including single or multiple combinations. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0216] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0217] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0218] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0219] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0220] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A large model-based robot arm control method, characterized by, The method comprises: in response to an operation task request, acquiring multi-modal information, the multi-modal information comprising original visual information and language instruction information, the language instruction information being natural language; acquiring a preset visual language action large model, the visual language action large model comprising a visual encoder, a language encoder and an action decoder; performing visual coding on the original visual information through the visual encoder to obtain a visual feature vector; performing semantic coding on the language instruction information through the language encoder to obtain a semantic feature vector; performing feature fusion on the visual feature vector and the semantic feature vector to obtain a multi-modal fusion feature; decoding the multi-modal fusion feature through the action decoder to obtain control instruction information; wherein the control instruction information is used for controlling a mechanical arm.

2. The method of claim 1, wherein, The visual language action large model further comprises a large language model, and the performing feature fusion on the visual feature vector and the semantic feature vector to obtain a visual semantic fusion vector comprises: performing enhancement processing on the semantic feature vector through the large language model to obtain an enhanced semantic representation; performing preliminary fusion on the visual feature vector and the semantic feature vector to obtain a preliminary fusion vector; splicing the preliminary fusion vector and the enhanced semantic representation to obtain the multi-modal fusion feature.

3. The method of claim 2, wherein, The performing preliminary fusion on the visual feature vector and the semantic feature vector to obtain a preliminary fusion vector comprises: performing normalization processing on the visual feature vector to obtain a visual normalized vector, and performing normalization processing on the semantic feature vector to obtain a semantic normalized vector; aligning the visual normalized vector and the semantic normalized vector in dimension to obtain a visual aligned vector and a semantic aligned vector having the same dimension; performing feature fusion on the visual aligned vector and the semantic aligned vector to obtain the preliminary fusion vector.

4. The method of claim 1, wherein, The decoding the multi-modal fusion feature through the action decoder to obtain the control instruction information comprises: performing key information extraction on the multi-modal fusion feature based on the action decoder to obtain preliminary key information; performing enhancement processing on the preliminary key information based on the action decoder to obtain enhanced key information; performing action prediction on the enhanced key information based on the action decoder to obtain an action prediction parameter; performing action planning on the action prediction parameter based on the action decoder to obtain a time sequence action sequence; performing format conversion on the time sequence action sequence based on the action decoder to obtain control instruction information.

5. The method of claim 1, wherein, The action decoder is trained through a reinforcement learning mechanism, and specifically comprises the following steps: acquiring sample visual information and sample semantic information; wherein the sample visual information and the sample semantic information are taken as a current state in the reinforcement learning mechanism; performing action sampling based on the sample visual information and the sample semantic information to obtain sample action information; wherein the sample action information is taken as a current action in the reinforcement learning mechanism; Reward feedback is performed based on the sample action information, to obtain sample reward information and an updated sample state; wherein the updated sample state is a next state of the current state; The action decoder is trained according to the sample reward information and the updated sample state.

6. The method according to any one of claims 1 to 5, characterized in that, The semantic encoding of the language instruction information by the language encoder includes: The semantic split of the language instruction information by the language encoder includes: The word segmentation of the semantic split unit by the language encoder includes: The semantic feature vector is obtained by performing semantic word segmentation sequence encoding on the semantic word segmentation sequence by the language encoder.

7. The method according to any one of claims 1 to 5, characterized in that, The visual encoder includes an enhancement layer, a feature extraction layer, a multi-scale feature fusion layer, and a feature full connection layer. The visual encoding of the original visual information by the visual encoder includes: The data enhancement processing of the original visual information by the enhancement layer includes: The feature extraction of the enhanced visual information by the feature extraction layer includes: The visual fusion feature is obtained by the multi-scale feature fusion layer based on the visual local feature and the visual global feature. The visual feature vector is obtained by mapping the visual fusion feature to a target dimension by the feature full connection layer.

8. A large model-based robot arm manipulation device, characterized by, The device includes: A multi-modal data acquisition module is configured to acquire multi-modal information in response to an operation task request, the multi-modal information including original visual information and language instruction information, and the language instruction information being natural language. A large model acquisition module is configured to acquire a preset visual language action large model, the visual language action large model including a visual encoder, a language encoder, and an action decoder. A visual encoding module is configured to perform visual encoding on the original visual information by the visual encoder to obtain a visual feature vector. A language encoding module is configured to perform semantic encoding on the language instruction information by the language encoder to obtain a semantic feature vector. A feature fusion module is configured to perform feature fusion on the visual feature vector and the semantic feature vector to obtain a multi-modal fusion feature. An action decoding module is configured to perform decoding on the multi-modal fusion feature by the action decoder to obtain control instruction information, wherein the control instruction information is used to control a robot arm.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the large model-based robot arm control method of any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the large model-based robot arm control method of any one of claims 1-7.

Images