Method and device for controlling robot arm based on large model, equipment and storage medium

Abstract

The embodiment of the application provides a kind of based on big model's mechanical arm control method and device, equipment and storage medium, belong to artificial intelligence technical field.The method comprises: in response to operation task request, obtain multi-modal information, the multi-modal information includes original visual information and language instruction information;Original visual information is visually encoded by visual encoder, and visual feature vector is obtained;The language instruction information is semantically encoded by language encoder, and semantic feature vector is obtained;The visual feature vector and the semantic feature vector are fused, and multi-modal fusion feature is obtained;The multi-modal fusion feature is decoded by action decoder, and control instruction information for controlling mechanical arm is obtained.The application can be applied in financial technology and health care and other business systems that need a lot of data, and the accuracy of mechanical arm control can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applied to the fields of fintech and healthcare, particularly to a method, device, equipment, and storage medium for manipulating a robotic arm based on a large model. Background Technology

[0002] Robotic arms can be used to grasp, move, and place objects. In fintech scenarios, for example, robotic arms can help banks count and sort banknotes or coins, thereby improving work efficiency and reducing human error. In healthcare scenarios, robotic arms can be used to grasp and manipulate surgical instruments to assist in medical procedures. In addition, robotic arms can also be used to sort, verify, and distribute medicines, thereby reducing the manpower required for manual medicine delivery and improving the efficiency and accuracy of medicine distribution.

[0003] In related technologies, the manipulation of robotic arms mainly relies on a task planning model for task planning and a strategy control model for control execution. These two models are trained independently, which not only consumes significant time and computational resources, but also, because the collaboration between the two models is not naturally close, information transmission often fails during the transition from task planning to actual control, leading to inaccurate robotic arm movements. Therefore, improving the accuracy of robotic arm manipulation has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to propose a robotic arm control method, device, equipment, and storage medium based on a large model, aiming to improve the accuracy of robotic arm control.

[0005] To achieve the above objectives, a first aspect of this application proposes a robotic arm manipulation method based on a large model, the method comprising: In response to an operation task request, multimodal information is acquired, including raw visual information and language instruction information, wherein the language instruction information is natural language; Obtain a preset visual language action model, which includes a visual encoder, a language encoder, and an action decoder. The original visual information is visually encoded by the visual encoder to obtain a visual feature vector. The language instruction information is semantically encoded using the language encoder to obtain a semantic feature vector; The visual feature vector and the semantic feature vector are fused to obtain multimodal fused features; The multimodal fusion features are decoded by the motion decoder to obtain control command information; wherein, the control command information is used to manipulate the robotic arm.

[0006] In some embodiments, the visual language action big model further includes a big language model, and the feature fusion of the visual feature vector and the semantic feature vector to obtain multimodal fusion features includes: The semantic feature vector is enhanced using the large language model to obtain an enhanced semantic representation; The visual feature vector and the semantic feature vector are initially fused to obtain a preliminary fused vector; The preliminary fusion vector and the enhanced semantic representation are concatenated to obtain the multimodal fusion feature.

[0007] In some embodiments, the visual feature vector and the semantic feature vector are initially fused to obtain a preliminary fused vector, including: The visual feature vector is normalized to obtain a visually normalized vector, and the semantic feature vector is normalized to obtain a semantically normalized vector. Align the visual normalized vector and the semantic normalized vector by dimension to obtain a visual alignment vector and a semantic alignment vector with the same dimension. The visual alignment vector and the semantic alignment vector are fused to obtain the preliminary fused vector.

[0008] In some embodiments, the step of decoding the multimodal fusion features through the action decoder to obtain the control command information includes: Based on the action decoder, key information is extracted from the multimodal fusion features to obtain preliminary key information; The preliminary key information is enhanced based on the action decoder to obtain enhanced key information; Based on the action decoder, action prediction is performed on the enhanced key information to obtain action prediction parameters; Based on the action decoder, action planning is performed on the action prediction parameters to obtain a temporal action sequence; The timing action sequence is format-converted based on the action decoder to obtain control instruction information.

[0009] In some embodiments, the action decoder is trained through a reinforcement learning mechanism, specifically including the following steps: Obtain sample visual information and sample semantic information; wherein, the sample visual information and the sample semantic information are used as the current state in the reinforcement learning mechanism; Action sampling is performed based on the sample visual information and the sample semantic information to obtain sample action information; wherein, the sample action information is used as the current action in the reinforcement learning mechanism; Reward feedback is performed based on the sample action information to obtain sample reward information and update the sample state; wherein, the updated sample state is the next state after the current state; The action decoder is trained based on the sample reward information and the updated sample state.

[0010] In some embodiments, the step of semantically encoding the language instruction information using the language encoder to obtain a semantic feature vector includes: The language instruction information is semantically segmented by the language encoder to obtain semantic segmentation units; The semantic segmentation unit is segmented into words by the language encoder to obtain a semantic word segmentation sequence; The semantic feature vector is obtained by performing hierarchical perceptual encoding on the semantic word segmentation sequence using the language encoder.

[0011] In some embodiments, the visual encoder includes an enhancement layer, a feature extraction layer, a multi-scale feature fusion layer, and a fully connected feature layer. The step of visually encoding the original visual information using the visual encoder to obtain a visual feature vector includes: The original visual information is augmented using the enhancement layer to obtain enhanced visual information. The enhanced visual information is extracted through the feature extraction layer to obtain local visual features and global visual features. The visual fusion feature is obtained by combining the visual local features and the visual global features through the multi-scale feature fusion layer. The visual fusion features are mapped to the target dimension through the fully connected feature layer to obtain the visual feature vector.

[0012] To achieve the above objectives, a second aspect of this application provides a robotic arm control device based on a large model, the device comprising: A multimodal data acquisition module is used to acquire multimodal information in response to an operation task request. The multimodal information includes raw visual information and language instruction information, wherein the language instruction information is natural language. The large model acquisition module is used to acquire a preset visual language action large model, which includes a visual encoder, a language encoder and an action decoder. A visual encoding module is used to visually encode the original visual information through the visual encoder to obtain a visual feature vector. The language encoding module is used to semantically encode the language instruction information through the language encoder to obtain a semantic feature vector; The feature fusion module is used to fuse the visual feature vector and the semantic feature vector to obtain multimodal fusion features; The motion decoding module is used to decode the multimodal fusion features through the motion decoder to obtain control command information; wherein the control command information is used to manipulate the robotic arm.

[0013] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0014] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0015] The robotic arm control method, apparatus, device, and storage medium based on a large model proposed in this application acquire multimodal information, including raw visual information and language command information, in response to an operation task request. The raw visual information is encoded by a visual encoder of the visual-language-action large model to obtain a visual feature vector, and the language command information is encoded by a language encoder of the same model to obtain a semantic feature vector. The visual and semantic feature vectors are then fused, resulting in a multimodal fusion feature that integrates visual and semantic information. This facilitates decoding of the multimodal fusion feature by the action decoder, thereby obtaining control command information for manipulating the robotic arm. In this application embodiment, the visual encoder, language encoder, and action decoder are integrated into the same large model, and the three are deeply fused and coordinated, thereby improving the accuracy of robotic arm control. Attached Figure Description

[0016] Figure 1 This is a flowchart of a robotic arm control method based on a large model provided in an embodiment of this application; Figure 2 yes Figure 1 The flowchart for step 103 in the document; Figure 3 yes Figure 1 The flowchart for step 105 in the document; Figure 4 yes Figure 3 The flowchart for step 302 in the document; Figure 5 yes Figure 1 The flowchart for step 106 in the document; Figure 6 This is a flowchart of a robotic arm control method based on a large model provided in another embodiment of this application; Figure 7 This is a schematic diagram of the structure of the robotic arm control device based on a large model provided in the embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0021] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0022] Large Language Model (LLM): LLM is a deep learning model trained on massive amounts of text data. It can generate natural language text or understand the meaning of language text. By training on huge datasets, LLM learns language patterns, enabling it to understand, generate, and reason about text, and achieves powerful general modeling and generalization capabilities. The core principle of LLM is to learn the patterns and structures of natural language through large-scale unsupervised training, simulating human language cognition and generation processes to a certain extent. LLM can perform not only simple language tasks such as spell checking and grammar correction, but also complex tasks such as text summarization, machine translation, sentiment analysis, dialogue generation, and content recommendation. Through pre-training on large datasets, LLM can be applied in fields such as finance, healthcare, and education.

[0023] Visual Encoder: A visual encoder can be used to transform images / videos (such as the pixel matrix of an image, the frame sequence of a video, etc.) into machine-understandable feature vectors, enabling machines to understand visual information such as color, texture, and object position in an image. Visual encoders can be used for multimodal general intelligence and self-supervised training models.

[0024] Language Encoder: A language encoder is an encoder used in deep learning to process text data and is a core component of Natural Language Processing (NLP). It converts text (such as sentences and paragraphs) into high-order feature representations by extracting contextual features layer by layer, ultimately transforming the input sequence into a hidden representation. Language encoders can be used for text classification, sentiment analysis, and more.

[0025] Vision-Language-Action (VLA): VLA is a large-scale model, an extension of multimodal large-scale models into embodied intelligence. VLA can form an end-to-end decision-making system by integrating visual perception, language understanding, and action control. VLA achieves closed-loop decision-making from environmental understanding to behavioral output by establishing a mapping between visual signals, language commands, and physical actions. Compared to traditional decision-making models, VLA models can overcome the limitations of single-modal processing.

[0026] Reinforcement Learning (RL) is a machine learning method that learns optimal decision-making strategies through trial and error and environmental feedback. Reinforcement learning involves an agent interacting with the environment, adjusting its strategy by trying different actions and receiving feedback of rewards or penalties without explicit guidance.

[0027] Embodied Intelligence: Embodied intelligence is a technology at the intersection of artificial intelligence and robotics, where intelligent agents achieve autonomous learning and evolution through dynamic interactions between their bodies and their environment. The core technology of embodied intelligence lies in the deep integration of perception, action, and cognition. Embodied intelligence can be applied to fields such as intelligent manufacturing, smart homes, smart healthcare, and intelligent services.

[0028] Embossed intelligent robots are robots equipped with embodied intelligence technology. They interact with their environment in real time through physical entities, achieving a closed loop of perception, cognition, decision-making, and action. Embossed intelligent robots mainly consist of three parts: a perception module, a decision-making system, and an actuator. The perception module acquires environmental information through multimodal sensors, including vision, touch, and hearing. The decision-making system integrates machine learning and cognitive reasoning algorithms to form dynamic adaptability. The actuator includes hardware systems such as bionic joints and flexible actuators.

[0029] Robotic arms can be used to grasp, move, and place objects. In fintech scenarios, for example, robotic arms can help banks count and sort banknotes or coins, thereby improving work efficiency and reducing human error. In healthcare scenarios, robotic arms can be used to grasp and manipulate surgical instruments to assist in medical procedures. In addition, robotic arms can also be used to sort, verify, and distribute medicines, thereby reducing the manpower required for manual medicine delivery and improving the efficiency and accuracy of medicine distribution.

[0030] In related technologies, the manipulation of robotic arms mainly relies on a task planning model for task planning and a strategy control model for control execution. These two models are trained independently, which not only consumes significant time and computational resources but also, due to the non-naturally close collaboration between them, often results in poor information transmission during the transition from task planning to actual control. This leads to inaccurate robotic arm movements and potential delays. For example, in complex industrial assembly scenarios, the task planning model formulates a plan for grasping and assembling parts. However, when this plan is transmitted to the control strategy model, differences in their understanding of the environment prevent the control strategy model from accurately executing the grasping action, affecting assembly efficiency and quality.

[0031] Furthermore, current technologies are mostly trained and designed for specific scenarios and items. When the environment changes slightly, such as the angle of the item's placement, the lighting conditions, or when faced with items of new shapes or materials, the robotic arm often struggles to grasp accurately. The model struggles to learn general grasping patterns and strategies, and cannot effectively transfer experience learned in one scenario to other scenarios. For example, in a logistics warehouse, if a previously trained model can only grasp regularly shaped packaging boxes, it will be unable to complete the grasping task when encountering irregularly shaped packages.

[0032] Furthermore, in current embodied intelligence technologies, the vision module is responsible for recognizing objects, and the language module is responsible for understanding task descriptions. However, there is a lack of deep integration between these two modules and the final motion execution module. Information acquired by the vision and language modules suffers significant loss when transmitted to the motion module, making it difficult for the robotic arm to accurately generate and execute actions based on visual and linguistic information. For example, when receiving the instruction to "grab the blue cylindrical part," the vision module may correctly identify the blue cylinder, but due to coordination issues, the robotic arm cannot grasp the part with the appropriate posture and force when the instruction is transmitted to the motion execution module.

[0033] Based on this, embodiments of this application provide a robotic arm control method, apparatus, device, and storage medium based on a large model, aiming to improve the accuracy of the robotic arm in grasping objects.

[0034] The robotic arm control method, apparatus, device, and storage medium based on a large model provided in this application are specifically described through the following embodiments. First, the robotic arm control method based on a large model in this application is described.

[0035] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0036] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0037] The large-model-based robotic arm control method provided in this application relates to the field of artificial intelligence technology and is specifically applied in fintech and healthcare scenarios. This large-model-based robotic arm control method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the large-model-based robotic arm control method, but is not limited to the above forms.

[0038] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0039] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0040] Figure 1 This is an optional flowchart of the robotic arm manipulation method based on a large model provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 101 to 106.

[0041] Step 101: In response to the operation task request, obtain multimodal information, which includes raw visual information and language instruction information, and the language instruction information is natural language; Step 102: Obtain a preset visual language action model, which includes a visual encoder, a language encoder, and an action decoder. Step 103: Visually encode the original visual information using a visual encoder to obtain a visual feature vector; Step 104: Semantically encode the language instruction information using a language encoder to obtain a semantic feature vector; Step 105: Perform feature fusion on the visual feature vector and the semantic feature vector to obtain multimodal fusion features; Step 106: Decode the multimodal fusion features using the motion decoder to obtain control command information; wherein, the control command information is used to manipulate the robotic arm.

[0042] Steps 101 to 106 as illustrated in this embodiment involve acquiring multimodal information, including raw visual information and language instruction information, in response to an operation task request. The raw visual information is encoded by the visual encoder of the visual-language-action large model to obtain a visual feature vector, and the language instruction information is encoded by the language encoder of the visual-language-action large model to obtain a semantic feature vector. The visual feature vector and the semantic feature vector are then fused to obtain a multimodal fused feature. Finally, the multimodal fused feature is decoded by the action decoder to obtain control instruction information for manipulating the robotic arm. In this embodiment, the visual encoder, language encoder, and action decoder are integrated into the same large model, and the three are deeply fused and coordinated, thereby improving the accuracy of robotic arm manipulation.

[0043] In step 101 of some embodiments, the operation task request is used to request the robotic arm to perform relevant operations on the manipulated object. The manipulated object can be an item. For example, in a financial technology scenario, the manipulated object can be a bill, seal, etc. The bill can be, for example, banknotes, checks, deposit slips, drafts, etc. The relevant operation can be, for example, grasping the bill and classifying it into a storage compartment, and it is required that the bill be wrinkle-free, undamaged, and accurately classified during grasping and classification. For example, in a health and medical scenario, the manipulated object can be medicine, medical device, etc. The medicine can be an ampoule or medicine box, and the medical device can be a scalpel, hemostat, etc. The relevant operation can be, for example, grasping the medicine or passing the medical device to the doctor, and it is required that the handover to the doctor be aseptic, the medicine be undamaged, and the handover be accurate (error ≤ 1cm).

[0044] In some embodiments, the multimodal information in step 101 includes at least visual information and textual information. The visual information is raw visual information; for example, in a fintech scenario, raw visual information could be images of bills, seals, or storage compartments; in a healthcare scenario, raw visual information could be images of medicines, medical devices, etc. The textual information is natural language-based text, specifically language instruction information. For example, in a fintech scenario, the language instruction could be, "Retrieve the bills of exchange from the smart vault, classify them, and place them in the corresponding storage compartments, ensuring the bills are wrinkle-free, undamaged, and accurately classified"; in a healthcare scenario, the language instruction could be, "Please pass the scalpel to Dr. Wang, ensuring aseptic operation, undamaged medicines, and a precise delivery error ≤1cm."

[0045] In step 102 of some embodiments, the visual language action big model is a VLA, which includes a visual encoder, a language encoder, and an action decoder. In one application scenario, the structure of the visual language action big model can be represented as VLA = f(visual encoder, language encoder, action decoder), where f represents the fusion and computation process within the VLA model.

[0046] In step 103 of some embodiments, the visual feature vector obtained by encoding the original visual information by the visual encoder can be represented as: V_{feat}=visual encoder(I_{input}), where the visual feature vector is V_{feat} and the original visual information is I_{input}.

[0047] In one application scenario, the visual encoder in step 103 can employ a deep residual network or ViT (VisionTransformer). The deep residual network can extract object appearance features, while ViT can capture spatial relationships. The deep residual network can be ResNet-50, a deep convolutional neural network based on the ResNet architecture. ResNet-50 introduces residual blocks and skip connections, solving the gradient vanishing and degradation problems in deep networks and supporting deeper network layers. ViT can be used to capture spatial relationships. ViT is a model that applies the Transformer architecture to computer vision tasks. The core principle of ViT is to segment an image into a sequence of patches and process the image data through a self-attention mechanism. ViT can segment an image into multiple 16×16 pixel blocks (patches), and each patch is converted into a vector through linear embedding, forming a sequence input Transformer encoder. Unlike traditional convolutional neural networks (CNNs), ViT directly treats the image as a sequential input, directly converting the 2D image into a 1D sequence and using a self-attention mechanism to process the pixel relationships in the image to capture the correlation between different locations. Furthermore, ViT completely abandons the CNN structure, using only Transformers to process image data, while CNNs rely on convolutional layers and pooling operations. This application's embodiments achieve stronger context modeling capabilities by employing ViT's fully connected layers.

[0048] The visual encoder includes an enhancement layer, a feature extraction layer, a multi-scale feature fusion layer, and a fully connected feature layer. (See also: [link to relevant documentation]) Figure 2 In some embodiments, step 103 may include, but is not limited to: Step 201: Perform data augmentation processing on the original visual information through the enhancement layer to obtain enhanced visual information; Step 202: Extract features from the enhanced visual information through the feature extraction layer to obtain local visual features and global visual features; Step 203: Visual fusion features are obtained by combining visual local features and visual global features through a multi-scale feature fusion layer; Step 204: The visual fusion features are mapped to the target dimension through a fully connected feature layer to obtain the visual feature vector.

[0049] In some embodiments, before performing step 201, the large-model-based robotic arm manipulation method further includes: preprocessing the original visual information, such as normalization, size adjustment, and channel alignment. Specifically, for the RGB image in the original visual information, normalization can be performed using ImageNet mean and standard deviation. For the depth map in the original visual information, pixels within a distance range of 0-2m can be normalized to [0,1], and pixels outside the range can be set to 0 to represent invalid regions. In one application scenario, the size can be adjusted to scale according to the model input resolution (e.g., scaling ResNet-50 to 224×224); channel alignment can be, for example, adjusting the number of channels in the depth map to 3 using a 1×1 convolution and stitching it with the RGB image to achieve a 6-channel input.

[0050] In step 201 of some embodiments, data augmentation processing of the original visual information may include at least one of the following: color enhancement, shape enhancement, material enhancement, layout enhancement, etc. Color enhancement may, for example, involve generating variants for color information (e.g., the anti-counterfeiting color of financial instruments, the label color of medical drugs, etc.) by adjusting brightness (±30%), contrast (±20%), saturation (±20%), etc., thereby avoiding the influence of changes in lighting. Shape enhancement may, for example, involve randomly rotating (e.g., -15° to 15°) and scaling (e.g., 0.8 to 1.2) shape information (e.g., the edge of a financial instrument, the outline of a surgical instrument, etc.) through random rotation (e.g., -15° to 15°) and scaling (e.g., 0.8 to 1.2). Variations can be generated by adding Gaussian noise to simulate rough surfaces, blurring (e.g., simulating smooth surfaces), and generating variations by adding Gaussian blur to metal instruments (e.g., needle holders) to simulate the glare of surgical lights. Layout enhancements can be generated by randomly cropping and randomly arranging objects (e.g., different placements of tickets on a table) to learn global layout rules.

[0051] In step 202 of some embodiments, local and global feature extraction is performed on the enhanced visual information through a feature extraction layer to obtain visual local features and visual global features. Visual local features are features extracted from local regions of the image and can reflect the detailed attributes of an object. Examples include local texture and detailed structure, attributes of key parts, and features at minute scales. Examples of visual local features include microtext on a bill, the raised texture of the metal thread on the edge of a deposit slip, the anti-counterfeiting serrations on the edge of a "financial seal," a tear in the upper right corner of a check, the sharpness of a scalpel blade, and the bite of a hemostat. Visual global features are features extracted from the overall perspective of an image and can reflect the macroscopic attributes of a scene, such as the overall distribution and relative positions of objects, and the environmental attributes of the scene. For example, the overall distribution and relative positions of objects can be the arrangement of bills on the vault table, three deposit slips stacked on the left, two checks laid flat on the right, a red stamp 10cm above the checks, and a magnetic stripe card reader 5cm to the right. Environmental attributes of the scene can be the lighting intensity of the operating room and the layout of the instrument table. Visual global features can also be a stack of five deposit slips with a face value of 10,000 yuan, most of the bills are flat, only one in the lower right corner has obvious wrinkles, the surgical instrument table is located in a sterile area and the doctor's gloves do not touch the non-sterile area, etc. Visual global features can also be scalpels, hemostatic forceps, and suture needles arranged in the order of use on the left side of the instrument tray, with two scalpels, three hemostatic forceps, and a box of suture needles in the surgical tray, the ampoules are unopened, the medicine boxes are undamaged, and all metal instruments have no obvious rust, etc.

[0052] In step 203 of some embodiments, visual local features and visual global features are pyramid pooled through a multi-scale feature fusion layer to convert the multi-scale feature maps into the same size. The number of channels is compressed by convolution, and then the multi-scale features are stitched together to obtain visual fusion features. For example, the number of channels is compressed to 64, 128, or 256 dimensions by a 1×1 convolution layer. The dimension of the visual fusion features obtained by stitching together the multi-scale features is: 64 + 128 + 256 = 448.

[0053] In step 204 of some embodiments, the visual fusion features are mapped to the target dimension through a fully connected feature layer. The target dimension may be, for example, 512 dimensions. The fully connected feature layer may be two layers. In one application scenario, 448-dimensional visual fusion features are input into the first fully connected feature layer for processing. The first fully connected feature layer outputs a 1024-dimensional feature vector, which is then fed into the second fully connected feature layer for processing. The second fully connected feature layer outputs a 512-dimensional visual feature vector. Furthermore, to ensure that the vectors are processed at a uniform scale, step 204 also includes normalizing the visual feature vector, for example, by performing L2 normalization.

[0054] The visual feature vectors obtained through steps 201 to 204 above can contain key information such as the target object's position, shape, color, material, and environmental layout.

[0055] Taking a fintech scenario as an example, a visual encoder encodes the raw visual information of bills (checks, deposit slips), seals, and cash extracted from vault monitoring images. The resulting visual feature vector can include: the position / shape of the bills (e.g., distinguishing between checks and deposit slips), color / material (e.g., the color and texture of anti-counterfeiting watermarks), and environmental layout (e.g., the stacking state of bills and the relative position of seals), which facilitates subsequent classification by the robotic arm according to instructions.

[0056] Taking a healthcare scenario as an example, a visual encoder encodes the raw visual information of surgical instruments (such as scalpels and hemostatic forceps) and medications (such as ampoules and medicine boxes) extracted from endoscopes or monitoring images in the operating room. The resulting visual feature vector can include: the shape / material of the instrument (e.g., distinguishing between metal scalpels and plastic forceps), the color / location of the medication, and the environmental layout (e.g., the placement of the instrument on the sterile table). The color of the medication can be used to distinguish the label colors of different medications, such as distinguishing between penicillin and cephalosporins, so that the robotic arm can assist the doctor in passing surgical instruments and medications according to instructions.

[0057] In step 104 of some embodiments, the semantic feature vector obtained by encoding the language instruction information through the language encoder can be represented as: L_{sem} = language encoder(S_{input}), where the semantic feature vector is L_{sem}, the language instruction information is S_{input}, and S_{input} is a task description based on natural language, such as "take the water glass on the table to the kitchen".

[0058] In one application scenario, the language encoder in step 104 can be either BERT-base or RoBERTa (Robustly Optimized BERT Pretraining Approach). BERT-base is a natural language processing (NLP) model based on the Transformer architecture and is the original version of BERT, which can better understand the semantics of instructions. RoBERTa is an improved version of BERT and is a robustly optimized BERT pretraining method that can handle complex sentences.

[0059] In some embodiments, step 104 may include, but is not limited to: The language instruction information is semantically decomposed by a language encoder to obtain semantic decomposition units; The semantic segmentation units are segmented into words by a language encoder to obtain a semantic word segmentation sequence; The semantic word segmentation sequence is subjected to hierarchical perceptual encoding by a language encoder to obtain semantic feature vectors.

[0060] In this process, the core elements are identified by semantically decomposing the language instruction information. These core elements include, for example, action elements, object elements, and constraint elements. Action elements may include, for example, grasping or classifying; object elements may include, for example, checks, deposit slips, storage compartments, scalpels, hemostatic forceps, medicines, and medicine boxes; and constraint elements may include, for example, no wrinkles, no damage, accurate classification, and ensuring sterility when passing the scalpel to Dr. Wang.

[0061] By performing word segmentation on the semantic splitting units to obtain a semantic word segmentation sequence. Taking BERT-base as an example, WordPiece can be used for word segmentation; taking RoBERTa as an example, ByteLevelBPE can be used for word segmentation, which can support rare words, such as the complete encoding of the medical term "ampoule bottle", to avoid semantic loss caused by excessive sub-word splitting. Taking the fintech scenario as an example, the semantic word segmentation sequence obtained after semantic splitting and word segmentation of the language instruction information "Grab the bill in the intelligent vault, classify it and put it into the corresponding storage grid, ensuring that the bill is wrinkle-free, undamaged, and accurately classified" includes: "in", "intelligent", "vault", "in", "grab", "bill", ",", "and", "classify", "put into", "corresponding", "storage grid", ",", "ensure", "bill", "wrinkle-free", "、", "undamaged", "、", "accurately classified"; taking the health care scenario as an example, the semantic word segmentation sequence obtained after semantic splitting and word segmentation of the language instruction information "Please pass the scalpel to Dr. Wang and ensure aseptic operation, the drugs are undamaged, and the error of accurate transmission ≤ 1 cm" includes: "please", "pass", "scalpel", "to", "Dr. Wang", ",", "and", "ensure", "aseptic operation", "、", "drugs undamaged", "、", "accurate transmission", "of", "error ≤ 1 cm".

[0062] Through BERT-base or RoBERTa, perform upper and lower position perception encoding on the semantic word segmentation sequence, and the obtained semantic feature vector can achieve semantic enhancement, which is convenient for the subsequent action encoder to perform semantic understanding.

[0063] In some embodiments, the visual language action large model further includes a large language model, which is LLM, and the structure of the visual language action large model can be expressed as VLA = f(visual encoder, language encoder, LLM, action decoder).

[0064] In an application scenario, the action encoder in step 105 can adopt Transformer-based Seq2Seq or MLP+MPC joint decoding. Through Transformer-based Seq2Seq, a joint trajectory sequence can be output, and through MLP+MPC joint decoding, dynamic constraints can be combined, thus.

[0065] Please refer to Figure 3 , in step 105 of some embodiments, it may include but is not limited to including: Step 301, perform enhancement processing on the semantic feature vector through the large language model to obtain an enhanced semantic representation. Step 302: Perform preliminary fusion of visual feature vectors and semantic feature vectors to obtain a preliminary fused vector; Step 303: The preliminary fusion vector and the enhanced semantic representation are concatenated to obtain the multimodal fusion feature.

[0066] In step 301 of some embodiments, the LLM, based on its pre-trained knowledge system, further understands and reasones the semantic feature vector obtained by the language encoder to obtain an enhanced semantic representation. In one application scenario, the semantic feature vector can be represented as L_{enhanced}.

[0067] Please see Figure 4 In some embodiments, step 302 may include, but is not limited to: Step 401: Normalize the visual feature vector to obtain a visually normalized vector, and normalize the semantic feature vector to obtain a semantically normalized vector. Step 402: Align the visual normalized vector and the semantic normalized vector in terms of dimensions to obtain a visual alignment vector and a semantic alignment vector with the same dimensions. Step 403: Perform feature fusion on the visual alignment vector and the semantic alignment vector to obtain a preliminary fusion vector.

[0068] In step 401 of some embodiments, the illumination differences in the image can be eliminated by normalizing the visual feature vector, and the influence of text length can be eliminated by normalizing the semantic feature vector.

[0069] In step 402 of some embodiments, the visual normalized vector and the semantic normalized vector are dimensionally aligned to ensure that they have the same dimension. Specifically, a fully connected layer can be used to project to the same dimension, for example, both can be projected to 512 dimensions.

[0070] After normalization and dimension alignment, step 403 is performed to fuse features, resulting in a preliminary fused vector. Specifically, a multilayer perceptron (MLP) can be used to fuse the visual alignment vector and the semantic alignment vector. An MLP is a feedforward neural network that typically consists of an input layer, at least one hidden layer, and an output layer. Neurons in each layer are fully connected to all neurons in adjacent layers, transmitting signals through weights and bias parameters. The hidden layers of an MLP introduce nonlinear transformations through activation functions (such as sigmoid or ReLU), thus overcoming the limitation of single-layer perceptrons that can only handle linear classification. The hierarchical structure of an MLP can be used to handle complex nonlinear problems.

[0071] Through steps 401 to 403 of the embodiments of this application, the visual feature vectors are normalized and dimension aligned to perform preliminary feature fusion through a fusion layer (e.g., MLP), thereby facilitating further fusion in step 303.

[0072] In step 302 of this application embodiment, the execution order of steps 401 and 402 is not limited. Step 401 can be executed first for normalization processing and then step 402 can be executed for dimension alignment, or dimension alignment can be performed first and then normalization processing can be performed. This application embodiment will be described using normalization processing first as an example.

[0073] In step 303 of some embodiments, the preliminary fusion vector and the enhanced semantic representation are concatenated to achieve deep fusion of visual, semantic, and enhanced semantic representations, thereby obtaining multimodal fusion features. Specifically, the preliminary fusion vector and the enhanced semantic representation can be fused and concatenated based on a cross-attention mechanism. In one application scenario, the preliminary fusion vector can be represented as F_{fusion}, and the multimodal fusion feature can be represented as I_{decoder}, which serves as the input to the action decoder in step 106. I_{decoder} = Concat(F_{fusion}, L_{enhanced}), where Concat represents the concatenation operation.

[0074] Through steps 301 to 303 of the embodiments of this application, the semantic feature vector is enhanced by the large language model, and the visual feature vector and the semantic feature vector are initially fused. Then, the initially fused vector and the enhanced semantic representation are deeply fused to obtain a multimodal fused feature. The multimodal fused feature contains the interaction information between vision and semantics, so that the action decoder can generate control command information for controlling the robot arm's movement in step 106.

[0075] This application's embodiments optimize through multimodal information fusion, enabling the robotic arm to more comprehensively and accurately understand task and environmental information, thereby generating more precise and appropriate grasping actions. For example, in a complex home setting, when receiving the task instruction "take the water glass from the table to the kitchen," the visual encoder in the VLA model can combine visual recognition of the table, water glass, and kitchen locations with the language encoder and LLM's understanding of the task instruction's meaning, allowing the motion encoder to plan and execute accurate grasping and carrying actions.

[0076] Please see Figure 5 In some embodiments, step 106 may include, but is not limited to: Step 501: Extract key information from the multimodal fusion features based on the action decoder to obtain preliminary key information; Step 502: Enhance the preliminary key information based on the action decoder to obtain enhanced key information; Step 503: Based on the action decoder, perform action prediction on the enhanced key information to obtain action prediction parameters; Step 504: Based on the action decoder, perform action planning on the action prediction parameters to obtain a temporal action sequence; Step 505: Based on the action decoder, the temporal action sequence is format-converted to obtain control instruction information.

[0077] In step 501 of some embodiments, key information can be extracted from the multimodal fusion features based on the Transformer layer and the fully connected layer. The preliminary key information obtained may include, but is not limited to, target object information, semantic constraint information, and environmental state information. The target object information may include, but is not limited to, 3D coordinates, shape type (such as rectangle / cylindrical shape), and material hardness. The semantic constraint information may include, but is not limited to, action type (such as gentle grasping, precise clamping), constraint threshold (such as clamping force ≤3N, position error ≤2mm, sterile area boundary, etc.). The environmental state information may include, but is not limited to, obstacle coordinates (such as a card reader in a vault, an instrument tray in an operating room), and the current joint angle of the robotic arm.

[0078] In step 502 of some embodiments, enhancing the preliminary key information may include, but is not limited to, explicitly enhancing the extracted information. For example, by calculating the relative distance between the target object and the current position of the robotic arm as the basis for motion planning, prohibitions in semantic constraints (such as prohibiting touching the blade, prohibiting wrinkles, etc.) are transformed into mask features (such as blade area coordinate mask), thereby preventing motion parameters from falling into prohibited areas.

[0079] In step 503 of some embodiments, hierarchical prediction of motion parameters is performed based on enhanced key information to obtain motion prediction parameters. For example, the core parameters of the robotic arm's grasping can be predicted hierarchically using a multi-branch neural network to ensure accurate matching between each motion prediction parameter and object attributes and semantic constraints. In practical application scenarios, the grasping action involves five core motion prediction parameters: position, posture, gripping force, movement speed, and movement duration.

[0080] In step 504 of some embodiments, motion planning is performed based on motion prediction parameters, and the resulting temporal motion sequence is used to represent the motion at each stage. The temporal motion sequence may include a proximity motion sequence, a contact motion sequence, a clamping motion sequence, and a lifting motion sequence. The proximity motion sequence represents the sequence of moving to a position close to an object during the proximity phase, such as moving to a safe distance above the object. The contact motion sequence represents the slow descent to the object surface during the contact phase. The clamping motion sequence represents the closing end to the target force during the clamping phase, such as clamping a bill of exchange in a fintech scenario, where the force can be set to 2.5N and last for 0 seconds. A 0.5-second interval ensures stability and prevents wrinkles. For example, in a healthcare setting, the clamping force for passing a scalpel can be set to 4.5N for 0.8 seconds. Since higher stability is required in medical settings, the duration needs to be set longer. The lifting action sequence is used to indicate lifting an object to a safe height during the lifting phase. For example, in a fintech setting, the speed of lifting a bill is 0.2m / s, and the position is (0.45m, 0.34m, 0.9m). Similarly, in a healthcare setting, the speed of passing a scalpel is 0.15m / s, and the position is (0.35m, 0.63m, 1.1m).

[0081] In this embodiment, a bidirectional LSTM network (512 hidden layer dimensions) can be used to plan the motion prediction parameters to ensure that the end position of the previous stage is the start position of the next stage, and the deceleration process lasts for 0.2s. In addition, each motion prediction parameter is associated through an attention mask. For example, only the force parameter is activated in the clamping stage, while it is frozen in other stages (approaching stage, contact stage, lifting stage).

[0082] In step 505 of some embodiments, the various time-series action sequences are converted into format instructions for controlling the robotic arm's actions to obtain control instruction information. In some embodiments, the control instruction information may include, but is not limited to, at least one of the following: target state (joint angle / Cartesian coordinates), motion constraints (speed limit, obstacle avoidance area), force control parameters (grasping force threshold), timing information (action stage identifier), and environmental feedback (such as object pose change compensation amount).

[0083] Please see Figure 6 In some embodiments, the robotic arm manipulation method based on a large model may include, but is not limited to: Step 601: Obtain sample visual information and sample semantic information; wherein, the sample visual information and sample semantic information are used as the current state in the reinforcement learning mechanism; Step 602: Perform action sampling based on sample visual information and sample semantic information to obtain sample action information; wherein, the sample action information is used as the current action in the reinforcement learning mechanism; Step 603: Provide reward feedback based on sample action information to obtain sample reward information and update sample state; wherein, the updated sample state is the next state after the current state. Step 604: Train the action decoder based on the sample reward information and the updated sample state.

[0084] In step 601 of some embodiments, the sample visual information and sample semantic information are used as the current state S_t in the reinforcement learning mechanism. Taking the fintech scenario as an example, the sample visual information can be an image of a bill, an image of a seal, or an image of a storage compartment. Taking the health scenario as an example, the sample visual information can be an image of medicine, an image of a medical device, etc. The sample semantic information is text based on natural language. This sample semantic information is language instruction information. Taking the fintech scenario as an example, the sample semantic information can be "Retrieve the bill in the smart vault, classify and place it into the corresponding storage compartment, and ensure that the bill is wrinkle-free, undamaged, and accurately classified." Taking the health and medical scenario as an example, the sample semantic information can be "Please pass the scalpel to Dr. Wang, and ensure aseptic operation, undamaged medicine, and a precise delivery error ≤1cm."

[0085] In step 602 of some embodiments, action sampling is performed according to the policy network. Specifically, sample visual information and sample semantic information are used as inputs to the policy network. The policy network obtains an action A_t, i.e., sample action information A_t, through random sampling. Taking a fintech scenario as an example, action A_t can be the "grab" action in "grab a bill"; taking a healthcare scenario as an example, action A_t can be the "pass to" action in "please pass the scalpel to Dr. Wang".

[0086] In one application scenario, the policy network is an Actor-Critic network, which includes an Actor network and a Critic network. The Actor network is used to output the action distribution based on the sample visual information and sample semantic information, and the Critic network is used to predict the state based on the sample visual information, sample semantic information and the action distribution output by the Actor network, and obtain the state value.

[0087] In step 603 of some embodiments, a new state S_{t+1} is obtained by executing action A_t, that is, the sample state S_{t+1} is updated, and the reward is calculated to obtain the sample reward information R_{t+1}, thereby training the action decoder in step 604.

[0088] This application's embodiments learn generalizable action policies by setting diverse reward mechanisms. During training, an action A_t is selected based on the current state S_t (including visual and semantic information). After executing the action, a reward R_{t+1} from the environment and a new state S_{t+1} are obtained, and the accumulated reward over a long period is maximized to train the action decoder. For example, in a grasping task application scenario, the policy network outputs joint torques. If the end effector of the robotic arm successfully touches the target object without collision, a reward is added to trigger the task completion reward, thereby accelerating policy convergence.

[0089] This application's embodiments can be trained in diverse simulated environments. These environments may include, but are not limited to, different object shapes, sizes, materials, placement positions and angles, and varying lighting and occlusion conditions. Through training in these diverse environments, the system continuously learns and gradually masters generalized grasping patterns and strategies for coping with different environmental changes. When encountering new scenes or objects, it can make action decisions based on the learned generalized strategies, thereby achieving precise grasping. For example, during training, the system learns how to adjust its grasping posture to ensure stable grasping of objects with different center-of-gravity distributions. When encountering objects with new center-of-gravity distributions, it can adjust its posture based on the learned strategies.

[0090] This application's embodiments utilize a single large VLA model for task planning and control, eliminating delays and information loss in inter-model collaboration. This allows the robotic arm to respond to task commands faster and increases the amount of tasks completed per unit time. In industrial production line applications, this application's embodiments can significantly improve production efficiency and reduce production costs. For example, a gripping and assembly task that originally required 10 seconds can be shortened to 5 seconds using this application's embodiments.

[0091] The VLA large model trained by reinforcement learning in this embodiment has strong generalization ability, enabling the robotic arm to accurately complete grasping tasks in various complex and changing environments. This greatly expands the application scope of embodied intelligence in real-world scenarios, such as handling goods of various shapes and packages in logistics sorting, and adapting to different home layouts and item placement in home services. Taking a logistics warehouse as an example, conventional robotic arms can only grasp a limited number of types of goods, while the robotic arm in this embodiment can grasp common types of goods, greatly improving the level of logistics automation.

[0092] The multimodal information fusion in this embodiment enables the robotic arm to understand the task and environment more accurately, and to perform more precise grasping actions, thereby reducing errors during the grasping process and improving the success rate of task execution. In medical surgical assistance scenarios, the robotic arm can grasp and manipulate surgical instruments more accurately, reducing surgical risks; in electronic manufacturing, it can improve the grasping and assembly precision of precision parts, thereby improving product quality.

[0093] Please see Figure 7 This application also provides a robotic arm control device based on a large model, the device comprising: The multimodal data acquisition module is used to acquire multimodal information in response to operation task requests. The multimodal information includes raw visual information and language instruction information, and the language instruction information is natural language. The large model acquisition module is used to acquire a preset visual language action large model, which includes a visual encoder, a language encoder, and an action decoder. The visual encoding module is used to visually encode the raw visual information through the visual encoder to obtain visual feature vectors; The language encoding module is used to semantically encode language instruction information through a language encoder to obtain semantic feature vectors. The feature fusion module is used to fuse visual feature vectors and semantic feature vectors to obtain multimodal fused features; The motion decoding module is used to decode the multimodal fusion features through the motion decoder to obtain control command information; the control command information is used to manipulate the robotic arm.

[0094] In some embodiments, the visual encoding module can specifically be used to implement: The original visual information is augmented by an enhancement layer to obtain enhanced visual information; The enhanced visual information is extracted through a feature extraction layer to obtain local visual features and global visual features; Visual fusion features are obtained by combining visual local features and visual global features through a multi-scale feature fusion layer. Visual feature vectors are obtained by mapping visual fusion features to the target dimension through a fully connected feature layer.

[0095] Specifically, the visual encoding module can be used to implement steps 201 to 204 above.

[0096] In some embodiments, the feature fusion module can specifically be used to implement: By enhancing the semantic feature vectors using a large language model, an enhanced semantic representation is obtained. The visual feature vector and the semantic feature vector are fused to obtain a preliminary fused vector; The initial fusion vector and the enhanced semantic representation are concatenated to obtain the multimodal fusion feature.

[0097] Specifically, the feature fusion module can be used to implement steps 301 to 303 above.

[0098] In some embodiments, the feature fusion module can specifically be used to implement: The visual feature vector is normalized to obtain the visual normalized vector, and the semantic feature vector is normalized to obtain the semantic normalized vector. Align the visual normalized vector and the semantic normalized vector in terms of dimensions to obtain a visually aligned vector and a semantically aligned vector with the same dimensions. The visual alignment vector and the semantic alignment vector are fused to obtain a preliminary fused vector.

[0099] Specifically, the feature fusion module can be used to implement steps 401 to 403 above.

[0100] In some embodiments, the action decoding module can specifically be used to implement: Based on the action decoder, key information is extracted from the multimodal fusion features to obtain preliminary key information; The initial key information is enhanced based on the action decoder to obtain enhanced key information. Action prediction parameters are obtained by performing action prediction on enhanced key information based on the action decoder. Action planning is performed on the action prediction parameters based on the action decoder to obtain a temporal action sequence; The timing action sequence is converted into a format based on the action decoder to obtain control command information.

[0101] Specifically, the action decoding module can be used to implement steps 501 to 505 above.

[0102] In some embodiments, the large-model-based robotic arm control device can also be used to achieve: Obtain visual and semantic information of the samples; wherein, the visual and semantic information of the samples are used as the current state in the reinforcement learning mechanism; Action sampling is performed based on sample visual and semantic information to obtain sample action information; the sample action information is then used as the current action in the reinforcement learning mechanism. Reward feedback is performed based on sample action information to obtain sample reward information and update sample state; where the updated sample state is the next state after the current state. The action decoder is trained based on sample reward information and updated sample states.

[0103] Specifically, the robotic arm control device based on the large model can be used to implement steps 601 to 604 above.

[0104] The specific implementation of the large-model-based robotic arm control device is basically the same as the specific implementation of the large-model-based robotic arm control method described above, and will not be repeated here.

[0105] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described robotic arm control method based on a large model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0106] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and called and executed by the processor 801 to implement the robotic arm control method based on a large model according to the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described robotic arm manipulation method based on a large model.

[0108] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0109] The robotic arm manipulation method, apparatus, device, and storage medium based on a large model provided in this application embodiment encodes raw visual information to obtain visual feature vectors through a visual encoder of a visual-language-action large model, and encodes language command information to obtain semantic feature vectors through a language encoder of the same large model. The visual and semantic feature vectors are then fused to obtain multimodal fusion features, which are then decoded by an action decoder to obtain control command information for manipulating the robotic arm. This application embodiment uses the same VLA large model for visual encoder, language encoder, and action decoder, and these components are deeply fused and coordinated to improve the accuracy of robotic arm manipulation. The VLA large model trained based on reinforcement learning in this application embodiment has strong generalization capabilities, enabling the robotic arm to accurately complete grasping tasks in various complex and changing environments. This greatly expands the application scope of embodied intelligence in real-world scenarios. Furthermore, multimodal information fusion makes the robotic arm's understanding of the task and environment more accurate, and the grasping actions more precise, thereby reducing errors during the grasping process and improving the success rate of task execution.

[0110] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0111] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0114] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0115] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A robotic arm manipulation method based on a large model, characterized in that, The method includes: In response to an operation task request, multimodal information is acquired, including raw visual information and language instruction information, wherein the language instruction information is natural language; Obtain a preset visual language action model, which includes a visual encoder, a language encoder, and an action decoder. The original visual information is visually encoded by the visual encoder to obtain a visual feature vector. The language instruction information is semantically encoded using the language encoder to obtain a semantic feature vector; The visual feature vector and the semantic feature vector are fused to obtain multimodal fused features; The multimodal fusion features are decoded by the motion decoder to obtain control command information; wherein, the control command information is used to manipulate the robotic arm. The visual language action model further includes a large language model. The feature fusion of the visual feature vector and the semantic feature vector to obtain multimodal fusion features includes: The semantic feature vector is enhanced using the large language model to obtain an enhanced semantic representation; The visual feature vector and the semantic feature vector are initially fused to obtain a preliminary fused vector; The preliminary fusion vector and the enhanced semantic representation are concatenated to obtain the multimodal fusion feature; The action decoder is trained through a reinforcement learning mechanism, specifically including the following steps: Reward feedback is performed based on sample action information to obtain sample reward information and update sample state; wherein, the sample action information is obtained based on action sampling and is used as the current action in the reinforcement learning mechanism; The action decoder is trained based on the sample reward information and the updated sample state; The visual encoder includes an enhancement layer, a feature extraction layer, a multi-scale feature fusion layer, and a fully connected feature layer. The process of visually encoding the original visual information using the visual encoder to obtain a visual feature vector includes: The original visual information is augmented using the enhancement layer to obtain enhanced visual information. The enhanced visual information is extracted through the feature extraction layer to obtain local visual features and global visual features. The visual fusion feature is obtained by combining the visual local features and the visual global features through the multi-scale feature fusion layer. The visual fusion features are mapped to the target dimension through the fully connected feature layer to obtain the visual feature vector.

2. The method according to claim 1, characterized in that, The visual feature vector and the semantic feature vector are initially fused to obtain an initial fused vector, including: The visual feature vector is normalized to obtain a visually normalized vector, and the semantic feature vector is normalized to obtain a semantically normalized vector. Align the visual normalized vector and the semantic normalized vector by dimension to obtain a visual alignment vector and a semantic alignment vector with the same dimension. The visual alignment vector and the semantic alignment vector are fused to obtain the preliminary fused vector.

3. The method according to claim 1, characterized in that, The step of decoding the multimodal fusion features using the action decoder to obtain the control command information includes: Based on the action decoder, key information is extracted from the multimodal fusion features to obtain preliminary key information; The preliminary key information is enhanced based on the action decoder to obtain enhanced key information; Based on the action decoder, action prediction is performed on the enhanced key information to obtain action prediction parameters; Based on the action decoder, action planning is performed on the action prediction parameters to obtain a temporal action sequence; The timing action sequence is format-converted based on the action decoder to obtain control instruction information.

4. The method according to claim 1, characterized in that, The action decoder is trained through a reinforcement learning mechanism and also includes the following steps: Obtain sample visual information and sample semantic information; wherein, the sample visual information and the sample semantic information are used as the current state in the reinforcement learning mechanism, and the updated sample state is the next state of the current state; Action sampling is performed based on the visual information and semantic information of the sample to obtain the action information of the sample.

5. The method according to any one of claims 1 to 4, characterized in that, The step of semantically encoding the language instruction information using the language encoder to obtain a semantic feature vector includes: The language instruction information is semantically segmented by the language encoder to obtain semantic segmentation units; The semantic segmentation unit is segmented into words by the language encoder to obtain a semantic word segmentation sequence; The semantic feature vector is obtained by performing hierarchical perceptual encoding on the semantic word segmentation sequence using the language encoder.

6. A robotic arm control device based on a large model, characterized in that, The device includes: A multimodal data acquisition module is used to acquire multimodal information in response to an operation task request. The multimodal information includes raw visual information and language instruction information, wherein the language instruction information is natural language. The large model acquisition module is used to acquire a preset visual language action large model, which includes a visual encoder, a language encoder and an action decoder. A visual encoding module is used to visually encode the original visual information through the visual encoder to obtain a visual feature vector. The language encoding module is used to semantically encode the language instruction information through the language encoder to obtain a semantic feature vector; The feature fusion module is used to fuse the visual feature vector and the semantic feature vector to obtain multimodal fusion features; The motion decoding module is used to decode the multimodal fusion features through the motion decoder to obtain control command information; wherein, the control command information is used to manipulate the robotic arm; The visual language action model further includes a large language model. The feature fusion of the visual feature vector and the semantic feature vector to obtain multimodal fusion features includes: The semantic feature vector is enhanced using the large language model to obtain an enhanced semantic representation; The visual feature vector and the semantic feature vector are initially fused to obtain a preliminary fused vector; The preliminary fusion vector and the enhanced semantic representation are concatenated to obtain the multimodal fusion feature; The action decoder is trained through a reinforcement learning mechanism, specifically including the following steps: Reward feedback is performed based on sample action information to obtain sample reward information and update sample state; wherein, the sample action information is obtained based on action sampling and is used as the current action in the reinforcement learning mechanism; The action decoder is trained based on the sample reward information and the updated sample state; The visual encoder includes an enhancement layer, a feature extraction layer, a multi-scale feature fusion layer, and a fully connected feature layer. The process of visually encoding the original visual information using the visual encoder to obtain a visual feature vector includes: The original visual information is augmented using the enhancement layer to obtain enhanced visual information. The enhanced visual information is extracted through the feature extraction layer to obtain local visual features and global visual features. The visual fusion feature is obtained by combining the visual local features and the visual global features through the multi-scale feature fusion layer. The visual fusion features are mapped to the target dimension through the fully connected feature layer to obtain the visual feature vector.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the robotic arm manipulation method based on a large model as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the robotic arm control method based on a large model as described in any one of claims 1 to 5.

Images