Industrial assembly part cross-modal positioning and grasping method and system based on visual language model and robot

Abstract

The application discloses an industrial assembly part cross-modal positioning and grabbing method and system based on a visual language model and a robot, and the method comprises the following steps: collecting an industrial assembly scene environment image, receiving a user voice instruction and converting the user voice instruction into a text instruction containing industrial core terms; an insert type industrial term feature enhancement layer is additionally arranged at the output end of the text encoder of the cross-modal visual language model, the semantic features of the industrial core terms are guided and strengthened through a preset industrial term dictionary, the industrial scene adaptation of the model is completed by adopting a phased training combined with a light weight parameter fine tuning strategy, the text instruction and the environment image are input into the adapted model to obtain candidate target assembly part bounding box information, the mechanical arm is controlled to move to the candidate area and collect a target local image again, the local image is input into a target detection model to complete accurate identification, and the mechanical arm is controlled to complete grabbing. The application improves the text-image cross-modal matching precision, and realizes voice-driven accurate identification and grabbing of industrial assembly parts.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and industrial automation, specifically to a method, system, and robot for cross-modal positioning and grasping of industrial components based on a visual language model. Background Technology

[0002] In the field of industrial assembly automation, the identification and grasping of assembly parts are the core execution links. Traditional industrial grasping solutions mostly rely on fixed-category target detection algorithms, which require pre-training of dedicated models for specific tasks. Moreover, the human-computer interaction is rigid, relying mostly on manual operation or fixed command triggering, making it difficult to adapt to the flexible industrial assembly requirements.

[0003] In recent years, the development of visual language models has provided a new technical path for cross-modal target localization, enabling the localization of open sets of targets by combining natural language instructions and adapting to flexible interaction needs. However, most general-purpose visual language models are pre-trained on large-scale data from general scenarios, and there are obvious adaptation problems when transferring them to industrial assembly scenarios: on the one hand, general-purpose models have a weak ability to perceive professional terms in the industrial field and are easily interfered with by non-core descriptive words in the text, resulting in insufficient cross-modal matching accuracy between text and images; on the other hand, industrial scenarios are difficult to provide large-scale labeled data. If the general-purpose model is fully fine-tuned, it is easy to destroy the general features of the pre-training, and overfitting is prone to occur with small samples, resulting in poor training stability. At the same time, the scene transfer cost of the model is high, making it difficult to quickly adapt to different assembly tasks.

[0004] Furthermore, while existing technologies offer mature solutions for fine-grained identification of industrial components, their limitations in the coarse-positioning stage result in insufficient overall positioning accuracy, failing to meet the demands of industrial grasping. Therefore, there is an urgent need for a visual language model solution adaptable to industrial scenarios, improving cross-modal positioning accuracy while also being compatible with small-sample training to reduce scenario migration costs. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a cross-modal localization and grasping method, system and robot for industrial assembly parts based on a visual language model. By adding an industrial term feature enhancement layer to the visual language model, the semantic feature representation of core industrial terms is strengthened, the accuracy of text-image matching is improved, and a two-stage detection strategy of preliminary coarse localization and fine recognition is combined with a voice-driven human-computer interaction method to achieve accurate and efficient recognition and grasping of industrial assembly parts, thereby improving the level of intelligence and operational flexibility of industrial assembly automation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a cross-modal localization and grasping method for industrial assemblies based on a visual language model, comprising the following steps:

[0008] Collect global environmental images of the industrial assembly scene, receive user voice commands, and convert the voice commands into text commands;

[0009] The text instructions are inferred by a pre-set visual language model adapted to the industrial assembly scenario and the global image input, to obtain the bounding box information of the candidate target assemblies corresponding to the core industrial terms in the text instructions; the visual language model is an adapted model obtained by adding a plug-in industrial term feature enhancement layer to the text encoder output of the basic visual language model; the industrial term feature enhancement layer is configured with an independent industrial term dictionary, which defines the core industrial terms according to the color, shape and category dimensions of the assemblies.

[0010] The position coordinates of the candidate target assembly in the image are calculated based on the bounding box information, and the position coordinates are converted into spatial coordinates in the robot arm coordinate system;

[0011] The robotic arm is controlled to move to the candidate target area corresponding to the spatial coordinates, and a local image of the target is acquired a second time through the camera at the end of the robotic arm;

[0012] The local image of the target is input into a pre-trained target detection model to complete the fine recognition of the assembly and obtain the target's category information and precise location information;

[0013] Based on the category information and precise location information, the robotic arm is controlled to complete the grasping operation of the target assembly.

[0014] As a preferred technical solution, the voice command includes an inherent attribute description of the target assembly's category, shape, and color;

[0015] The text instructions include core industrial terms and non-core descriptive terms. Core industrial terms are keywords that characterize the inherent attributes of the assembly, while non-core descriptive terms are auxiliary terms used for sentence connection and scenario description.

[0016] As a preferred technical solution, the feature processing of the industrial terminology feature enhancement layer specifically includes:

[0017] Based on the industrial terminology dictionary, the word sequence after the text instruction is segmented is matched to generate a binary term mask of the same length as the word sequence, marking the position corresponding to the core industrial term.

[0018] The original text features output by the text encoder are input into the feature projection unit to complete the feature space adaptation and obtain the adapted intermediate features.

[0019] Based on the term mask, term-oriented attention enhancement is performed on the intermediate features to strengthen the feature weights of the corresponding positions of core industrial terms and weaken the feature weights of non-core descriptive words.

[0020] The attention-enhanced features are weighted and fused with the original text features through a gated residual fusion unit to output the final semantic features enhanced with industrial terminology.

[0021] As a preferred technical solution, the bounding box information of the candidate target assembly is obtained through the following steps:

[0022] The Swin Transformer image encoder of the visual language model encodes the environmental image at multiple scales to capture the visual feature information of the assembly in the image;

[0023] The BERT text encoder of the visual language model encodes the initial semantic features of the text instructions and outputs word embedding features. The word embedding features are input to the industrial terminology feature enhancement layer, which completes the feature enhancement of the core industrial terms through four steps: terminology recognition, feature projection adaptation, terminology orientation attention enhancement, and gated residual fusion, and outputs the semantic features of the enhanced core industrial terms.

[0024] The feature enhancement module of the visual language model completes cross-modal fusion and alignment of visual features and enhanced semantic features;

[0025] The language-guided query selection module of the visual language model filters features that are highly relevant to the text semantics from visual features and constructs the initial query vector for the decoder. The cross-modal decoder iteratively optimizes the query vector, completes target bounding box regression and text semantic matching, and finally outputs the bounding box coordinate information of the candidate target assembly.

[0026] As a preferred technical solution, the visual language model employs a phased training approach combined with a lightweight parameter fine-tuning strategy to achieve scene adaptation. It is trained on a text-image pairing dataset based on an industrial assembly scenario. The phased training specifically includes:

[0027] First training phase: Freeze all parameters of the basic visual language model, unfreeze only the parameters of the industrial terminology feature enhancement layer, train the enhancement layer based on the image-text pairing dataset of industrial assembly scenario, and save the training weights of the enhancement layer after completion;

[0028] The second training phase involves loading the pre-trained weights of the base visual language model and the training weights of the enhancement layer, freezing the bottom-level parameters of the base visual language model, unfreezing only the top-level parameters of the base visual language model and the parameters of the enhancement layer, and performing joint fine-tuning based on the same dataset.

[0029] As a preferred technical solution, converting the position coordinates into spatial coordinates in the robotic arm coordinate system specifically includes:

[0030] The camera is fixed to the end of the robotic arm using an eye-on-hand method. Hand-eye calibration of the camera and the robotic arm base coordinate system is completed in advance, and the mapping transformation from image coordinates to robotic arm coordinates is completed based on the hand-eye calibration matrix.

[0031] As a preferred technical solution, the pre-trained object detection model is an improved YOLOv8 model, and the improvement methods include:

[0032] In the Backbone feature extraction stage, a single-head vision Transformer module is used to replace the original C2f module to enhance the ability to model global features and capture fine-grained features.

[0033] In the Neck feature fusion stage, the original SPPF module is replaced by a focus modulation module to improve multi-scale feature fusion and context awareness capabilities.

[0034] In the head detection stage, the detection head is reconstructed by drawing on the RT-DETR decoder to achieve end-to-end detection, and a new high-resolution detection branch is added to enhance the feature extraction of small target assemblies.

[0035] As a preferred technical solution, the target's category information and precise location information are obtained, specifically including:

[0036] After the robotic arm moves to the candidate target area, the control end image acquisition device is adjusted to the preset shooting distance and angle to ensure the quality of local image acquisition;

[0037] High-resolution close-up local images of the target area are acquired using an end-image acquisition device to reduce interference from complex backgrounds;

[0038] The local image is input into a pre-trained target detection model, which then performs category identification and precise location regression of the assemblies in the image.

[0039] Output the precise bounding box coordinates and recognition confidence level of the target assembly, and filter the results with confidence levels higher than a preset threshold as the basis for the robotic arm's grasping execution.

[0040] Secondly, the present invention provides a cross-modal positioning and grasping system for industrial assemblies based on a visual language model, which is applied to the aforementioned cross-modal positioning and grasping method for industrial assemblies based on a visual language model, including an image acquisition and speech processing module, an inference module, a coordinate transformation module, a robotic arm control module, a target detection module, and a positioning and grasping module;

[0041] The image acquisition and voice processing module is used to acquire global images, receive user voice commands, and convert the voice commands into text commands.

[0042] The reasoning module is used to reason with the text command and a pre-set visual language model adapted to the industrial assembly scenario of the global image input to obtain the bounding box information of the candidate target assembly parts corresponding to the core industrial terms in the text command; the visual language model is an adapted model obtained by adding a plug-in industrial term feature enhancement layer to the text encoder output of the basic visual language model; the industrial term feature enhancement layer is configured with an independent industrial term dictionary, which defines the core industrial terms according to the color, shape and category dimensions of the assembly parts.

[0043] The coordinate transformation module is used to calculate the position coordinates of the candidate target assembly in the image based on the bounding box information, and convert the position coordinates into spatial coordinates in the robot arm coordinate system;

[0044] The robotic arm control module is used to control the robotic arm to move to the candidate target area corresponding to the spatial coordinates, and to acquire a second local image of the target through the camera at the end of the robotic arm;

[0045] The target detection module is used to input the local image of the target into a pre-trained target detection model to complete the fine recognition of the assembly and obtain the target's category information and precise location information.

[0046] The positioning and grasping module is used to control the robotic arm to complete the grasping operation of the target assembly based on the category information and precise location information.

[0047] Thirdly, the present invention provides a robot, the robot comprising:

[0048] At least one processor; and,

[0049] A memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the cross-modal positioning and grasping method for industrial components based on a visual language model.

[0051] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0052] (1) This invention adds an industrial term feature enhancement layer to the GroundingDINO model and adopts a term mask-guided directional attention enhancement and gated residual fusion mechanism to achieve accurate focus on core industrial terms after voice commands are converted to text, effectively weakens the interference of non-core descriptive words, and solves the problems of insufficient attention to industrial terms and low text-image feature matching accuracy of general visual language models.

[0053] (2) This invention achieves efficient adaptation of the improved GroundingDINO model to industrial scenarios: It adopts a two-stage training strategy of first training the industrial terminology feature enhancement layer separately and then fine-tuning it together, combined with LoRA lightweight fine-tuning technology, which avoids the interference of randomly initialized ITFE parameters with the model's pre-training weights and improves the model's adaptability to industrial small sample datasets; at the same time, the industrial terminology feature enhancement layer is a lightweight incremental module, which is connected in a decoupled manner, without destroying the original internal network structure of the GroundingDINO model, with few new parameters and no additional inference burden; LoRA fine-tuning only needs to update a small number of low-rank matrix parameters, and the proportion of trainable parameters is significantly lower than that of the full parameter fine-tuning scheme.

[0054] (3) Reduced model scene migration cost: This invention enables rapid adaptation to different industrial assembly recognition scenarios through an independent configurable industrial terminology dictionary. No model code needs to be modified; only the terminology in the dictionary needs to be added, deleted, or modified to migrate the model to recognition tasks of different industrial assemblies such as nuts, bolts, washers, and rivets, which greatly improves the model's versatility and engineering applicability.

[0055] (4) Improve the recognition accuracy and success rate of industrial assembly parts: This invention constructs a two-stage recognition and positioning system that uses a visual language model with an industrial terminology feature enhancement layer for coarse localization and a target detection model for precise recognition. First, the optimized visual language model is used to achieve coarse localization of candidate targets. Then, high-quality local images are obtained through secondary close-range shooting, and the target detection model completes the precise recognition. This effectively solves the problems of misidentification and missed identification caused by similar appearance of assemblies, complex backgrounds, and occlusion in complex industrial assembly scenarios.

[0056] (5) Flexible voice-driven human-computer interaction is realized: In this invention, users only need to use simple voice description to complete the identification and grasping control of target assembly parts, which replaces the traditional manual operation or fixed control command mode, improves the convenience and flexibility of industrial assembly operation, and adapts to the flexible industrial assembly needs. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of a cross-modal localization and grasping method for industrial assemblies based on a visual language model, according to an embodiment of the present invention.

[0059] Figure 2 This is a structural diagram of the Industrial Terminology Feature Enhancement Layer (ITFE) in an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the LoRA training process in an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of the network structure of the improved visual language model (GroundingDINO) in an embodiment of the present invention;

[0062] Figure 5 This is a schematic diagram of the improved YOLOv8 model network structure used for two-stage accurate detection in an embodiment of the present invention;

[0063] Figure 6 This is a block diagram of an industrial assembly cross-modal positioning and grasping system based on a visual language model, according to an embodiment of the present invention.

[0064] Figure 7 This is a structural diagram of the robot according to an embodiment of the present invention. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0066] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0067] This embodiment uses voice-driven data capture of nuts and bolts in an industrial assembly scenario as an application example, combined with... Figure 1 The overall process shown illustrates the method of this invention in detail. In this embodiment, the basic visual language model is the GroundingDINO model, its text encoder is the BERT model based on the Transformer architecture, and its image encoder is the Swin Transformer model based on the Transformer architecture; the object detection model uses the improved YOLOv8 model; the robotic arm is the UR5 industrial robotic arm, the image acquisition device is the ZED 2i binocular depth camera, and the speech recognition uses Huawei Cloud voice interaction service. Figure 1 As shown, the voice-driven recognition and capture method for industrial assembly parts includes the following steps:

[0068] S1. Collect global environmental images of the industrial assembly scene, receive user voice commands, and convert the voice commands into text commands containing core industrial terminology.

[0069] Furthermore, step S1 specifically involves:

[0070] A global image of the industrial assembly scene is acquired through an image acquisition device, which is an RGB-D camera, capable of capturing the spatial position and visual feature information of the assembly parts. A voice acquisition module receives voice commands issued by the user, which include descriptions of the inherent attributes of the target assembly parts, such as category, shape, and color. The voice acquisition module transmits the voice signal to a voice recognition module, which converts the voice signal into text commands. The text commands include core industrial terms and non-core descriptive words, where the core industrial terms are keywords representing the inherent attributes of the assembly parts, and the non-core descriptive words are auxiliary words used for sentence connection and scene description.

[0071] For example, in an industrial assembly scenario, a ZED 2i binocular depth camera fixed to the end effector of the UR5 robotic arm acquires RGB images and depth information of the overall environment of the assembly table, capturing the visual features and spatial position information of all components within the scene. The acquired images cover different shooting angles, lighting conditions, background environments, and occlusion states, adapting to the complex scenarios of industrial assembly. Simultaneously, the operator issues voice commands through the voice acquisition module. These voice commands include descriptions of the inherent attributes of the target component, such as its category, specifications, shape, and color. For example, the user issues the voice command "Find the silver hexagonal nut on the assembly table." The voice acquisition module transmits the voice signal to the voice recognition module, which converts the voice signal into a text command. In this text command, the core industrial terms are "silver," "hexagonal," and "nut," while non-core descriptive words include "find" and "on the assembly table." The core industrial terms are key words representing the inherent attributes of the components and are crucial for subsequent cross-modal matching by the visual language model.

[0072] S2. The text instruction is inferred by the global image input to a preset visual language model adapted to the industrial assembly scenario to obtain the bounding box information of the candidate target assembly matching the text instruction.

[0073] Furthermore, the visual language model is an adapted model obtained by adding a plug-in Industrial Terminology Feature Enhancement Layer (ITFE) to the output of the text encoder of the basic visual language model. The ITFE is an incremental plug-in module, connected to the output of the text encoder as an independent plug-in, retaining the core architecture, training process, and deployment logic of the basic visual language model. The basic visual language model is the GroundingDINO model, with its text encoder being a BERT model based on the Transformer architecture and its image encoder being a Swin Transformer model based on the Transformer architecture. The ITFE configures an industrial terminology dictionary, which defines core industrial terms according to dimensions such as the color, shape, and category of assemblies, and supports adding, deleting, and modifying terminology content according to different recognition scenarios.

[0074] Furthermore, the text instructions obtained in step S1 and the global RGB image are input into the improved GroundingDINO visual language model for inference. The core improvement of this model lies in the addition of a plug-in industrial terminology feature enhancement layer (ITFE) at the output of the BERT text encoder. The overall network structure of the improved GroundingDINO is as follows: Figure 4 As shown. The reasoning process is as follows:

[0075] First, the model's Swing Transformer image encoder performs multi-scale visual feature encoding on the input global environment image, capturing the visual feature information of all components in the image.

[0076] Meanwhile, the model's BERT text encoder encodes the initial semantic features of the text instructions and outputs word embedding features.

[0077] Subsequently, the word embedding features are input into the industrial terminology feature enhancement layer. Through four processing steps—terminology dictionary loading and mask generation, text feature projection adaptation, terminology-oriented attention enhancement, and gated residual fusion—the feature enhancement of the core industrial terms is completed, and the enhanced semantic features of the core industrial terms are output.

[0078] Then, the model's feature enhancement module completes the cross-modal fusion and alignment of visual features and enhanced semantic features through intramodal self-attention and bidirectional cross-modal cross-attention.

[0079] Next, the language-guided query selection module filters out features that are highly relevant to the text semantics from the fused visual features to construct the decoder's initial query vector.

[0080] Finally, the cross-modal decoder iteratively optimizes the query vector through a multi-layer structure of self-attention, image cross-attention, and text cross-attention, completes target bounding box regression and text semantic matching, and outputs the bounding box coordinates and confidence scores of candidate target assemblies corresponding to the core industrial terms in the text instructions.

[0081] In some specific embodiments, combined with Figure 2 The specific structure and workflow of the Industrial Terminology Feature Enhancement Layer (ITFE) described in step S2 will be further elaborated in detail.

[0082] 1. Configuration of an industrial terminology dictionary;

[0083] The ITFE configuration includes an industrial terminology dictionary, a JSON-formatted text file that defines core industrial terms categorized by color, shape, and type. For example, for nuts and bolts, colors include yellow, silver, black, and red; shapes include hexagonal, round, and square; and categories include nuts, bolts, and rivets. This dictionary is an independent configuration file, allowing for flexible addition, deletion, and modification of terminology based on different industrial assembly scenarios without requiring modifications to the GroundingDINO model code, enabling rapid model migration across various industrial assembly scenarios.

[0084] 2. Overall structure and workflow of ITFE;

[0085] like Figure 2As shown, ITFE is plugged into the output of the BERT text encoder of the GroundingDINO model. ITFE takes the general text features output by the BERT text encoder as input, and while preserving the original features, generates enhanced industrial terminology features through a parallel enhancement branch. Then, a gating mechanism is used to fuse the enhanced information back into the original features, outputting the enhanced text features. ITFE consists of four units: a terminology dictionary loading and mask generation unit, a text feature projection adaptation unit, a terminology-oriented attention enhancement unit, and a gating residual fusion unit.

[0086] 3. Detailed workflow of each unit in ITFE;

[0087] Combination Figure 2 The specific feature processing flow for each unit of ITFE is as follows:

[0088] (1) Terminology dictionary loading and mask generation unit;

[0089] After the input text command is processed by the BERT tokenizer, a fixed-length sequence of tokens is obtained. Considering that the BERT tokenizer may split some long words into multiple sub-words (e.g., hexagonal is split into he, ##xa, ##gonal), a token number matching strategy is not suitable for sub-word segmentation. Therefore, this terminology dictionary loading and mask generation unit adopts a character-level matching strategy: first, each token is restored to its original character and sub-word prefixes (such as "##") are removed; then, the restored character sequence is concatenated into a complete string in sequence, and it is determined whether the current character segment belongs to a complete word in the terminology dictionary. If it does, the corresponding position is marked as a valid term. Finally, a binary term mask of the same length as the token sequence is generated, with the corresponding position set to 1 and the remaining positions set to 0.

[0090] (2) Text feature projection adaptation unit;

[0091] BERT pre-training is geared towards general natural language scenarios, but the distribution of its output raw text features, Fbert, is not optimized for industrial terminology. Directly using it for subsequent attention calculations can easily lead to insufficient representation of terminology features. This paper's feature projection adaptation unit performs lightweight nonlinear transformations and feature distribution optimization on Fbert. For example... Figure 2 As shown, this text feature projection adaptation unit consists of two layers of linear transformation, layer normalization, and the GELU activation function. It optimizes the feature distribution while maintaining the feature dimension unchanged, avoiding the introduction of additional dimension transformation overhead. The specific calculation process is as follows:

[0092]

[0093] Where W1 and W2 are the learnable weight matrices of the linear transformation, and b1 and b2 are the bias vectors.

[0094] (3) Terminology-oriented attention enhancement unit;

[0095] The core objective of this terminology-oriented attention enhancement unit is to strengthen the semantic connections between industrial terms while filtering out interference from non-terminology areas. For example... Figure 2 As shown, this term-oriented attention enhancement unit uses the projected and adapted feature Fproj as the query (Q), key (K), and value vector (V) for attention calculation. It employs a two-head attention mechanism (h=2 attention heads) and constrains the attention calculation range by using the complement of the term mask, allowing only features at term positions to calculate attention weights together, while suppressing attention interactions at non-term positions. Finally, residual connections ensure the stability of the training process. The calculation process is as follows:

[0096] in, The complement of the term mask, , , Let h be the projected weight of the k-th attention head, and h be the number of attention heads. Pay attention to the linear transformation weights after splicing the attention heads.

[0097] (4) Gated residual fusion unit;

[0098] To balance preserving the general semantics of the original text with precise control over the enhancement intensity of industrial terminology features, this gated residual fusion unit uses a gating mechanism to achieve dynamic weighted fusion, replacing the manually set fixed weighting method. This allows the enhancement intensity to be dynamically adjusted according to the importance of terms in different texts. Figure 2 As shown, this gated residual fusion unit first generates gate weights based on the original BERT features, then constrains the scope of the gate weights through a terminology mask to ensure that the enhancement only applies to the industrial terminology region. Finally, the weighted enhanced features are fused with the original BERT features. The specific calculation process is as follows:

[0099]

[0100] Where σ(·) is the Sigmoid function, The global gating weight matrix, where the weight value at each position represents the enhancement strength of the feature at that position. This is an element-wise multiplication operation.

[0101] In some specific embodiments, combined with Figure 3 and Figure 4The training strategy for the improved GroundingDINO model with added ITFE is further elaborated in detail. The visual language model employs a phased training approach combined with a LoRA lightweight fine-tuning strategy to achieve scene adaptation. Training is based on an image-text pairing dataset from an industrial assembly scenario. LoRA is a lightweight fine-tuning technique for Transformer architectures, and its training process is as follows: Figure 3 As shown. For any linear layer in the Transformer architecture that needs fine-tuning, its original pre-trained weight matrix is ​​W0. LoRA injects two low-rank matrices A and B next to W0, transforming the forward propagation calculation of the original linear layer into y = W0x + BAx. Matrix A is randomly initialized using a Gaussian distribution, and matrix B is initialized as a zero matrix. During training, the pre-trained weight W0 does not participate in parameter updates; only the low-rank matrices A and B are trained. During model inference, the trained low-rank matrices are fused with the original weight matrix, and the fused weight matrix participates in the forward propagation. The phased training combined with the lightweight parameter fine-tuning strategy specifically includes two training phases:

[0102] The first training phase (training ITFE separately): The official pre-trained weights of the GroundingDINO model are loaded. All parameters of the model's Swing Transformer image encoder, BERT text encoder layers, feature enhancement module, cross-modal decoder, and detector head are frozen. Only the parameters of the newly added industrial terminology feature enhancement layer are unfrozen. The Adam optimizer is used, setting the ITFE to twice the base learning rate and with weight decay of 0. Training is performed on the aforementioned image-text pairing dataset, allowing ITFE to gradually learn and form stable rules for recognizing core industrial terms and strengthening weights. After training, the training weights of ITFE are saved separately to prepare for subsequent joint fine-tuning.

[0103] The second training phase (joint fine-tuning) involves loading the pre-trained weights of the GroundingDINO model and the ITFE weights trained in the first phase. Combined with... Figure 4 LoRA low-rank matrices are injected into the key linear layers of the image backbone network (Swin Transformer) and the text backbone network (BERT). The original pre-trained weights of the image backbone network and the text backbone network are frozen, and only the injected LoRA low-rank matrix parameters, the top-level parameters of the BERT text encoder, the ITFE parameters, the feature enhancement module, the cross-modal decoder, and the detector head parameters are unfrozen. Training continues using the same image-text pairing dataset to optimize the cross-modal matching accuracy of industrial core terminology enhancement features and image features.

[0104] Through the aforementioned two-stage training strategy, the first stage of individual training allows ITFE to gradually converge without being affected by the gradients of other modules, forming a stable term augmentation capability. The second stage, based on this, performs joint fine-tuning to avoid the interference of randomly initialized ITFE parameters on the cross-modal matching process in the early stages of training, thus synergistically improving term augmentation, lightweight backbone adaptation, and localization capabilities. Simultaneously, the LoRA lightweight fine-tuning technique updates only a small number of low-rank matrix parameters, achieving efficient model adaptation to industrial assembly scenarios while retaining the general cross-modal matching capability of pre-trained weights, effectively mitigating overfitting issues under small sample conditions.

[0105] S3. Calculate the position coordinates of the candidate target assembly in the image based on the bounding box information, and convert the position coordinates into spatial coordinates in the robot arm coordinate system.

[0106] Furthermore, based on the bounding box information of the candidate target assembly obtained in step S2, the pixel coordinates of the bounding box center in the image are calculated. Combining the ZED 2i camera intrinsic parameters and the depth value at the corresponding position in the depth image, these pixel coordinates are converted into three-dimensional coordinates in the camera coordinate system. Then, based on the pre-completed hand-eye calibration matrix, the three-dimensional coordinates in the camera coordinate system are converted into spatial coordinates in the UR5 robotic arm base coordinate system. The hand-eye calibration adopts an eye-on-hand calibration method, pre-collecting multiple sets of checkerboard images in different poses and corresponding robotic arm end-effector pose data to solve for the rotation matrix and translation vector from the camera coordinate system to the robotic arm end-effector coordinate system.

[0107] S4. Control the robotic arm to move to the candidate target area corresponding to the spatial coordinates, and acquire a local image of the target through the camera at the end of the robotic arm.

[0108] Furthermore, the spatial coordinates in the robot arm's base coordinate system obtained in step S3 are sent to the UR5 robot arm control system. The control system uses joint motion commands and linear motion commands to plan the motion trajectory, driving the robot arm to move to the area where the candidate target assembly is located, so that the end effector of the robot arm stops at a preset distance directly above the candidate target. Then, the ZED 2i camera installed at the end effector of the robot arm is adjusted to a preset shooting angle to take a second vertical downward shot of the candidate target area, obtaining a local close-up high-definition image of the target assembly. This local image effectively reduces the impact of background interference, object overlap, and viewpoint occlusion on recognition in industrial scenes, significantly improving the pixel ratio and detail clarity of the target assembly in the image, providing high-quality image input for subsequent fine recognition.

[0109] S5. Input the local image of the target into the pre-trained target detection model to complete the fine recognition of the assembly and obtain the target's category information and precise location information.

[0110] Furthermore, the local close-range high-resolution image acquired in step S4 is input into the pre-trained improved YOLOv8 object detection model, which performs accurate category identification and location regression of the assemblies in the image. After inference, the model outputs the precise category information, bounding box coordinates, and recognition confidence score of the target assembly. The system filters recognition results with confidence scores higher than a preset threshold as valid criteria for robotic arm grasping. If the target category identified by the model matches the core industrial terminology described in the voice command, the candidate target is confirmed as the correct grasping object, and the pixel coordinates of its precise bounding box center are used as the precise grasping point.

[0111] In some specific embodiments, the network structure of the pre-trained object detection model—the improved YOLOv8 model—is described in detail. The improved YOLOv8 model improves upon the original YOLOv8 model in three stages: feature extraction, feature fusion, and the detection head. Figure 5 As shown, the details are as follows:

[0112] (1) In the Backbone feature extraction stage, a single-head visual Transformer module (SHViTBlock) is introduced to replace the original C2f module. SHViTBlock consists of a deep convolution, a single-head self-attention module and a feedforward network. It extracts detailed features of local regions of the image through deep convolution and efficiently captures the correlation between distant regions in the image through the single-head self-attention module, thereby enhancing the ability to capture global contextual dependencies and long-distance structural relationships. Without significantly increasing the computational overhead, it effectively improves the richness of features and the ability to distinguish fine-grained features.

[0113] (2) In the Neck feature fusion stage, the original fast spatial pyramid pooling module (SPPF) is replaced by the Focal modulation module. The Focal modulation module consists of two parts: context aggregation and query-guided modulation. First, it achieves progressive extraction from local fine-grained features to global coarse-grained features by stacking multiple deep convolutions. Then, it aggregates the context features of different levels into a unified global context feature by gated weighted fusion. Finally, it injects the aggregated context features into the query features by element-wise multiplication, thereby achieving efficient fusion of multi-scale features and improving context awareness.

[0114] (3) In the Head detection section, the detection head is reconstructed by referencing the RT-DETR decoder mechanism to achieve end-to-end target detection, and a new P2 high-resolution detection branch is added to enhance the extraction of small target features. The reconstructed detection head consists of five parts: multi-scale feature alignment and unified embedding, multi-scale feature fusion, target query construction, deformable Transformer decoding, and classification and regression output. Among them, multi-scale feature alignment maps features of different scales from P2 to P5 to a unified dimension; target query construction adopts an uncertainty-minimizing query selection strategy, selecting the features with the highest classification and localization quality from the fused features as the initial query; the deformable Transformer decoder updates the query features and reference box positions through multi-layer iteration and adopts a sparse sampling deformable attention mechanism to gradually approach the true boundary and category information of the target. The improved YOLOv8 model significantly improves the ability to distinguish the categories of parts and the localization accuracy while maintaining real-time inference speed, providing reliable recognition and localization support for the precise grasping of robotic arms.

[0115] S6. Based on the category information and precise location information, control the robotic arm to complete the grasping operation of the target assembly.

[0116] Furthermore, the precise pixel coordinates of the target assembly accurately identified in step S5 are converted back into precise spatial coordinates in the robotic arm's base coordinate system using the hand-eye calibration matrix and camera intrinsic parameters described in step three. A grasping command is sent to the robotic arm control system. The end effector's electric gripper receives the opening control command via serial communication and adaptively adjusts the opening size according to the target assembly's dimensions, driving the robotic arm to perform the grasping action. Based on the category information inferred from the improved YOLOv8 model, the grasped assembly is placed into a pre-defined designated area for that category, completing the entire process of identifying, grasping, and sorting the "silver hexagonal nut".

[0117] After inference, the model outputs the precise category information, bounding box coordinates, and recognition confidence score of the target assembly. The system filters recognition results with a confidence score higher than a preset threshold as valid criteria for robotic arm grasping. If the target category recognized by the model matches the core industrial terminology described in the voice command, the candidate target is confirmed as the correct grasping object, and the pixel coordinates of its precise bounding box center are used as the precise grasping point.

[0118] Please see Figure 6 In another embodiment of this application, a cross-modal positioning and grasping system 100 for industrial assembly parts based on a visual language model is provided. The system includes an image acquisition and voice processing module 101, an inference module 102, a coordinate transformation module 103, a robotic arm control module 104, a target detection module 105, and a positioning and grasping module 106.

[0119] The image acquisition and voice processing module 101 is used to acquire global images, receive user voice commands, and convert the voice commands into text commands.

[0120] The reasoning module 102 is used to reason with the text instruction and the global image input preset visual language model adapted to the industrial assembly scenario to obtain the bounding box information of the candidate target assembly parts corresponding to the core industrial terms in the text instruction; the visual language model is an adapted model obtained by adding a plug-in industrial term feature enhancement layer to the text encoder output of the basic visual language model; the industrial term feature enhancement layer is configured with an independent industrial term dictionary, which defines the core industrial terms according to the color, shape and category dimensions of the assembly parts.

[0121] The coordinate transformation module 103 is used to calculate the position coordinates of the candidate target assembly in the image based on the bounding box information, and convert the position coordinates into spatial coordinates in the robot arm coordinate system.

[0122] The robotic arm control module 104 is used to control the robotic arm to move to the candidate target area corresponding to the spatial coordinates, and to acquire a second local image of the target through the camera at the end of the robotic arm.

[0123] The target detection module 105 is used to input the local image of the target into a pre-trained target detection model to complete the fine recognition of the assembly and obtain the target's category information and precise location information.

[0124] The positioning and grasping module 106 is used to control the robotic arm to complete the grasping operation of the target assembly based on the category information and precise location information.

[0125] It should be noted that the cross-modal localization and grasping system for industrial assemblies based on visual language models of the present invention corresponds one-to-one with the cross-modal localization and grasping method for industrial assemblies based on visual language models of the present invention. The technical features and beneficial effects described in the embodiments of the cross-modal localization and grasping method for industrial assemblies based on visual language models are applicable to the embodiments of cross-modal localization and grasping of industrial assemblies based on visual language models. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0126] Furthermore, in the above embodiments of the cross-modal positioning and grasping system for industrial assembly parts based on visual language models, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the cross-modal positioning and grasping system for industrial assembly parts based on visual language models can be divided into different program modules to complete all or part of the functions described above.

[0127] Please see Figure 7 In one embodiment, a robot is provided that implements a cross-modal localization and grasping method for industrial assemblies based on a visual language model. The robot 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a cross-modal localization and grasping program 203 for industrial assemblies based on a visual language model.

[0128] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the robot 200, such as the robot 200's portable hard drive. In other embodiments, the first memory 202 can also be an external storage device of the robot 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the robot 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the robot 200. The first memory 202 can be used not only to store application software and various types of data installed on the robot 200, such as the code of the cross-modal positioning and grasping program 203 for industrial parts based on a visual language model, but also to temporarily store data that has been output or will be output.

[0129] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the robot, connecting various components of the robot through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the robot 200 and process data.

[0130] Figure 7 Only a robot with parts is shown; those skilled in the art will understand that... Figure 7 The structure shown does not constitute a limitation on the robot 200, and may include fewer or more parts than shown, or combine certain parts, or have different arrangements of parts.

[0131] The first memory 202 in the robot 200 stores a cross-modal localization and grasping program 203 for industrial assembly parts based on a visual language model. This program is a combination of multiple instructions, and when run in the first processor 201, it can achieve the following:

[0132] Collect global environmental images of the industrial assembly scene, receive user voice commands, and convert the voice commands into text commands;

[0133] The text instructions are inferred by a pre-set visual language model adapted to the industrial assembly scenario and the global image input, to obtain the bounding box information of the candidate target assemblies corresponding to the core industrial terms in the text instructions; the visual language model is an adapted model obtained by adding a plug-in industrial term feature enhancement layer to the text encoder output of the basic visual language model; the industrial term feature enhancement layer is configured with an independent industrial term dictionary, which defines the core industrial terms according to the color, shape and category dimensions of the assemblies.

[0134] The position coordinates of the candidate target assembly in the image are calculated based on the bounding box information, and the position coordinates are converted into spatial coordinates in the robot arm coordinate system;

[0135] The robotic arm is controlled to move to the candidate target area corresponding to the spatial coordinates, and a local image of the target is acquired a second time through the camera at the end of the robotic arm;

[0136] The local image of the target is input into a pre-trained target detection model to complete the fine recognition of the assembly and obtain the target's category information and precise location information;

[0137] Based on the category information and precise location information, the robotic arm is controlled to complete the grasping operation of the target assembly.

[0138] Furthermore, if the modules / units integrated into the robot 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A cross-modal localization and grasping method for industrial assemblies based on a visual language model, characterized in that, Includes the following steps: Collect global environmental images of the industrial assembly scene, receive user voice commands, and convert the voice commands into text commands; The text instructions are inferred by a pre-set visual language model adapted to the industrial assembly scenario and the global image input, to obtain the bounding box information of the candidate target assemblies corresponding to the core industrial terms in the text instructions; the visual language model is an adapted model obtained by adding a plug-in industrial term feature enhancement layer to the text encoder output of the basic visual language model; the industrial term feature enhancement layer is configured with an independent industrial term dictionary, which defines the core industrial terms according to the color, shape and category dimensions of the assemblies. The position coordinates of the candidate target assembly in the image are calculated based on the bounding box information, and the position coordinates are converted into spatial coordinates in the robot arm coordinate system; The robotic arm is controlled to move to the candidate target area corresponding to the spatial coordinates, and a local image of the target is acquired a second time through the camera at the end of the robotic arm; The local image of the target is input into a pre-trained target detection model to complete the fine recognition of the assembly and obtain the target's category information and precise location information; Based on the category information and precise location information, the robotic arm is controlled to complete the grasping operation of the target assembly.

2. The cross-modal localization and grasping method for industrial assemblies based on a visual language model according to claim 1, characterized in that, The voice commands include descriptions of the inherent attributes of the target assembly, such as its category, shape, and color. The text instructions include core industrial terms and non-core descriptive terms. Core industrial terms are keywords that characterize the inherent attributes of the assembly, while non-core descriptive terms are auxiliary terms used for sentence connection and scenario description.

3. The cross-modal positioning and grasping method for industrial assemblies based on a visual language model according to claim 1, characterized in that, The feature processing of the industrial terminology feature enhancement layer specifically includes: Based on the industrial terminology dictionary, the word sequence after the text instruction is segmented is matched to generate a binary term mask of the same length as the word sequence, marking the position corresponding to the core industrial terminology. The original text features output by the text encoder are input into the feature projection unit to complete the feature space adaptation and obtain the adapted intermediate features. Based on the term mask, term-oriented attention enhancement is performed on the intermediate features to strengthen the feature weights of the corresponding positions of core industrial terms and weaken the feature weights of non-core descriptive words. The attention-enhanced features are weighted and fused with the original text features through a gated residual fusion unit to output the final semantic features enhanced with industrial terms.

4. The cross-modal localization and grasping method for industrial assemblies based on a visual language model according to claim 1, characterized in that, The bounding box information of the candidate target assembly is obtained through the following steps: The Swin Transformer image encoder of the visual language model encodes the environmental image at multiple scales to capture the visual feature information of the assembly in the image; The BERT text encoder in the visual language model encodes initial semantic features of text instructions and outputs word embedding features; Word embedding features are input into the industrial terminology feature enhancement layer. The feature enhancement of core industrial terms is completed through four steps: terminology recognition, feature projection adaptation, terminology orientation attention enhancement, and gated residual fusion. The semantic features of the enhanced core industrial terms are output. The feature enhancement module of the visual language model completes cross-modal fusion and alignment of visual features and enhanced semantic features; The language-guided query selection module of the visual language model filters features that are highly relevant to the text semantics from visual features and constructs the initial query vector for the decoder. The cross-modal decoder iteratively optimizes the query vector, completes target bounding box regression and text semantic matching, and finally outputs the bounding box coordinate information of the candidate target assembly.

5. The cross-modal positioning and grasping method for industrial assemblies based on a visual language model according to claim 1, characterized in that, The visual language model employs a phased training approach combined with a lightweight parameter fine-tuning strategy to achieve scene adaptation. It is trained on a text-image pairing dataset based on industrial assembly scenarios. The phased training specifically includes: First training phase: Freeze all parameters of the basic visual language model, unfreeze only the parameters of the industrial terminology feature enhancement layer, train the enhancement layer based on the image-text pairing dataset of industrial assembly scenario, and save the training weights of the enhancement layer after completion; The second training phase involves loading the pre-trained weights of the base visual language model and the training weights of the enhancement layer, freezing the bottom-level parameters of the base visual language model, unfreezing only the top-level parameters of the base visual language model and the parameters of the enhancement layer, and performing joint fine-tuning based on the same dataset.

6. The cross-modal positioning and grasping method for industrial assemblies based on a visual language model according to claim 1, characterized in that, Converting the position coordinates into spatial coordinates in the robotic arm coordinate system specifically includes: The camera is fixed to the end of the robotic arm using an eye-on-hand method. Hand-eye calibration of the camera and the robotic arm base coordinate system is completed in advance, and the mapping transformation from image coordinates to robotic arm coordinates is completed based on the hand-eye calibration matrix.

7. The cross-modal localization and grasping method for industrial assemblies based on a visual language model according to claim 1, characterized in that, The pre-trained object detection model is an improved YOLOv8 model, and the improvements include: In the Backbone feature extraction stage, a single-head vision Transformer module is used to replace the original C2f module to enhance the ability to model global features and capture fine-grained features. In the Neck feature fusion stage, the original SPPF module is replaced by a focus modulation module to improve multi-scale feature fusion and context awareness capabilities. In the head detection stage, the detection head is reconstructed by drawing on the RT-DETR decoder to achieve end-to-end detection, and a new high-resolution detection branch is added to enhance the feature extraction of small target assemblies.

8. The cross-modal positioning and grasping method for industrial assemblies based on a visual language model according to claim 1, characterized in that, Obtain the target's category information and precise location information, specifically including: After the robotic arm moves to the candidate target area, the control end image acquisition device is adjusted to the preset shooting distance and angle to ensure the quality of local image acquisition; High-resolution close-up local images of the target area are acquired using an end-image acquisition device to reduce interference from complex backgrounds; The local image is input into a pre-trained target detection model, which then performs category identification and precise location regression of the assemblies in the image. Output the precise bounding box coordinates and recognition confidence level of the target assembly, and filter the results with confidence levels higher than a preset threshold as the basis for the robotic arm's grasping execution.

9. A cross-modal positioning and grasping system for industrial assemblies based on a visual language model, characterized in that, The cross-modal positioning and grasping method for industrial assemblies based on a visual language model, as described in any one of claims 1-8, includes an image acquisition and speech processing module, an inference module, a coordinate transformation module, a robotic arm control module, a target detection module, and a positioning and grasping module. The image acquisition and voice processing module is used to acquire global images, receive user voice commands, and convert the voice commands into text commands. The reasoning module is used to reason with the text command and a pre-set visual language model adapted to the industrial assembly scenario of the global image input to obtain the bounding box information of the candidate target assembly parts corresponding to the core industrial terms in the text command; the visual language model is an adapted model obtained by adding a plug-in industrial term feature enhancement layer to the text encoder output of the basic visual language model; the industrial term feature enhancement layer is configured with an independent industrial term dictionary, which defines the core industrial terms according to the color, shape and category dimensions of the assembly parts. The coordinate transformation module is used to calculate the position coordinates of the candidate target assembly in the image based on the bounding box information, and convert the position coordinates into spatial coordinates in the robot arm coordinate system; The robotic arm control module is used to control the robotic arm to move to the candidate target area corresponding to the spatial coordinates, and to acquire a second local image of the target through the camera at the end of the robotic arm; The target detection module is used to input the local image of the target into a pre-trained target detection model to complete the fine recognition of the assembly and obtain the target's category information and precise location information. The positioning and grasping module is used to control the robotic arm to complete the grasping operation of the target assembly based on the category information and precise location information.

10. A robot, characterized in that, The robot includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the cross-modal positioning and grasping method for industrial assembly based on a visual language model as described in any one of claims 1-8.

Images