A mechanical arm grasping method and system based on semantic constraint and multi-modal fusion

Abstract

This invention provides a robotic arm grasping method and system based on semantic constraints and multimodal fusion, belonging to the field of robot grasping technology. The method includes: acquiring scene visual information and user voice commands; detecting and locating target objects using a lightweight improved MSBD-YOLO model; semantically encoding the voice commands to obtain text semantic features; spatially expanding and pixel-level fusing the text semantic features and visual features; predicting grasping parameters and generating multiple candidate grasping points based on the fused features; evaluating the semantic consistency of the candidate points using the text semantic features; selecting the target grasping point based on grasping quality parameters; and finally controlling the robotic arm to perform the grasping action. This invention, by introducing a semantic consistency evaluation mechanism, incorporates user intent into the grasping decision, solving the technical problem that robots struggle to accurately select and grasp specific targets based on complex semantic commands, thus improving the accuracy and intelligence level of human-computer interaction grasping.

Description

Technical Field

[0001] This invention relates to the field of robot grasping technology, and in particular to a robotic arm grasping method and system based on semantic constraints and multimodal fusion. Background Technology

[0002] Existing robotic grasping systems typically rely on RGB or RGB-D images for grasping detection, using deep learning models to predict the grasping position, angle, and width of the target object. However, these methods primarily model based on visual geometric information, making it difficult to select specific grasping targets based on human semantic commands in complex scenes.

[0003] In practical applications, when there are multiple graspable objects in a scene or when selection is required based on object attributes and spatial relationships, relying solely on visual grasping detection networks is insufficient to meet the needs of interactive voice grasping. Furthermore, in existing technologies, speech recognition, semantic understanding, and grasping execution are often independent of each other. Semantic information largely remains at the feature layer or perception layer, lacking a mechanism to semantically constrain specific grasping schemes during the grasping decision-making stage, leading to a mismatch between the grasping target and semantic instructions. Summary of the Invention

[0004] To address the technical problem in the aforementioned background art where robots struggle to accurately select and grasp specific targets based on complex semantic instructions, this invention provides a robotic arm grasping method and system based on semantic constraints and multimodal fusion. By combining voice command semantic encoding, visual detection and localization, multimodal feature fusion, and semantic constraint decision-making, lightweight high-precision object detection, cross-modal feature alignment, and grasping point selection based on semantic consistency are integrated, effectively solving the problem of the disconnect between semantic instructions and visual grasping results.

[0005] To achieve the above objectives, a first aspect of the present invention provides a robotic arm grasping method based on semantic constraints and multimodal fusion, comprising:

[0006] The system acquires visual information of the scene to be captured and the user's voice commands, identifies and locates target objects in the scene based on the visual information, and performs recognition and semantic encoding on the voice commands to obtain text semantic features that represent the user's intent.

[0007] The text semantic features are fused with the information of the target object, and the crawling parameters are predicted based on the fused features to generate multiple candidate crawling points.

[0008] Using the text semantic features as semantic constraints, the semantic consistency of the multiple candidate crawling points is evaluated, the target crawling point is selected, and crawling control instructions are generated.

[0009] The robotic arm is controlled to perform a grasping operation according to the grasping control command.

[0010] Furthermore, semantic consistency evaluation is performed on the multiple candidate crawling points, including:

[0011] Based on the prediction results of the crawling parameters, the crawling quality parameters corresponding to each candidate crawling point are obtained;

[0012] From the fused features, regional features centered on each candidate grasping point are extracted as corresponding local visual features.

[0013] Calculate the semantic correlation between the text semantic features and each local visual feature to obtain the semantic consistency score of each candidate crawling point;

[0014] The target crawl point is determined based on the crawling quality parameters and semantic consistency score of each candidate crawl point.

[0015] Furthermore, the determination of the target capture point is achieved through the following weighted decision function:

[0016] ;

[0017] in, For the index of the target crawl point, This indicates the capture quality parameters; Indicates semantic consistency score; Weighting coefficients for quality parameters This is a weighting coefficient for semantic consistency, used to balance geometric catchability and semantic matching degree.

[0018] Furthermore, the target objects in the scene are identified and located based on the visual information using a pre-trained MSBD-YOLO detection model. The backbone network of the MSBD-YOLO detection model adopts a lightweight MobileNetv4 network structure, and its last ordinary convolutional layer is replaced with a lightweight convolutional module DWGConv, denoted as MobileNetv4-I. The neck network of the MSBD-YOLO detection model adopts an SDII-BiFPN structure, and a RECSA attention module is introduced into it. The detection head of the MSBD-YOLO detection model adopts a Dynamic Head structure.

[0019] Furthermore, the lightweight convolutional module DWGConv performs the following processing steps:

[0020] Perform the DepthwiseConv operation on the input features to obtain initial features containing basic contour information;

[0021] Perform a GhostConv operation on the initial features to generate detailed features containing detailed texture information;

[0022] The initial feature is concatenated with the detailed feature;

[0023] The concatenated features are subjected to convolution, batch normalization, and ReLU activation to obtain the final output features of the lightweight convolution module DWGConv.

[0024] Furthermore, the text semantic features are fused with the information of the target object, including:

[0025] The text semantic features are spatially expanded to generate a semantic feature tensor with the same spatial resolution as the visual features.

[0026] The semantic feature tensor is concatenated with the visual features containing the target object information at the pixel level to obtain the concatenated fused feature tensor.

[0027] Perform a convolution operation on the spliced ​​fused feature tensor to generate the fused features.

[0028] A second aspect of the present invention provides a robotic arm grasping system based on semantic constraints and multimodal fusion, comprising:

[0029] The multimodal perception module is used to acquire visual information of the scene to be captured and the user's voice commands, identify and locate target objects in the scene based on the visual information, and recognize and semantically encode the voice commands to obtain text semantic features that represent the user's intention.

[0030] The multimodal fusion module is used to fuse the semantic features of the text with the information of the target object, predict the grasping parameters based on the fused features, and generate multiple candidate grasping points.

[0031] The semantic constraint module is used to use the semantic features of the text as semantic constraints, perform semantic consistency evaluation on the multiple candidate crawling points, filter out the target crawling points, and generate crawling control instructions.

[0032] The grasping execution module is used to control the robotic arm to perform grasping operations according to the grasping control instructions.

[0033] A third aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the robotic arm grasping method based on semantic constraints and multimodal fusion as described in the first aspect of the present invention.

[0034] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the robotic arm grasping method based on semantic constraints and multimodal fusion as described in the first aspect of the present invention.

[0035] A fifth aspect of the present invention provides a computer program product comprising software code, wherein the program in the software code performs the steps of the robotic arm grasping method based on semantic constraints and multimodal fusion as described in the first aspect of the present invention.

[0036] Compared with existing technologies, the robotic arm grasping method and system based on semantic constraints and multimodal fusion provided by this invention has the following beneficial effects:

[0037] (1) This invention uses text semantic features as semantic constraints to evaluate and screen candidate crawling points. By calculating the correlation between local visual features and global semantic instructions, and combining the crawling quality to make joint decisions, the selection of crawling points not only satisfies geometric crawlability, but also is highly consistent with the user's semantic intent, thereby realizing semantically driven accurate crawling and improving the accuracy and intelligence of human-computer interaction.

[0038] (2) The backbone network provided by the present invention adopts an improved lightweight MobileNetv4-I structure and embeds a DWGConv module. It replaces the traditional standard convolution with an efficient depthwise separable convolution and feature reuse structure. While significantly reducing model parameters and computational complexity, it ensures the effective extraction of basic features and improves the performance of the model when deployed on resource-constrained devices.

[0039] (3) The neck network provided by the present invention adopts the SDII-BiFPN structure and introduces the RECSA attention module. Through bidirectional multi-scale feature fusion and channel-space attention joint calibration, it enhances the transmission of detailed information of small targets and suppresses background redundant noise, thereby improving the detection rate of small objects and the detection robustness in cluttered environments.

[0040] (4) The detection head of the present invention adopts a Dynamic Head, which integrates the dynamic attention mechanism of scale, space and task perception, enabling the model to adaptively focus on key features of different scales and spatial locations, and dynamically allocate weights according to classification and localization tasks, ultimately achieving accurate and stable detection of objects with a wide scale range, from large-sized packaging boxes to small-scale parts or daily necessities. Attached Figure Description

[0041] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0042] Figure 1 The flowchart shows the robotic arm grasping method based on semantic constraints and multimodal fusion provided in Embodiment 1 of the present invention.

[0043] Figure 2 This is a schematic diagram of the MSBD-YOLO model architecture provided in Embodiment 1 of the present invention;

[0044] Figure 3 This is a schematic diagram of the MobileNetv4-I module architecture provided in Embodiment 1 of the present invention;

[0045] Figure 4 This is a schematic diagram of the DWGConv module architecture provided in Embodiment 1 of the present invention;

[0046] Figure 5 This is a schematic diagram of the RECSA module architecture provided in Embodiment 1 of the present invention;

[0047] Figure 6 This is a schematic diagram of the SDII-BiFPN module architecture provided in Embodiment 1 of the present invention;

[0048] Figure 7 This is a schematic diagram of the Dynamic Head module architecture provided in Embodiment 1 of the present invention;

[0049] Figure 8 This is a comparison chart of the detection before and after the model improvement provided in Embodiment 1 of the present invention;

[0050] Figure 9 This is a schematic diagram of the data flow during the execution of a grasping task by a robotic arm, provided in Embodiment 1 of the present invention.

[0051] Figure 10 This is an architecture diagram of the robotic arm grasping system based on semantic constraints and multimodal fusion provided in Embodiment 2 of the present invention. Detailed Implementation

[0052] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0053] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0054] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0055] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0056] Example 1

[0057] like Figure 1 This embodiment provides a robotic arm grasping method based on semantic constraints and multimodal fusion, including:

[0058] S1. Obtain visual information of the scene to be captured and the user's voice command, identify and locate the target object in the scene based on the visual information, and identify and semantically encode the voice command to obtain text semantic features that represent the user's intention.

[0059] S2. The text semantic features are fused with the information of the target object, and the crawling parameters are predicted based on the fused features to generate multiple candidate crawling points.

[0060] S3. Using the text semantic features as semantic constraints, perform semantic consistency evaluation on the multiple candidate crawling points, filter out the target crawling points, and generate crawling control instructions.

[0061] S4. Control the robotic arm to perform a grasping operation according to the grasping control command.

[0062] The robotic arm grasping method provided in this embodiment first acquires scene visual information and user voice commands simultaneously through multimodal perception, providing a foundation for subsequent fusion. Then, it uses a lightweight improved detection model (MSBD-YOLO) to accurately locate all potential target objects in the scene, providing candidate objects for grasping. Next, it deeply fuses textual semantic features representing the user's abstract intent with specific visual features, making the grasping parameter prediction process semantically guided. Most importantly, after generating multiple geometrically feasible grasping points, it does not simply select the highest quality point, but again introduces textual semantic features as a screening criterion to evaluate the semantic consistency of each candidate point, ensuring that the finally selected target grasping point is not only mechanically graspable but also semantically consistent with the user's commands. This series of interconnected steps fundamentally changes the limitations of traditional grasping systems that "only use their hands, not their brains," achieving a leap from "grabbing what they see" to "grabbing what they understand the commands," significantly improving the success rate of grasping tasks and user satisfaction.

[0063] Specifically, semantic consistency evaluation is performed on the multiple candidate crawling points, including:

[0064] Based on the prediction results of the crawling parameters, the crawling quality parameters corresponding to each candidate crawling point are obtained;

[0065] From the fused features, regional features centered on each candidate grasping point are extracted as corresponding local visual features.

[0066] Calculate the semantic correlation between the text semantic features and each local visual feature to obtain the semantic consistency score of each candidate crawling point;

[0067] The target crawl point is determined based on the crawling quality parameters and semantic consistency score of each candidate crawl point.

[0068] Grabbing quality parameters (such as gripping width and angle confidence) reflect the geometric graspability of a point, but cannot distinguish whether the grasped item is a "red pillbox" or a "blue pill bottle". By extracting local visual features centered on candidate points and calculating their similarity (e.g., cosine similarity) with global textual semantic features (such as vector representations of "red" and "pillbox"), the degree of matching between the visual content and semantic instructions in that local area can be quantified, i.e., semantic consistency score. Finally, the two are combined for decision-making, for example, by weighted summation, to select the point with the highest overall score. The technical effect of this mechanism is that it transforms abstract semantic instructions into quantifiable evaluation indicators that can be optimized together with geometric grasping quality. This makes the grasping decision no longer a simple geometrically optimal solution, but rather the best balance between semantic intent and physical feasibility, effectively solving the core problem of instruction-based grasping.

[0069] Specifically, the determination of the target capture point is achieved through the following weighted decision function:

[0070] ;

[0071] in, For the index of the target crawl point, This indicates the capture quality parameters; Indicates semantic consistency score; Weighting coefficients for quality parameters This is a weighting coefficient for semantic consistency, used to balance geometric catchability and semantic matching degree.

[0072] The coefficients α and β can be set or learned according to the actual application scenario. For example, in scenarios requiring absolute safety and prioritizing grip stability, α can be increased; in interactive scenarios with extremely high requirements for instruction compliance, β can be increased. The technical effect of this function is that it incorporates the abstract concept of semantic matching degree into the decision-making closed loop of robot gripping control, enabling the system to autonomously weigh and optimize between "stable gripping" and "correct gripping," thereby enhancing the system's adaptability and controllability under different task requirements.

[0073] Specifically, the target objects in the scene are identified and located based on the visual information using a pre-trained MSBD-YOLO detection model. The backbone network of the MSBD-YOLO detection model adopts a lightweight MobileNetv4 network structure, and its last ordinary convolutional layer is replaced with a lightweight convolutional module DWGConv, denoted as MobileNetv4-I. The neck network of the MSBD-YOLO detection model adopts an SDII-BiFPN structure, and a RECSA attention module is introduced into it. The detection head of the MSBD-YOLO detection model adopts a Dynamic Head structure.

[0074] The pre-trained MSBD-YOLO detection model ensures basic efficiency through a lightweight backbone, improves the quality of multi-scale feature fusion and anti-interference ability through an enhanced neck network, and achieves adaptive and accurate prediction through a dynamic detection head. Together, these features enable the MSBD-YOLO model to quickly and accurately detect and locate various targets (especially small targets) in a scene on a resource-constrained robot platform, providing a reliable visual perception foundation for subsequent semantic fusion and grasping decisions.

[0075] Specifically, the DWGConv module embedded in the backbone network performs the following processing:

[0076] A DepthwiseConv operation is performed on the input features to obtain initial features containing basic contour information. In this operation, an independent convolution kernel is used to perform convolution operations for each channel of the input features. Through this channel-wise convolution processing, each channel interacts with the corresponding convolution kernel and extracts features based on its own characteristics. The resulting initial features are composed of a series of feature maps that have undergone independent convolution transformations. These feature maps are consistent in number with the number of channels in the input features, and each retains the local feature information of the corresponding input channel after depthwise convolution, forming a feature set with specific spatial distribution and local feature patterns.

[0077] The initial features are subjected to a GhostConv operation to generate detailed features containing detailed texture information. The GhostConv operation first processes the initial features using ordinary convolution operations to generate some basic feature maps. These basic feature maps capture some of the main feature information in the initial features. Subsequently, a low-cost linear transformation operation is performed on these basic feature maps, leveraging the flexibility of linear transformation to derive more Ghost feature maps from the basic feature maps. The detailed features generated through this series of operations not only cover the main feature information retained after ordinary convolution of the initial features, but also uncover subtle features and complex texture information hidden in the initial features through linear transformation. Overall, the detailed features present a richer and more detailed feature expression than the initial features. The number of feature maps is increased compared to the initial features, and the information relationships between the feature maps are more complex and diverse. They can reflect the detailed feature patterns in the initial features from multiple dimensions and levels, providing a more comprehensive and detailed feature foundation for subsequent feature fusion and information processing.

[0078] The initial feature is concatenated with the detailed feature;

[0079] The concatenated features are subjected to convolution, batch normalization, and ReLU activation to obtain the final output features of the lightweight convolution module DWGConv.

[0080] The DWGConv module employs a two-stage feature generation and fusion strategy. DepthwiseConv captures the global contour and spatial structure of the input features at extremely low computational cost, forming skeleton information (initial features). Subsequently, GhostConv performs a linear transformation on the skeleton, generating a series of features that supplement texture, edges, and other details. Finally, the two are concatenated and unified, achieving complementarity between the skeleton and details. This structure, through the idea of ​​feature reuse, generates rich and multi-scale feature representations with far fewer parameters than standard convolutions. Its technical advantage lies in significantly reducing the module's computational complexity and number of parameters while maintaining feature expressive power, directly contributing to the model's lightweight design.

[0081] Specifically, the lightweight MobileNetv4-I architecture adopted by the backbone network includes:

[0082] Initial convolutional layers and shallow modules are used to extract low-level features;

[0083] Intermediate feature module, used to dynamically adjust channel dimensions;

[0084] The core feature extraction module consists of stacked general inverted bottleneck UIB blocks, which sequentially include 1×1 pointwise convolution for dimensionality upscaling, 3×3 depthwise separable convolution, dynamic ReLU activation, and 1×1 pointwise convolution for dimensionality downscaling, and introduce SE channel attention in the skip connection branch;

[0085] The tail feature adjustment module is used to generate high-dimensional feature representations.

[0086] MobileNetv4-I is an improved architecture based on MobileNetv4. Its lightweight nature stems from its core component—the Universal Inverted Bottleneck (UIB) block—and the replacement of the original ordinary convolutional layers with DWGConv modules at the end of the network. The UIB block adopts a structure of dimensionality increase -> depthwise convolution -> nonlinear activation -> dimensionality reduction. Depthwise separable convolution decomposes standard convolution into depthwise convolution and pointwise convolution, which is key to reducing computational cost. Dynamic ReLU enhances the adaptability of nonlinear expressive power, and the SE attention mechanism in skip connections achieves adaptive calibration between channels, increasing the weight of important features. Through this carefully designed block-structure stacking, the entire network maintains low computational density throughout the process of extracting and abstracting features layer by layer. Its technical effect is to provide an extremely efficient and expressive feature extraction foundation for the entire detection model, which is the core pillar enabling the model to achieve lightweight design.

[0087] Specifically, the RECSA attention module performs the following processing:

[0088] The input features are convolutional, batch normalized, and ReLU activated to obtain the feature map;

[0089] The feature map is simultaneously input into both the channel attention branch and the spatial attention branch;

[0090] In the channel attention branch, global average pooling is performed on the feature map, and channel attention weights are generated by adaptive 1-D convolution and Sigmoid activation.

[0091] In the spatial attention branch, the feature map is subjected to 3×3 Depthwise convolution and Sigmoid activation to generate spatial attention weights;

[0092] After multiplying the channel attention weights and spatial attention weights element by element, we obtain attention features that fuse channel and spatial information. Then, after convolution, batch normalization, and ReLU activation, the processed features are added to the original input features through residual connections to obtain the enhanced features.

[0093] The RECSA module simulates the attention mechanism of human vision, focusing on key information in two steps. First, the channel attention branch learns the importance of each channel by compressing and activating feature channels, allowing the model to focus more on feature channels containing target object information and suppress background or noise channels. Second, the spatial attention branch scans the feature map with a small convolutional kernel, learning the importance of each spatial location, thus focusing on the area where the target object is likely to appear (such as the center of a medicine box) and ignoring irrelevant background areas. Finally, residual connections are used to add the attention-enhanced features to the original features, amplifying the key signal while retaining all original information, avoiding information loss that may occur due to the attention mechanism. The technical effect is a significant improvement in the representation quality of model features, enabling subsequent feature fusion and detection heads to make decisions based on more prominent features, effectively reducing false positives and false negatives.

[0094] Specifically, the SDII-BiFPN structure in the neck network performs the following processing:

[0095] Construct a feature pyramid network that includes both top-down and bottom-up bidirectional paths;

[0096] An SDII module is embedded at the fusion node of the bidirectional path;

[0097] In the SDII module, the GSConv module is first applied to the original features of each level output by the encoder. After channel dimensionality reduction, the original features of each level are scale-aligned based on the target level features. Then, 3×3 convolution is used for smoothing and denoising. Finally, all the aligned original features of each level are non-linearly fused through element-wise Hadamard product to obtain the fused multi-scale features.

[0098] Output the fused multi-scale features to the detection head.

[0099] BiFPN provides a bidirectional information flow path, allowing high-level semantic features (facilitating the identification of large objects) to propagate downwards to enhance the semantics of low-level features, while low-level detail features (facilitating the localization of small objects) propagate upwards to supplement the details of high-level features. At the traditional weighted summation fusion node, this invention embeds an SDII module. The SDII module is an improved version of the SDI module. The GSConv module (replacing the spatial attention and channel attention of the original SDI module) is used for the following improvement:

[0100] 1. By using the dual-branch design of standard convolution, depthwise separable convolution and channel shuffling in GSConv, the feature association across channels and space is strengthened, so that the fused features have both richer semantic information and more accurate detailed representation.

[0101] 2. Leveraging GSConv's lightweight features, the number of module parameters and GPU memory usage are significantly reduced, further compressing model computation and achieving module lightweighting.

[0102] Specifically, the detection head employs a Dynamic Head structure comprising, in series:

[0103] A scale-aware attention module is used to adaptively adjust the feature focusing range through average pooling, 1×1 convolution and Hard Sigmoid activation;

[0104] The spatially aware attention module is used to learn spatial offsets and weights through deformable convolutions, capture spatial dependencies, and suppress irrelevant regions.

[0105] The task-aware attention module is used to dynamically optimize the weight allocation of feature channels according to different task objectives through a dynamic activation mechanism consisting of a globally average pooling layer, a fully connected layer, a normalization layer, and an offset sigmoid function connected in sequence.

[0106] Dynamic Head applies an attention mechanism to the detection head itself, enabling dynamic optimization during the prediction phase. The scale-aware module aggregates and reweights features at different scales before they enter the detection head, allowing the head to adaptively focus on the feature layers most relevant to the current target scale. The spatial awareness module uses deformable convolutions, allowing the sampling points of the convolution kernels to adaptively shift according to the feature content, thus more accurately fitting target objects of arbitrary shapes and focusing on the main target region. The task-aware module dynamically adjusts the activation threshold of each feature channel based on whether the final task is classification (determining what the object is) or regression (locating where the object is), achieving task-specific optimization of features. These three modules are interconnected, achieving comprehensive dynamic perception and adaptation from feature scale and spatial location to the task target. This significantly improves the detection head's discrimination ability and localization accuracy in complex scenes (such as occlusion and deformation), resulting in more accurate and stable final detection boxes.

[0107] Specifically, fusing the text semantic features with the information of the target object includes:

[0108] The text semantic features are spatially expanded to generate a semantic feature tensor with the same spatial resolution as the visual features.

[0109] The semantic feature tensor is concatenated with the visual features containing the target object information at the pixel level to obtain the concatenated fused feature tensor.

[0110] Perform a convolution operation on the spliced ​​fused feature tensor to generate the fused features.

[0111] The textual semantic features are initially a global vector (e.g., representing "red medicine box"), while the visual features are feature maps with spatial dimensions (H×W). Through spatial expansion (e.g., duplication), the global semantic vector is expanded into a semantic feature tensor of the same size as the visual feature map, ensuring that each pixel location is associated with the same semantic information. Subsequent pixel-level concatenation and convolutional fusion effectively allow each visual feature location to see and integrate the global semantic instructions. The technical advantage of this mechanism lies in achieving spatial alignment and deep fusion of cross-modal information. This enables the subsequent grasping parameter prediction network to simultaneously perceive "what's here" (visual) and "what the user wants" (semantic), thus predicting grasping points that naturally tend towards the target region described by the semantic instructions, providing a solid feature foundation for semantic constraint decisions.

[0112] In one specific embodiment, the training steps of the detection model provided by this method include:

[0113] Step 1: Build a dataset by collecting or downloading an image dataset containing various objects (such as the pharmaceutical dataset SAP_BABA) and preprocessing the dataset to obtain a dataset for training the model.

[0114] Step 2: Design a lightweight convolutional module DWGConv. The module adopts a structure that combines DepthwiseConv and GhostConv. This structure significantly compresses the number of parameters while improving the receptive field coverage through feature reuse and multi-path fusion, thus meeting the needs of efficient feature extraction for multi-scale objects.

[0115] Step 3: Design the RECSA attention module (residual high-efficiency channel and spatial attention module). The module achieves channel-space joint calibration in a lightweight manner, and the residual structure takes into account both feature enhancement and information preservation.

[0116] Step 4: Construct an improved MSBD-YOLO detection model, which consists of a backbone network, a neck network, and a head. The backbone network uses an improved lightweight MobileNetv4-I structure to replace the original YOLOv11 backbone network. This structure replaces the last ordinary convolutional layer of MobileNetv4 with the DWGConv module designed in Step 2 to further reduce computational complexity. The neck network uses an SDII-BiFPN structure and introduces the RECSA attention module designed in Step 3 to improve the targeting and effectiveness of feature fusion. The head uses a Dynamic Head to replace the original YOLOv11 head, enhancing the robustness of detection for small targets and complex backgrounds.

[0117] Furthermore, such as Figure 2 The diagram shows the network structure of MSBD-YOLO. This diagram illustrates the overall structure of a service robot perception and grasping network using a Backbone–Neck–Head architecture. The Backbone part extracts basic features step-by-step through multiple layers of ConvBN, and introduces modules such as UIB and DWGConv to enhance feature representation while ensuring computational efficiency. Subsequently, SPPF expands the receptive field, and the C2PSA module further strengthens channel and spatial attention, thereby obtaining multi-scale, highly semantic feature representations. The Neck part employs a top-down and bottom-up multi-scale feature fusion strategy, using structures such as Upsample, C3K2, and Conv to achieve feature alignment and fusion. At key fusion nodes, SDII and RECSA modules are introduced to strengthen the interaction modeling and contextual awareness between features of different scales, improving adaptability to complex scenes and changes in target scale. The Head part introduces Dynamic Heads on features of multiple scales to achieve adaptive feature weighting and prediction, thereby improving the robustness and accuracy of target detection and grasping tasks in complex environments.

[0118] Specifically, BackBone is responsible for extracting basic features from the input image, and its structure is as follows:

[0119] 1. Input: The input image, which serves as the starting point for the model.

[0120] 2. ConvBN: A combination of multiple consecutive convolutional layers (Conv) and batch normalization layers (BN) used for initial image feature extraction.

[0121] 3. UIB: General Inverted Bottleneck Module, which achieves flexible feature extraction through multi-path design, balancing model performance and efficiency.

[0122] 4. DWG Conv: A depthwise separable convolution module that combines efficient convolution methods such as Ghost convolution to reduce computation while maintaining feature extraction capabilities.

[0123] 5. SPPF: Spatial Pyramid Pooling Fast Module, which extracts multi-scale features through pooling operations at different scales.

[0124] 6. C2PSA: A composite feature extraction module that integrates the CSP cross-stage structure and PSA attention. Through a collaborative attention mechanism of cross-channel interaction and multi-scale spatial location awareness, it effectively enhances the model's ability to model key regions and important feature channels.

[0125] The Neck network is responsible for further fusing and refining the features extracted from the Backbone for object detection in the Head section. Its structure is as follows:

[0126] 1. Conv: Convolutional layer, used to adjust the number of channels or size of feature maps.

[0127] 2. RECSA: An improved attention module that combines channel and spatial attention to enhance the model's focus on important features.

[0128] 3. C3K2: A lightweight feature extraction module in the backbone network of the YOLO11 series models, used to efficiently extract high-quality, multi-scale effective features from feature maps.

[0129] 4. SDII: Semantic and Detail Injection Module, which combines cross-level feature interaction operations and element-wise Hadamard product to inject high-level semantic information and low-level detail information into the target level features, thereby enhancing the feature fusion effect.

[0130] 5. Upsample: An upsampling layer used to increase the size of the feature map in order to fuse it with higher resolution features from the Backbone.

[0131] The Head (head network) is responsible for object detection based on the features provided by the Neck and outputs the detection results. Its structure is as follows:

[0132] 1. Dynamic-head: The dynamic head module dynamically adjusts the parameters or structure of the detection head based on the input features to improve detection accuracy and robustness.

[0133] 2. Conv: Convolutional layer, used to adjust the number of channels in the feature map to meet the input requirements of the detection head.

[0134] Furthermore, such as Figure 3The image shows the MobileNetv4-I module architecture. The MobileNetv4-I module architecture employs a layer-by-layer progressive feature extraction strategy, striking a balance between computational efficiency and robust feature representation, making it suitable for resource-constrained applications. The network architecture begins with an initial convolutional layer, using a 3×3 kernel and a stride of 3 for downsampling, expanding the channel dimension to 32. Subsequently, a lightweight shallow module consisting of two convolutional layers extracts and enhances low-level features while minimizing computational overhead. The intermediate feature module dynamically adjusts the channel dimension from 32 to 96, then compresses it to 64, optimizing feature quality and parameter efficiency. In the core of the network, stacked UIB blocks in the deep feature module enhance high-order semantic modeling capabilities through depthwise separable convolutions and skip connections. Finally, the tail feature adjustment module expands the channels from 128 to 960, and then further to 1280 through two convolutional layers, generating high-dimensional feature representations for downstream classification or detection tasks.

[0135] Specifically, the overall structure of MobileNetv4-I consists of six modules, starting with the input image (Img) and extracting features step by step in six layers:

[0136] 1. Input layer (Img): The input is the original image, which serves as the starting point of the network.

[0137] 2. Layer 0 (ConvBN×1): Contains a standard 2D convolution (Conv2d), followed by a batch normalization (BN) layer and a ReLU activation function. This layer is used for initial image feature extraction.

[0138] 3. Layer 1 (ConvBN×2): Consists of two consecutive ConvBN modules, each containing a convolutional layer, a batch normalization layer, and a ReLU activation function. This layer further extracts and refines features.

[0139] 4. Layer 2 (ConvBN×2): Similar to Layer 1, it consists of two consecutive ConvBN modules, which continue to extract and refine the features.

[0140] 5. Layer 3 (UIB×6): Contains six UIB modules, each of which is a general inverted bottleneck structure containing components such as depthwise convolution, pointwise convolution, batch normalization, and ReLU activation function. The UIB modules are designed to balance the model's performance and efficiency, and are optimized through Neural Architecture Search (NAS).

[0141] 6. Layer 4 (UIB×6): Similar to Layer 3, it also contains six UIB modules for further extraction and transformation of features.

[0142] 7. Layer 5 (DWGConv×2): Contains two DWGConv modules. Each module includes operations such as depthwise convolution, Ghost convolution, feature concatenation (Concat), standard convolution (Conv2d), batch normalization, and ReLU activation function. This layer is used to further reduce computational cost while performing final feature extraction and fusion.

[0143] The ConvBN module, as the basic computational unit of the network, consists of a series of standard convolutional layers (Conv), batch normalization layers (BN), and ReLU activation functions. The UIB module structure is an inverted bottleneck block composed of depthwise convolution, batch normalization, pointwise convolution, and ReLU activation functions.

[0144] In summary, this invention replaces the YOLOv11 backbone network with a MobileNetv4 structure, retaining its Universal Inverted Bottleneck (UIB) and Moving Query Attention (MQA) core modules to ensure a balance between feature extraction and lightweight design. To further reduce the number of parameters, the last ordinary convolutional layer of MobileNetv4 is replaced with a custom DWGConv module, forming an improved backbone network structure, namely MobileNetv4-I. An SPPF module and a C2PSA module are sequentially connected in series at the DWGConv output. The scale of the MobileNetv4-I output feature map is adjusted to perfectly match the input dimension of the subsequent SDII-BiFPN neckline, avoiding feature dimension gaps.

[0145] Furthermore, such as Figure 4 The diagram shows the architecture of the DWGConv module. DWGConv consists of a series of interconnected components: DepthwiseConv, GhostConv, a connection layer, a 1×1 convolution, batch normalization, and an activation function. The module first uses DepthwiseConv for low-computation initial feature extraction, maintaining the depth dimension unchanged during computation and only altering the height and width parameters. Then, GhostConv, supplemented by residual connections, reuses the basic features output by DepthwiseConv, avoiding detail loss and generating detailed features through inexpensive linear transformations. Subsequently, feature fusion and unification are performed, concatenating the GhostConv output features and residual features using Concat, followed by 1×1 convolution to compress the channels to the target dimension. Batch normalization and ReLU activation enhance feature stability. This structure combines lightweight design with rich feature content, adapting to multi-scale object detection needs.

[0146] Furthermore, such as Figure 5The diagram illustrates the RECSA attention module architecture of this invention. The RECSA module consists of a channel attention branch, a spatial attention branch, and a residual connection. The specific process is as follows: Input features are first extracted using a CBR (convolution, batch normalization, and ReLU) module, and then processed in two paths. One path is the original feature branch; the other path performs the following operations: it is first divided into two branches. One branch sequentially connects to a global average pooling (GAP), an adaptive 1-D convolution, and a sigmoid activation function to generate channel attention weights; the other branch sequentially connects to a depthwise convolutional layer and a sigmoid activation function to generate spatial attention weights. Subsequently, the outputs of the two branches are multiplied element-wise to obtain attention features that fuse channel and spatial information. These are then further processed by the CBR module, and finally, the fused attention features are residually connected to the original feature branch (element-wise addition), and then activated by ReLU before outputting. This design enhances feature expressiveness and alleviates the gradient vanishing problem in deep network training, allowing the model to further improve performance with deeper network structures.

[0147] Furthermore, such as Figure 6 The diagram shows the SDII-BiFPN module architecture of this invention. BiFPN (Bidirectional Feature Pyramid Network) builds upon the traditional Feature Pyramid Network (FPN) by enabling bidirectional interaction of multi-scale features through top-down and bottom-up bidirectional paths. It also introduces learnable weight parameters to adaptively adjust the importance of different input features. Structurally, it further optimizes the system by removing single-input nodes, adding cross-layer shortcut connections, and repeatedly stacking bidirectional paths, thereby reducing computational redundancy while enhancing feature fusion performance.

[0148] The specific process is as follows: perform cross-scale aggregation on the input P3, P4, and P5 features:

[0149] P5 branch: P5 first undergoes Conv to adjust its dimensions, then Upsamples it to the size of P4, and merges it with the features of P4;

[0150] P4 branch: P4 first fuses the features after Conv and upsampling with P5, then sequentially connects SDII→C3K2→RECSA to extract features, and then upsamples again to the size of P3 and fuses with the features of P3;

[0151] P3 branch: P3 is fused with the features upsampled from P4 via Conv→SDII→C3K2→RECSA→Conv, and then sequentially connected to SDII→C3K2→RECSAE→Conv→SDII→C3K2→RECSA to finally output the fused multi-scale features.

[0152] This design not only enhances the recognition of small objects and effectively reduces the interference of redundant background information, but also preserves rich semantic information in high-level features, thereby improving the detection accuracy of large targets. Furthermore, BiFPN can strengthen the spatial location information of low-level features, a feature also applicable to solving the detection challenge of objects occluding each other. BiFPN not only possesses bidirectional fusion paths from top to bottom and bottom to top, but also supports the interactive transfer of feature information within the same level, making feature fusion more comprehensive. The feature fusion formula of BiFPN is as follows:

[0153]

[0154] The above formula is the fourth layer's top-down fusion formula. This represents the original input features of the 4th layer of BiFPN. The original input features are from the 5th layer. For resolution matching operations, here is... Upsampling is performed to make its resolution consistent with... Consistent; , These are learnable weight parameters used for adaptive learning. and Contribution to the fusion results; =0.0001 is the minimum value to avoid numerical instability caused by a denominator of 0. This is an intermediate feature after the fourth layer is fused from top to bottom to ensure the stability of the value.

[0155]

[0156] The above formula is the fourth layer's bottom-up fusion formula. This is the final output feature after complete bidirectional fusion of the third layer. This is for Downsampling is performed to match the resolution of layer 4; w1′, w2′, and w3′ are newly added learnable weights, corresponding to... , and The importance of; This is the final output feature after complete bidirectional fusion of the 4th layer.

[0157] Furthermore, in step two, the SDII-BiFPN feature fusion network integrates the improved multi-level feature fusion module SDII into the BiFPN network.

[0158] Furthermore, in SDII-BiFPN, an SDII module is embedded in the BiFPN fusion node to enhance the expressive power of multi-scale feature fusion. The SDII module first applies GSConv to the original multi-scale features output by the encoder, refining the features through standard convolutional branches (capturing cross-channel semantic associations), depthwise separable convolutional branches (extracting spatial details), and channel shuffling (uniformly spreading information), achieving feature enhancement and noise suppression with low computational cost. Then, a 1×1 convolution is used to unify the number of feature channels at each level, reducing subsequent fusion costs. Next, adaptive scale alignment is performed: based on the resolution of the target level features, high-level features are upsampled using bilinear interpolation to maintain edge continuity, low-level features are downsampled using adaptive average pooling to suppress noise, and mid-level features maintain their original resolution to reduce redundancy, ensuring consistent resolution across all levels. Subsequently, the aligned features are smoothed and denoised using 3×3 SmoothConv, and then nonlinearly fused using element-wise Hadamard product. This fusion method enhances cross-scale feature interaction, allowing the target-level features to simultaneously possess high-level semantics and low-level details. The core advantage of this design is that it preserves cross-level semantics and complementary details while achieving lightweight design and avoiding redundant computation.

[0159] SDII and BiFPN work collaboratively through core node embedding: The SDII module and BiFPN are fused through core node embedding. BiFPN provides bidirectional cross-scale feature transfer paths from top to bottom and bottom to top, ensuring the effective flow of features at different levels. The SDII module replaces the simple weighting or concatenation operations of BiFPN, playing a role at the fusion node of each transfer path, enhancing the complementarity of semantic and detailed information by integrating multi-scale input features. Finally, the P3, P4, and P5 features enhanced by SDII are fed into the detection head, forming a neck structure that maintains the efficiency of BiFPN and improves the quality of SDII, significantly improving the detection accuracy of small objects in complex scenes.

[0160] Furthermore, such as Figure 7 The diagram shows the structure of the Dynamic Head module. This model integrates three complementary attention mechanisms. The diagram illustrates the application flow of the attention mechanisms in the dynamic head, starting from the input features and sequentially passing through scale-aware attention (π). L Spatial awareness attention (π) S ) and task-aware attention (π) C The process involves processing the data to ultimately output attention-adjusted features. Specifically:

[0161] Scale-aware attention (π) L Located on the left side of the diagram, it consists of the following components:

[0162] 1) avg pool: Average pooling layer, used to downsample the input features, reduce the size of the feature map, and retain the main feature information.

[0163] 2) conv 1x1: 1x1 convolutional layer, used to adjust the number of channels of the feature and perform linear combination between feature channels.

[0164] 3) ReLU: Activation function layer, introduces non-linearity, and enhances the expressive power of the model.

[0165] 4) Hard sigmoid: An activation function that restricts the output to between 0 and 1, used to generate attention weights. Its output is multiplied with features from other parts to adjust the strength of the features.

[0166] Spatial awareness attention (π) S It contains the following components:

[0167] 1) index: may be used to generate an index of the sampling position, providing positional information for subsequent deformable convolutions.

[0168] 2) conv 3x3: 3x3 convolutional layer, used to extract local spatial features.

[0169] 3) sigmoid: an activation function that generates attention weights for spatial locations, used to adjust the feature importance of different spatial locations.

[0170] 4) offset: The offset generation part combines the weights output by sigmoid to generate offsets for deformable convolution, enabling the convolution to focus on more important spatial regions.

[0171] Task-aware attention (π) C The structure is as follows:

[0172] 1) avg pool: The average pooling layer performs global average pooling on the input features, compressing the spatial dimension into a single vector.

[0173] 2) fc: Fully connected layer, used to perform linear transformation on the pooled features and learn the complex relationships between features.

[0174] 3) ReLU: Activation function that adds non-linearity.

[0175] 4) Normalize: The normalization layer may be used to normalize the output to a specific range, such as [-1, 1]. Its output is added to the input features (represented by the plus sign inside the circle in the figure), and combined with the parameters α1, β1, α2, β2 learned from the fully connected layer, it is used to dynamically adjust the activation state of the feature channels to adapt to different task requirements.

[0176] The scale-aware attention module addresses multi-scale feature differences by adaptively adjusting the focus range to enhance feature representation; the spatial-aware attention module captures spatial dependencies by analyzing location associations while suppressing interference from irrelevant regions; and the task-aware attention module dynamically optimizes feature weight allocation based on differences in task objectives to improve multi-task collaboration.

[0177] Furthermore, π L π S and π C These are scale-aware attention modules, spatial-aware attention modules, and task-aware attention modules, which are connected in series to form a single Dynamic Head module. After the feature map is input into the Dynamic Head module, it is first processed at π... L The process involves average pooling, followed by 1×1 convolution, activation using the Hard Sigmoid activation function, and finally multiplying the resulting tensor with the input features to achieve scale awareness. This process then proceeds to the π... S First, a 3×3 convolution operation is performed to obtain the offset of the feature map and its offset weights. DCNv2 is used to learn sparsity, thereby completing spatial perception. Then, a dynamic linear rectifier function is used to adjust the channel values ​​of the feature map according to different tasks, thereby completing task perception.

[0178] For small object detection, by strengthening the dynamic correlation between low-level spatial features and high-level semantic features, the problem of indistinct features and easy missed detection of small objects is effectively solved. At the same time, it can adaptively adjust the weight of features at different scales, taking into account the detection needs of both large and small objects, and fully adapting to the characteristics of different object sizes in various application scenarios (such as different sizes of medicines in nursing home scenarios).

[0179] Furthermore, such as Figure 8 The figure shows a comparison of the detection performance of the MSBD-YOLO model and the YOLOv11 model before and after improvement, as proposed in this embodiment. The first column represents the detection output of the YOLOv11 model, and the second column represents the detection output of the MSBD-YOLO model. The bounding boxes in the figure indicate the detected drug categories and their confidence scores, with different colors representing different categories. As can be seen from the figure, compared to the YOLOv11 model on the left (which has lower confidence scores in some areas), the MSBD-YOLO model on the right has a higher and more stable overall confidence score. Therefore, the MSBD-YOLO model has superior predictive performance and higher detection accuracy, making it more suitable for drug detection tasks in elderly care scenarios.

[0180] In one specific embodiment, such as Figure 9 The robotic arm grasping method includes the following steps:

[0181] S1. Obtain visual information of the scene to be captured and the user's voice command. Identify and locate the target object in the scene based on the visual information, and recognize and semantically encode the voice command to obtain text semantic features that represent the user's intention.

[0182] Specifically, step S1 includes the following sub-steps:

[0183] S1-1, Speech-to-Text: A lightweight speech recognition model is used to recognize input speech commands and convert them into text commands to improve system real-time performance and response speed.

[0184] S1-2, Text Normalization Processing: Semantic cleaning and standardization are performed on the identified text, including filtering redundant words, standardizing object names and instructions, extracting core semantic information related to the crawling task, and generating standardized text instructions.

[0185] S1-3, Text Semantic Encoding: Normalized text instructions are input into a text encoding network for semantic encoding, generating text semantic feature vectors. The text encoding network maps text into a vector sequence through an embedding layer, then performs contextual semantic modeling through an attention mechanism, and finally obtains the overall semantic representation through a feature aggregation layer. The specific implementation is as follows:

[0186] Let the normalized text instructions be: The text instructions are mapped into a vector sequence through an embedding layer: ,in, Then, contextual semantic features are calculated using a self-attention mechanism:

[0187] ;

[0188] in, Finally, the text semantic feature vector is obtained by average pooling aggregation:

[0189] ;

[0190] S1-4, Semantic Feature Mapping and Spatial Expansion: [The following text appears to be incomplete and requires further context:] Perform a linear mapping to obtain Then, it is expanded into a semantic feature tensor with the same spatial resolution as the visual feature map by copying. .

[0191] S2. The semantic features of the text are fused with the information of the target object, and the crawling parameters are predicted based on the fused features to generate multiple candidate crawling points.

[0192] Specifically, step S2 includes the following sub-steps:

[0193] S2-1, Visual Feature Extraction: Input the RGB-D image into the visual feature extraction network (i.e., the backbone network of MSBD-YOLO) to extract visual feature maps. .

[0194] S2-2, Multimodal Feature Fusion: Integrating semantic feature tensors Visual feature map Pixel-level stitching is performed to obtain a fused feature tensor with a size of [size missing]. Then, channel compression is performed using 1×1 convolution to generate the fused feature representation.

[0195] S2-3, Grabbing Parameter Prediction and Candidate Point Generation: Based on the fused features, the grabbing parameters (grabbing position, angle, and width) are predicted using a grabbing detection network (MSBD-YOLO's detection head), and multiple sets of grabbing candidate points are generated. Each candidate point is represented as These correspond to pixel coordinates, grab angle, grab width, and grab quality parameters, respectively.

[0196] S3. Using the semantic features of the text as semantic constraints, perform semantic consistency evaluation on the multiple candidate crawling points, select the target crawling point, and generate crawling control instructions.

[0197] Specifically, step S3 includes the following sub-steps:

[0198] S3-1, Local visual feature extraction: using each candidate grasping point pixel coordinates Centered on the visual feature map, extract local regions. The features within the image are analyzed, and local visual feature vectors are obtained through pooling operations. :

[0199] ;

[0200] in, The preset region radius is used to cover the visual context near the candidate point being captured;

[0201] Extract the visual features of the region from the visual feature map and perform pooling:

[0202] ;

[0203] in, This represents the pooling operator.

[0204] S3-2, Semantic Consistency Score Calculation: Calculating the semantic features of the text. With local visual features Perform feature space alignment (optionally linear mapping) and calculate semantic consistency scores. A typical method is cosine similarity.

[0205] ;

[0206] in, and These are the aligned feature vectors.

[0207] S3-3, Semantic-Geometric Joint Decision-Making: Combining Crawling Quality Parameters semantic consistency score The target grab point is determined by a weighted decision function:

[0208] ;

[0209] in, For the index of the target crawl point, This indicates the capture quality parameters; Indicates semantic consistency score; Weighting coefficients for quality parameters This is a weighting coefficient for semantic consistency, used to balance geometric catchability and semantic matching degree.

[0210] S4. Control the robotic arm to perform a grasping operation according to the grasping control command.

[0211] Specifically, step S4 includes the following sub-steps:

[0212] S4-1 3D Grab Point Calculation: Based on the pixel coordinates and corresponding depth information of the target grab point in the image, and combined with the camera intrinsic parameters, calculate its 3D coordinates in the camera coordinate system.

[0213] S4-2, Grasping posture generation: based on grasping angle and crawl width Construct the grasping posture information (position, orientation, opening and closing width).

[0214] S4-3, Coordinate System Transformation: By calibrating the coordinate system of the camera and the robot arm base, the grasping posture is transformed to the robot arm base coordinate system.

[0215] S4-4, Motion Planning and Execution: Input the target grasping pose into the robotic arm control module, generate a trajectory through motion planning, control the movement of the robotic arm, and drive the end effector to complete the grasping.

[0216] Example 2

[0217] like Figure 10 As shown, this embodiment provides a robotic arm grasping system based on semantic constraints and multimodal fusion, including:

[0218] The multimodal perception module is used to acquire visual information of the scene to be captured and the user's voice commands, identify and locate target objects in the scene based on the visual information, and recognize and semantically encode the voice commands to obtain text semantic features that represent the user's intention.

[0219] The multimodal fusion module is used to fuse the semantic features of the text with the information of the target object, predict the grasping parameters based on the fused features, and generate multiple candidate grasping points.

[0220] The semantic constraint module is used to use the semantic features of the text as semantic constraints, perform semantic consistency evaluation on the multiple candidate crawling points, filter out the target crawling points, and generate crawling control instructions.

[0221] The grasping execution module is used to control the robotic arm to perform grasping operations according to the grasping control instructions.

[0222] Example 3

[0223] Embodiment 3 of the present invention provides an electronic device.

[0224] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, it implements the steps in the robotic arm grasping method based on semantic constraints and multimodal fusion as described in Embodiment 1 of the present invention.

[0225] The detailed steps are the same as those of the robotic arm grasping method based on semantic constraints and multimodal fusion provided in Example 1, and will not be repeated here.

[0226] Example 4

[0227] Embodiment 4 of the present invention provides a computer-readable storage medium.

[0228] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the robotic arm grasping method based on semantic constraints and multimodal fusion as described in Embodiment 1 of the present invention.

[0229] The detailed steps are the same as those of the robotic arm grasping method based on semantic constraints and multimodal fusion provided in Example 1, and will not be repeated here.

[0230] Example 5

[0231] Embodiment 5 of the present invention provides a computer program product.

[0232] A computer program product includes software code, wherein the program in the software code executes the steps of the robotic arm grasping method based on semantic constraints and multimodal fusion as described in Embodiment 1 of the present invention.

[0233] The detailed steps are the same as those of the robotic arm grasping method based on semantic constraints and multimodal fusion provided in Example 1, and will not be repeated here.

[0234] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0235] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0236] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0237] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0238] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A method for robot arm grasping based on semantic constraints and multi-modal fusion, characterized in that, Includes the following steps: The system acquires visual information of the scene to be captured and the user's voice commands, identifies and locates target objects in the scene based on the visual information, and performs recognition and semantic encoding on the voice commands to obtain text semantic features that represent the user's intent. The text semantic features are fused with the information of the target object, and the crawling parameters are predicted based on the fused features to generate multiple candidate crawling points. The text semantic features are fused with the information of the target object, including: The text semantic features are spatially expanded to generate a semantic feature tensor with the same spatial resolution as the visual features. The semantic feature tensor is concatenated with the visual features containing the target object information at the pixel level to obtain the concatenated fused feature tensor. Perform a convolution operation on the spliced ​​fused feature tensor to generate the fused features; Using the text semantic features as semantic constraints, the semantic consistency of the multiple candidate crawling points is evaluated, the target crawling point is selected, and crawling control instructions are generated. The semantic consistency of the multiple candidate crawl points is evaluated, including: Based on the prediction results of the crawling parameters, the crawling quality parameters corresponding to each candidate crawling point are obtained; From the fused features, regional features centered on each candidate grasping point are extracted as corresponding local visual features. Calculate the semantic correlation between the text semantic features and each local visual feature to obtain the semantic consistency score of each candidate crawling point; The target crawling point is determined based on the crawling quality parameters and semantic consistency score of each candidate crawling point; The determination of the target capture point is achieved through the following weighted decision function: ； in, For the index of the target crawl point, This indicates the capture quality parameters; Indicates semantic consistency score; Weighting coefficients for quality parameters This is a weighting coefficient for semantic consistency, used to balance geometric graspability and semantic matching degree; The robotic arm is controlled to perform a grasping operation according to the grasping control command.

2. The method as described in claim 1, characterized in that, The target objects in the scene are identified and located based on the visual information using a pre-trained MSBD-YOLO detection model. The backbone network of the MSBD-YOLO detection model adopts a lightweight MobileNetv4 network structure, and its last ordinary convolutional layer is replaced with a lightweight convolutional module DWGConv, denoted as MobileNetv4-I. The neck network of the MSBD-YOLO detection model adopts an SDII-BiFPN structure, and a RECSA attention module is introduced into it. The detection head of the MSBD-YOLO detection model adopts a Dynamic Head structure.

3. The method as described in claim 2, characterized in that, The lightweight convolutional module DWGConv performs the following processing steps: Perform the DepthwiseConv operation on the input features to obtain initial features containing basic contour information; Perform a GhostConv operation on the initial features to generate detailed features containing detailed texture information; The initial feature is concatenated with the detailed feature; The concatenated features are subjected to convolution, batch normalization, and ReLU activation to obtain the final output features of the lightweight convolution module DWGConv.

4. A robotic arm grasping system based on semantic constraints and multimodal fusion, characterized in that, include: The multimodal perception module is used to acquire visual information of the scene to be captured and the user's voice commands, identify and locate target objects in the scene based on the visual information, and recognize and semantically encode the voice commands to obtain text semantic features that represent the user's intention. The multimodal fusion module is used to fuse the semantic features of the text with the information of the target object, predict the grasping parameters based on the fused features, and generate multiple candidate grasping points. The text semantic features are fused with the information of the target object, including: The text semantic features are spatially expanded to generate a semantic feature tensor with the same spatial resolution as the visual features. The semantic feature tensor is concatenated with the visual features containing the target object information at the pixel level to obtain the concatenated fused feature tensor. Perform a convolution operation on the spliced ​​fused feature tensor to generate the fused features; The semantic constraint module is used to use the semantic features of the text as semantic constraints, perform semantic consistency evaluation on the multiple candidate crawling points, filter out the target crawling points, and generate crawling control instructions. The semantic consistency of the multiple candidate crawl points is evaluated, including: Based on the prediction results of the crawling parameters, the crawling quality parameters corresponding to each candidate crawling point are obtained; From the fused features, regional features centered on each candidate grasping point are extracted as corresponding local visual features. Calculate the semantic correlation between the text semantic features and each local visual feature to obtain the semantic consistency score of each candidate crawling point; The target crawling point is determined based on the crawling quality parameters and semantic consistency score of each candidate crawling point; The determination of the target capture point is achieved through the following weighted decision function: ； in, For the index of the target crawl point, This indicates the capture quality parameters; Indicates semantic consistency score; Weighting coefficients for quality parameters This is a weighting coefficient for semantic consistency, used to balance geometric graspability and semantic matching degree; The grasping execution module is used to control the robotic arm to perform grasping operations according to the grasping control instructions.

5. An apparatus comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the robotic arm grasping method based on semantic constraints and multimodal fusion as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the robotic arm grasping method based on semantic constraints and multimodal fusion as described in any one of claims 1 to 3.

7. A computer program product, comprising software code, characterized in that, The program in the software code executes the steps of the robotic arm grasping method based on semantic constraints and multimodal fusion as described in any one of claims 1 to 3.

Images