Intelligent mobile grabbing robot control method and system based on modular architecture

Abstract

The application belongs to the field of intelligent robots, and discloses a control method and system of an intelligent mobile grabbing robot based on a modular architecture, which comprises the following steps: collecting image information of a target object in real time, processing image data in real time, identifying the target object and outputting the position and category of the target object, and acquiring depth information of the target object; a user terminal sends natural language text, a cloud server receives the text information, a large language model is used to generate language reply information through natural language interaction and to generate decomposed action instructions through action decomposition; the language reply information is returned to the user terminal through the cloud server; the decomposed action instructions are sent to an edge computing device, so that the decomposed action instructions are combined with the depth information of the target object to obtain a final action instruction sequence; and an IPC controls the mobile grabbing robot to complete related actions such as movement and grabbing. The application improves the naturalness of robot interaction, the intelligence of decision-making and the adaptability of the system, and is suitable for flexible intelligent manufacturing requirements.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robots, and more specifically, relates to a method and system for an intelligent mobile grasping robot based on a modular "vision-language-action" architecture. Background Technology

[0002] Industrial production is transforming from traditional mechanization and automation to intelligent and flexible manufacturing. Intelligent robots, as one of the key technologies for realizing intelligent manufacturing, play a crucial role in industrial production.

[0003] Therefore, industrial robots urgently need to evolve towards natural human-machine interaction, rapid perception, and intelligent decision-making. However, current industrial robots still face many limitations in their development:

[0004] 1) Complex interaction methods: Although traditional physical buttons or teach pendants can achieve precise operation, they have high requirements and require pre-programming or recording of action sequences, making it difficult to cope with dynamically changing task requirements. The operation threshold is high and the flexibility is insufficient.

[0005] 2) Limited decision-making ability: The decision-making mechanism based on rule engines relies on a large number of predefined templates. When faced with instructions or expressions in natural language, it is prone to semantic ambiguity, which leads to decision failure and poor intelligence.

[0006] 3) Insufficient system architecture adaptability: Most existing intelligent robot systems adopt an end-to-end holistic design, with perception, decision-making and execution being highly coupled. This not only requires a lot of resources to build and deploy models, but also requires significant modifications to the existing system when upgrading or expanding, resulting in insufficient adaptability.

[0007] In recent years, the rise of large language models has opened up new possibilities for natural language interaction and decision-making. With their powerful understanding and generation capabilities, large language models can engage in natural human-computer dialogue, process complex natural language commands, and generate dynamic decisions. However, how to efficiently integrate large language models with robot control systems remains an under-researched issue. Summary of the Invention

[0008] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a control method and system for an intelligent mobile grasping robot based on a modular architecture, which aims to solve the problems of complex interaction and limited decision-making ability of traditional robots.

[0009] To achieve the above objectives, according to one aspect of the present invention, a control method for an intelligent mobile grasping robot based on a modular architecture is provided, comprising the following steps:

[0010] S1: Real-time acquisition of image information of target objects, real-time processing of image data, identification of target objects and output of the position and category of target objects, acquisition of depth information of detected target objects, and provision of basic detection results for subsequent decision-making;

[0011] S2: The user sends natural language text, the cloud server receives the text information, uses a large language model to perform natural language interaction to generate language response information, and performs action decomposition to generate decomposed action instructions; the language response information is returned to the user through the cloud server; the decomposed action instructions are sent to the edge computing device, so that the decomposed action instructions are combined with the target object depth information obtained in step S1 to obtain the final action instruction sequence, thereby realizing intelligent decision-making.

[0012] S3: Send the sequence of motion instructions generated in step S2 to the IPC. The IPC controls the mobile grasping robot to complete the movement and grasping actions, ensuring the accuracy and stability of the actions.

[0013] Preferably, step S1 specifically includes the following steps:

[0014] S11: Collect images through web crawlers and camera capture, manually annotate each image, and generate an annotation file for object detection;

[0015] S12: Perform preprocessing and data augmentation operations on the original images generated in step S11 to expand the dataset, which is then divided into a training set, a validation set, and a test set.

[0016] S13: Uses a depth camera to acquire RGB images and depth maps of the target object in real time;

[0017] S14: Select the improved YOLOv10 model and train it on the training set obtained in step S12 on the cloud server. After training, a model file is obtained. The model file is quantized to obtain a quantized model, which is then deployed to an edge computing device to achieve local operation. At the same time, the RGB image collected in step S13 is detected in real time based on the quantized model to obtain the bounding box and classification label of the target object. The depth value of the object is obtained by combining the depth map collected in step S13, and the detection result is output.

[0018] As a preferred option, the specific steps of the preprocessing and data augmentation operations in step S12 are as follows: automatically remove the background from some images, extract the target element map, and use random transformation and background synthesis to generate diverse synthetic images and corresponding annotation files, thereby enhancing the diversity and robustness of the dataset.

[0019] Preferably, the improved YOLOv10 model in step S14 includes the ResSPPF module, the OGCA module, and the C2f_ParNetAttention module.

[0020] Among them, the ResSPPF module optimizes the multi-scale feature extraction capability by introducing a cascaded residual structure, thereby improving the detection performance of small targets and multi-scale targets.

[0021] The OGCA module enhances the model's adaptability to complex scenes by introducing a global feature fusion mechanism and an occlusion awareness branch.

[0022] The C2f_ParNetAttention module enhances detection performance by replacing labeled bottleneck blocks with bottleneck blocks improved based on ParNetAttention, thereby improving the spatial awareness of features and the ability to model relationships between channels.

[0023] Preferably, the quantization method in step S14 is to convert the model weights from floating-point numbers to integers, perform format conversion and quantization processing on the trained model, and generate a quantized model suitable for NPU operation.

[0024] Preferably, step S2 specifically includes the following steps:

[0025] S21: The user inputs natural language text information through the public platform. The message interface of the public platform sends the instruction to the cloud server, and the cloud server receives and preprocesses it.

[0026] S22: Call the large language model to perform natural language processing on the text information obtained in step S21. Utilize the semantic understanding capability of the large language model, generate language response information for natural language interaction based on CoT and few-shot prompting methods, and decompose actions to generate executable action instruction sequences.

[0027] S23: The language response information generated in step S22 is returned to the user terminal via the passive response interface of the public platform to achieve a two-way interactive experience;

[0028] S24: The sequence of action instructions generated in step S22 is sent to the edge computing device through the cloud server, so that the decomposed action instructions are combined with the target category, bounding box and depth information of the target object obtained in step S1 to construct a complete sequence of action instructions.

[0029] S24: For the issues of not detecting the target object in step S1 and the incorrect parameter of the instruction sequence generated in step S24, the message is sent to the cloud server through the edge computing device, and the cloud server actively sends it to the user terminal by calling the template message interface of the public platform.

[0030] Preferably, step S3 specifically includes the following steps:

[0031] S31: Write the control program to realize the path planning and motion control of the mobile grasping robot;

[0032] S32: The sequence of action instructions generated in step S2 is transmitted from the edge computing device to the IPC. The IPC controls the mobile grasping robot to complete the movement and grasping related actions, ensuring the accuracy and stability of the actions.

[0033] To achieve the above objectives, according to another aspect of the present invention, a control system for an intelligent mobile grasping robot based on a modular architecture is provided, comprising:

[0034] The vision module acquires image information of the target object in real time, processes image data in real time, identifies the target object and outputs the position and category of the target object, and obtains the depth information of the detected target object to provide basic detection results for subsequent decision-making.

[0035] The language module involves the user sending natural language text, the cloud server receiving the text information, using a large language model to perform natural language interaction to generate language response information, and performing action decomposition to generate decomposed action instructions; the language response information is returned to the user through the cloud server; the decomposed action instructions are sent to the edge computing device, where they are combined with the target object depth information obtained by the vision module to obtain the final action instruction sequence, thus achieving intelligent decision-making.

[0036] The motion module sends the sequence of motion instructions generated by the language module to the IPC. The IPC controls the mobile grasping robot to complete the movement and grasping actions, ensuring the accuracy and stability of the actions.

[0037] Preferably, the vision module includes:

[0038] The image acquisition unit collects high-quality images through web crawlers and camera capture, manually annotates each image, and generates an annotation file for target detection.

[0039] The image preprocessing unit performs preprocessing and data augmentation operations on the raw images generated by the image acquisition unit, expands the dataset, and divides the dataset into training set, validation set and test set.

[0040] The depth information acquisition unit uses a depth camera to acquire RGB images and depth maps of the target object in real time;

[0041] The model training unit selects an improved YOLOv10 model and trains it on a cloud server, sets the model training parameters, and obtains the model file after training. The model file is then quantized to obtain a quantized model, which is deployed to an edge computing device for localized operation. Simultaneously, based on the quantized model, RGB images are detected in real time to obtain the bounding box and classification label of the target object. The depth value of the object is obtained by combining the depth map, and the detection results are output.

[0042] Preferably, the language module includes:

[0043] The input unit allows users to input natural language text information through the public platform. The public platform's message interface sends the instructions to the cloud server, which then receives and preprocesses them.

[0044] The processing unit calls the large language model to perform natural language processing on the text information, utilizes the semantic understanding capability of the large language model, generates language response information for natural language interaction based on CoT and few-shot prompting methods, and performs action decomposition to generate executable action instruction sequences.

[0045] The interaction unit calls the passive reply interface of the public platform to return the language reply information obtained by the processing unit to the user, realizing a two-way interactive experience;

[0046] The decomposed action unit sends the sequence of action instructions obtained by the processing unit from the cloud server to the edge computing device. The decomposed action instructions are combined with the target category, bounding box and depth information of the target object obtained by the vision module to construct a complete sequence of action instructions, which is then passed to the action module.

[0047] The error handling unit will send the error message to the cloud server via the edge computing device for issues such as the vision module failing to detect the target object or errors in the instruction sequence parameters generated by the decomposition action unit. The cloud server will then call the template message interface of the public platform to actively send the message to the user.

[0048] Preferably, the CoT and few-shot prompting technology involves constructing complete prompt words by designing system prompts, task instructions, few-shot examples, and their thought chains.

[0049] Preferably, the action module includes:

[0050] Control unit: Write control programs to realize path planning and motion control of the mobile grasping robot;

[0051] The communication unit transmits the sequence of action commands generated by the language module to the IPC via the edge computing device. The IPC controls the mobile grasping robot to complete the movement and grasping actions, ensuring the accuracy and stability of the actions.

[0052] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0053] The method and system of this invention decouple and optimize the functions of visual perception, natural language interaction, intelligent decision-making based on large language models, and action control execution through modular design. It can not only identify the category, bounding box, and depth information of objects, but also meet the real-time requirements of industry. At the same time, it solves the problems of complex interaction and limited decision-making ability of traditional robots, as well as the complex construction and poor flexibility of embodied intelligent robots. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the overall system architecture provided in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the visual module structure provided in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the improved YOLOv10 network structure provided in an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the language module structure provided in an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the action module structure provided in an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0060] like Figure 1 As shown, addressing the problems of complex interaction, limited intelligent decision-making capabilities, and complex and inflexible construction of embodied intelligent robots in current traditional robots, this invention provides an intelligent mobile grasping robot control method based on a modular vision-language-action architecture. By decoupling and coordinating visual perception, natural language interaction, intelligent decision-making using a large language model, and action control execution, it enables intelligent mobile grasping tasks in industrial scenarios. This architecture fully combines cloud computing power, edge real-time performance, and local stability to form a highly efficient cloud-edge-local collaborative system.

[0061] The method includes the following steps:

[0062] S1: Real-time acquisition of image information of target objects, real-time processing of image data, identification of target objects and output of the position and category of target objects, acquisition of depth information of detected target objects, and provision of basic detection results for subsequent decision-making;

[0063] Step S1 specifically includes the following steps:

[0064] S11: Collect images through web crawlers and camera capture, manually annotate each image, and generate an annotation file for object detection;

[0065] S12: Perform preprocessing and data augmentation operations on the original images generated in step S11 to expand the dataset, which is then divided into a training set, a validation set, and a test set.

[0066] The specific steps of preprocessing and data augmentation are as follows: automatically remove the background from some images, extract the target element map, and use random transformation and background synthesis to generate diverse synthetic images and corresponding annotation files to enhance the diversity and robustness of the dataset;

[0067] S13: Uses a depth camera to acquire RGB images and depth maps of the target object in real time;

[0068] S14: Select the improved YOLOv10 model and train it on the training set obtained in step S12 on the cloud server. After training, a model file is obtained. The model file is quantized to obtain a quantized model, which is then deployed to an edge computing device to achieve local operation. At the same time, the RGB image collected in step S13 is detected in real time based on the quantized model to obtain the bounding box and classification label of the target object. The depth value of the object is obtained by combining the depth map collected in step S13, and the detection result is output.

[0069] Step S14 improves the YOLOv10 model by including the ResSPPF module, the OGCA module, and the C2f_ParNetAttention module.

[0070] Among them, the ResSPPF module optimizes the multi-scale feature extraction capability by introducing a cascaded residual structure, thereby improving the detection performance of small targets and multi-scale targets.

[0071] The OGCA module enhances the model's adaptability to complex scenes by introducing a global feature fusion mechanism and an occlusion awareness branch.

[0072] The C2f_ParNetAttention module enhances detection performance by replacing labeled bottleneck blocks with bottleneck blocks improved based on ParNetAttention, thereby improving the spatial awareness of features and the ability to model relationships between channels.

[0073] In step S14, the quantization method is to convert the model weights from floating-point numbers to integers, perform format conversion and quantization processing on the trained model, and generate a quantized model suitable for NPU operation.

[0074] S2: The user sends natural language text, the cloud server receives the text information, uses a large language model to perform natural language interaction to generate language response information, and performs action decomposition to generate decomposed action instructions; the language response information is returned to the user through the cloud server and the decomposed action instructions are sent to the edge computing device, so that the decomposed action instructions are combined with the target object depth information obtained in step S1 to obtain the final action instruction sequence and realize intelligent decision-making.

[0075] Step S2 specifically includes the following steps:

[0076] S21: The user inputs natural language text information through the public platform. The message interface of the public platform sends the instruction to the cloud server, and the cloud server receives and preprocesses it.

[0077] S22: Call the large language model to perform natural language processing on the text information obtained in step S21. Utilize the semantic understanding capability of the large language model, generate language response information for natural language interaction based on CoT and few-shot prompting methods, and decompose actions to generate executable action instruction sequences.

[0078] S23: The language response information generated in step S22 is returned to the user terminal via the passive response interface of the public platform to achieve a two-way interactive experience;

[0079] S24: The sequence of action instructions generated in step S22 is sent to the edge computing device through the cloud server, so that the decomposed action instructions are combined with the target category, bounding box and depth information of the target object obtained in step S1 to construct a complete sequence of action instructions, and then passed to the action module.

[0080] S24: For the issues of not detecting the target object in step S1 and the incorrect parameter of the instruction sequence generated in step S24, the message is sent to the cloud server through the edge computing device, and the cloud server actively sends it to the user terminal by calling the template message interface of the public platform.

[0081] S3: Send the sequence of motion instructions generated in step S2 to the IPC. The IPC controls the mobile grasping robot to complete the movement and grasping actions, ensuring the accuracy and stability of the actions.

[0082] Step S3 specifically includes the following steps:

[0083] S31: Write the control program to realize the path planning and motion control of the mobile grasping robot;

[0084] S32: The sequence of action instructions generated in step S2 is transmitted from the edge computing device to the IPC. The IPC controls the mobile grasping robot to complete the movement and grasping related actions, ensuring the accuracy and stability of the actions.

[0085] Meanwhile, this invention provides an intelligent mobile grasping robot control system based on a modular vision-language-action architecture, including:

[0086] The vision module acquires image information of the target object in real time, processes image data in real time, identifies the target object and outputs the position and category of the target object, and obtains the depth information of the detected target object to provide basic detection results for subsequent decision-making.

[0087] The language module involves the user sending natural language text, the cloud server receiving the text information, using a large language model to perform natural language interaction to generate language response information, and performing action decomposition to generate decomposed action instructions; the language response information is returned to the user through the cloud server; the decomposed action instructions are sent to the edge computing device, where they are combined with the target object depth information obtained by the vision module to obtain the final action instruction sequence, thus achieving intelligent decision-making.

[0088] The motion module sends the sequence of motion instructions generated by the language module to the IPC. The IPC controls the mobile grasping robot to complete the movement and grasping actions, ensuring the accuracy and stability of the actions.

[0089] The vision module includes:

[0090] The image acquisition unit collects high-quality images through web crawlers and camera capture. Each image is manually annotated using the LabelImg tool to generate an annotation file for object detection in a YOLO-compatible txt file format.

[0091] The image preprocessing unit performs preprocessing and data augmentation operations on the raw images generated by the image acquisition unit, and expands the dataset. The dataset is divided into training set, validation set and test set in a ratio of 8:1:1.

[0092] The depth information acquisition unit uses a depth camera to acquire RGB images and depth maps of the target object in real time;

[0093] The model training unit selects an improved YOLOv10 model and trains it on a cloud server using the PyTorch framework. The model training parameters are set, and the model file is obtained after training. The model file is quantized to obtain a quantized model, which is then deployed to an edge computing device for localized operation. Simultaneously, based on the quantized model, RGB images are detected in real time to obtain the bounding box and classification label of the target object. The depth value of the object is obtained by combining the depth map. The detection results are output in JSON format.

[0094] Post-Training Quantization (PTQ) is performed after model training is complete, using PTQ technology to guide the quantization process. It eliminates the need for retraining the original model, directly converting it into a fixed-point computation network with only a few hyperparameters required. The quantization method involves converting model weights from 32-bit floating-point numbers to 8-bit integers using PTQ, converting the trained PT format model to ONNX format, and then using the RKNNToolkit2 development kit to quantize the ONNX model, generating a quantized model suitable for NPU operation.

[0095] The specific steps of the preprocessing and data augmentation operations are as follows: using the Pixian.AI tool to automatically remove the background from some images, extract the target element image, and use random transformations (such as rotation, scaling, brightness adjustment, etc.) and background synthesis (such as industrial scenes, office scenes, natural environments, etc.) to generate diverse synthetic images and corresponding annotation files, thereby enhancing the diversity and robustness of the dataset.

[0096] The depth information acquisition unit uses a depth camera as the image acquisition device, which has the ability to acquire RGB images and depth information, and provides dual-mode data streams of RGB images and depth maps.

[0097] The base model used in the model training unit is an improved YOLOv10; the improved YOLOv10 includes a ResSPPF module, an OGCA module, and a C2f_ParNetAttention module.

[0098] The ResSPPF (Residual Spatial Pyramid Pooling Fast) module optimizes multi-scale feature extraction capabilities by introducing a cascaded residual structure, such as... Figure 3 As shown, the input features of the ResSPPF module are first converted into initial features through a 1×1 convolution. Where N, C, H, and W represent the batch size, number of channels, height, and width, respectively, preserving the original information for subsequent operations and storing the initial features as residual branches. Unlike the SPPF module, the ResSPPF module does not halve the number of channels, retaining all input channel information. Subsequently, the module uses max pooling layers for multi-scale pooling operations and introduces cascaded residual connections. Specifically, after each pooling operation, the pooling result is added element-wise to the current feature, as shown in the following equation:

[0099] x = MaxPool(x) + x

[0100] MaxPool is the max pooling layer, which generates multi-scale features through three iterations. The four sets of features [x0, x1, x2, x3], including the original features, are concatenated along the channel dimension to generate a shape... The feature vector is calculated, where x0 is the original feature and x1, x2, and x3 are the results of three pooling operations. A 1×1 convolution is used to adjust the number of channels back to the target value, generating the fused feature. Finally, a global residual connection is introduced to add the fused feature to the initial feature to further preserve the original information, generating the final output, as shown in the equation:

[0101] x = x + residual

[0102] The ResSPPF module retains the efficient multi-scale feature extraction capabilities of the SPPF module and further optimizes the detection performance of small and multi-scale targets. Cascaded residual connections dynamically fuse the current pooling features with those from the previous stage in each pooling operation, mitigating the spatial resolution compression problem of small targets caused by continuous pooling, enhancing the expression of edge and texture details of small targets, and iteratively optimizing the consistency of feature hierarchy across different scales. Global residual connections further compensate for the potential weakening of small target details during pooling by adding the fused features to the initial features, improving the detection accuracy and localization accuracy of small targets. ResSPPF does not follow the SPPF module's design of halving the number of channels, retaining all input channel information. Furthermore, cascaded residual connections and global residual connections are implemented only through element-wise addition operations, without introducing additional parameters.

[0103] To overcome the shortcomings of the CA module in 2D spatial relationship modeling and occlusion scenarios, an improved Occlusion-aware Global CoordAttention (OGCA) module is proposed. By introducing a global feature fusion mechanism and an occlusion-aware branch, the model's adaptability to complex scenarios is further enhanced.

[0104] The input feature map of the OGCA module is Where N, C, H, and W represent the batch size, number of channels, height, and width, respectively. First, adaptive average pooling is performed along the height and width dimensions to generate height-oriented features. and width direction features Next, the height-direction feature x h With width direction feature x w Concatenate using the Concat operation The channels are adjusted using 1×1 convolutions. To enhance the ability to perceive global contextual information, the OGCA module introduces a global pooling operation to generate global features. And extend to x cThe dimension of feature alignment, relative to x c The features are added together, fusing global contextual information so that the generation of attention weights can comprehensively consider global and local spatial dependencies. The fused features are then processed by batch normalization and activation functions to provide non-linear expressive power. The processed features are then segmented along the height and width dimensions. and Attention weights 'a' in the height and width directions are generated using 1×1 convolutions. h With a w To address the issues of instance overlap and occlusion, the OGCA module incorporates an occlusion-aware branch. This branch uses a 1×1 convolution to reduce the dimensionality of the input features to a single channel, generating a preliminary occlusion map. The image is then locally refined using 3×3 convolutions to further enhance its ability to detect overlapping regions. The image is then normalized to the [0,1] range using the Sigmoid function to obtain the occlusion probability. Subsequently, the occlusion map is adaptively averaged along the height and width dimensions to generate the occlusion distribution in the height direction. Distribution of occlusion in the width direction After the attention weights are normalized using the Sigmoid function, they are adjusted in conjunction with the occlusion distribution to reduce the attention weights in overlapping regions and enhance attention to non-overlapping target regions, as shown in the following equation:

[0105] a h =Sigmoid(a h )·(1-occ h )

[0106] a w =Sigmoid(a w )·(1-occ w )

[0107] Finally, the OGCA module will adjust the attention weights a h With a w Applied to the input feature map x, the output features are generated through element-wise multiplication, as shown in the following equation:

[0108] x out =x·a h ·a w

[0109] The OGCA module significantly improves the accuracy of feature representation and model robustness through the synergy of a global feature fusion mechanism and an occlusion-aware branch. Global pooling enhances the model's ability to model two-dimensional space by integrating height and width information, making attention weight generation more globally semantically grounded. Meanwhile, the occlusion-aware branch dynamically generates occlusion distributions, precisely suppressing the influence of overlapping regions and ensuring the model focuses on the unoccluded target area. Compared to the CA module, it not only retains its lightweight design advantages but also significantly improves the accuracy of feature representation through the introduction of global contextual information and the ability to perceive occlusion regions. This enhances spatial awareness while maintaining adaptability to complex scenes and computational efficiency.

[0110] To overcome the shortcomings of the C2f module in multi-scale feature extraction, the C2f_PNA module is proposed. By replacing the labeled bottleneck block with an improved bottleneck block based on ParNetAttention (ParNet Block Attention), it further enhances the spatial awareness and inter-channel relationship modeling capabilities of features, thereby improving detection performance. The C2f_PNA module first generates initial features through a 1×1 convolution of the input features. Where N, C, H, and W represent the batch size, number of channels, height, and width, respectively. The initial features are divided into two parts along the channel dimension: one part features... Retain directly, and retain the other features. Processing is performed through n Bottleneck_PNA modules. In each Bottleneck_PNA module, the input features... First, the number of channels is adjusted using 1×1 convolutions, followed by the extraction of local spatial features using 3×3 convolutions, and then enhanced using the ParNetAttention module. ParNetAttention employs a multi-branch parallel structure; the first branch generates channel enhancement features using 1×1 convolutions. The second branch extracts local spatial features through 3×3 convolution. The third branch utilizes the SSE mechanism to generate global features through adaptive average pooling, and then generates spatial attention weights through 1×1 convolution and the Sigmoid function, which are multiplied with the original features to obtain... [x 21 ,x 22 ,x 23 After the three branch features are fused, if residual connections are enabled, they are added to the input features x2 to generate the bottleneck block output. Finally, a 1×1 convolution is applied to adjust to the target number of channels, outputting the final features.

[0111] The C2f_PNA module, by introducing ParNetAttention, significantly enhances the expressive power of spatial context and channel relationships of features, enabling it to capture more comprehensive feature information than C2f in complex scenes. ParNetAttention's multi-branch design combines 1×1 and 3×3 convolutions to extract multi-level features and utilizes the SSE mechanism to enhance global perception, making C2f_PNA more robust in detecting dense or small targets. Simultaneously, its lightweight structure optimizes computational efficiency, ensuring applicability in real-time detection tasks. Therefore, the C2f_PNA module not only integrates the cross-stage fusion and multi-branch characteristics of C2f, maintaining its ability to capture multi-scale targets, but also achieves a balance between feature representation power and computational efficiency through the introduction of ParNetAttention, providing an efficient and robust solution for target detection in complex scenes.

[0112] The improved YOLOv10, by integrating the ResSPPF, OGCA, and C2f_ParNetAttention modules, significantly enhances detection accuracy, robustness, and adaptability to complex scenes while maintaining high real-time performance. The ResSPPF module enhances the representation of edge and texture details of small targets, improves the hierarchical consistency of multi-scale features, and increases detection accuracy, particularly in small target localization and recognition. The OGCA module improves the accuracy of feature representation and the model's perception of two-dimensional spatial relationships, effectively suppresses interference from occluded regions, and enhances the ability to focus on unoccluded targets, thus exhibiting stronger robustness in scenes with dense or overlapping targets. The C2f_ParNetAttention module significantly improves feature capture capabilities in complex scenes through more comprehensive spatial context and channel relationship modeling, demonstrating excellent performance, especially in the detection of dense and small targets, while optimizing computational efficiency to meet real-time requirements. The synergistic effect of the three modules enables YOLOv10 to achieve a comprehensive breakthrough in small target detection accuracy, multi-scale target capture capability, and adaptability to complex scenes, providing an efficient, robust, and high-precision solution for industrial-grade real-time detection tasks.

[0113] The language module includes:

[0114] The input unit allows users (WeChat) to input natural language text information through a public platform (such as the WeChat Official Accounts Platform). The message interface of the public platform sends the instructions to the cloud server in the form of an HTTPS request. The cloud server deploys the FLASK framework to receive and preprocess the information.

[0115] The processing unit calls the API of the large language model to perform natural language processing on the text information, utilizes the semantic understanding capability of the large language model, generates language response information for natural language interaction based on CoT and few-shot prompting methods, and performs action decomposition to generate executable action instruction sequences.

[0116] The interaction unit calls the passive reply interface of the public platform (such as WeChat public account) in JSON format to return the language reply information obtained by the processing unit to the user, thus realizing a two-way interactive experience;

[0117] The decomposed action unit sends the sequence of action instructions obtained by the processing unit to the edge computing device via a SCOket object based on the TCP protocol. This decomposed action instructions are then combined with the target category, bounding box, and depth information of the target object obtained by the vision module to construct a complete sequence of action instructions, which is then passed to the action module.

[0118] CoT and few-shot prompting technology construct complete prompt words by designing system prompts, task instructions, few-shot examples, and their thought chains. System prompts clearly define the robot's identity as a motion planning assistant, focusing on mobile grasping tasks in industrial scenarios; they provide environmental information and assume potential dynamic changes in the scene. Task instructions include robot movement, robotic arm movement, object detection, and gripper opening and closing. Few-shot examples provide diverse examples covering scenarios such as dialogues without action intent, ambiguous instructions, single tasks, and complex tasks, guiding the model to learn action decomposition logic and output specifications. The thought chain design involves a step-by-step reasoning process: 1. Identify user intent and determine if an action instruction exists; 2. Extract core information such as actions, target objects, and position parameters, clarifying any ambiguities; 3. Generate an action sequence based on safety and dependency principles, generating a natural language response and action sequence to explain the plan and request confirmation.

[0119] The language module also includes an error handling unit. For problems such as the vision module failing to detect the target object or errors in the instruction sequence parameters generated by the decomposition action unit, the problem situation is sent to the cloud server via the edge computing device using a Socket object based on the TCP protocol. The cloud server then calls the template message interface of the public platform to actively send the message to the user terminal.

[0120] The action module includes:

[0121] The control unit, based on the HPAC programming platform, writes control programs to realize path planning and motion control of the mobile grasping robot;

[0122] The communication unit transmits the sequence of action instructions generated by the language module from the edge computing device to the IPC via the Modbus protocol. The Modbus protocol uses RTU mode to ensure the stability and real-time performance of the communication.

[0123] Modbus is a simple bus protocol used in industry for communication. It boasts advantages such as excellent compatibility, high reliability, high-speed data transmission, high scalability, and ease of use.

[0124] Specifically, the present invention has the following beneficial effects:

[0125] (1) The vision module of this preferred embodiment uses a depth camera to acquire RGB images and depth information, and combines a pre-trained target detection model with post-training quantization technology to achieve high-precision recognition and localization of target objects. The system can not only identify the category, bounding box and depth information of objects, but also meet the real-time requirements of industrial applications through the localized deployment of edge computing devices.

[0126] (2) This preferred embodiment achieves a zero-training natural language interaction method by combining the WeChat official account platform with a large language model. Users do not need to learn complex operation interfaces or predefined commands; they can communicate two-way with the robot using only everyday language.

[0127] (3) This preferred embodiment utilizes the powerful semantic understanding capabilities of the large language model, combined with thought chain and few-shot prompting techniques, to significantly improve the naturalness of human-computer interaction and the ability to understand natural language instructions and decompose tasks. With the help of CoT and few-shot prompting, the action decomposition capability and adaptability of the large language model to diverse instructions are enhanced, which not only improves the intelligence level of the system, but also enhances the robustness and flexibility of unstructured language input.

[0128] (4) This preferred embodiment decouples visual perception, natural language interaction, large language model intelligent decision-making, and action execution through a modular architecture. Each module runs independently and collaborates through a standardized JSON interface. This design greatly improves the system's adaptability and scalability, facilitating rapid deployment and iteration in different industrial scenarios. The modular architecture adopts a cloud-edge-local collaborative mechanism, with each module running on a cloud server, edge computing device, and IPC, respectively, fully utilizing cloud computing capabilities, the real-time performance of edge computing devices, and the stability of local hardware.

[0129] The present invention will be further described below with reference to specific embodiments.

[0130] Example 1: Take the screwdriver from the table to the table on the right.

[0131] Visual modules, such as Figure 2 As shown, specifically:

[0132] High-quality images of objects such as screwdrivers (but not limited to screwdrivers) are collected through web crawling and camera photography. Each image is manually labeled using the LabelImg tool to generate an object detection annotation file in a YOLO-compatible txt file format.

[0133] The original images are preprocessed and data augmented to expand the dataset, which is then divided into training, validation, and test sets in an 8:1:1 ratio.

[0134] An improved YOLOv10 model was selected as the base object detection model, and the model was trained on a cloud server using the PyTorch framework. Training parameters such as batch size, initial learning rate, optimizer, and number of training epochs were set, and the model file was obtained after training.

[0135] The model weights are converted from 32-bit floating-point numbers to 8-bit integers using PTQ technology. The trained pt format model is then converted to an ONNX format model. The ONNX model is quantized using the RKNN Toolkit2 development kit to generate an RKNN model suitable for NPU operation, reducing the model's storage space and improving its running efficiency on the target hardware.

[0136] The quantization model was deployed on a LubanCat-RK3588 edge computing device, and a depth camera was used as the image acquisition device to acquire RGB images and depth maps in real time. Code was written to detect RGB images in real time based on the YOLOv8 quantization model, obtaining the bounding boxes and classification labels of objects, and simultaneously combining the depth map to obtain the object's depth value. The detection results were output in JSON format, for example: {"object":"Screwdriver","bbox":[20,30,100,150],"confidence":0.95,"depth":50,}, where "object" is the object label, "bbox" is the bounding box position parameter, "confidence" is the object's confidence score, and "depth" is the object's depth value.

[0137] Language modules, such as Figure 4 As shown, specifically:

[0138] Users input the natural language text message "Take the screwdriver on the table to the right table" through the WeChat Official Account platform and send it. The WeChat Developer Platform's message interface sends the instruction from the WeChat server to the cloud server in the form of an HTTPS request.

[0139] The cloud server deploys the FLASK framework to receive HTTPS requests from the WeChat server. After receiving the request, it extracts the text field from the JSON data, obtaining "Take the screwdriver on the table to the right table", and performs preprocessing.

[0140] The preprocessed instructions call a large language model, such as GLM-4, deployed in the cloud via API. To improve decomposition accuracy, CoT and few-shot prompting techniques are used. This involves designing system prompts, task instructions, few-shot examples, and their thought chains to construct complete prompt words, generating language response information through natural language interaction, and decomposing actions to generate executable action instruction sequences.

[0141] The reply message is encapsulated in JSON format and returned to the user through the WeChat Official Account's passive reply interface, realizing a two-way interactive experience.

[0142] The decomposed sequence of motion instructions is sent from the cloud server to the LubanCat-RK3588 via a Socket object using the TCP protocol, with the data format being JSON. Upon receiving the instruction sequence, the LubanCat-RK3588 combines the screwdriver position and depth information obtained from the vision module to supplement and refine the parameters of the motion instruction sequence. For example, {

[0143] "action":[

[0144] {"object_detection":{"object":"Screwdriver","bbox":[20,30,100,150],"confidence":0.95,"depth":50,}},

[0145] {"gripper":"open"},

[0146] {"arm_move":[20,30,100,150,50]},

[0147] {"gripper":"close"},

[0148] {"arm_move_safe":[]},

[0149] {"move":[200,10],}

[0150] {"object_detection":{"object":"table","bbox"[5,5,620,610],"confidence":0.95,"depth":60,}},

[0151] {"arm_move":[5,5,620,610,60]},

[0152] {"gripper":"open"},

[0153] {"arm_move_safe":[]},

[0154] {"gripper":"close"}, ]

[0156] }

[0157] If problems occur, such as the vision module failing to detect the target object or errors in the instruction sequence parameters generated by the decomposed action unit, the LubanCat-RK3588 sends the problem information to the cloud server using the Socket object based on the TCP protocol. The cloud server then proactively notifies the user by calling the template message interface of the WeChat official account.

[0158] Action modules, such as Figure 5 As shown, specifically:

[0159] Based on the HPAC programming platform, control programs for robot movement and robotic arm movement are written to realize path planning and motion control of the mobile grasping robot.

[0160] The action commands and parameters obtained by the language module are transmitted to the IPC via LubanCat-RK3588 based on the Modbus protocol. Modbus uses RTU mode to ensure the stability and real-time performance of the communication.

[0161] For the statement "move the screwdriver from the table to the right table", the mobile gripper performs the following actions in sequence: target detection of the screwdriver's position and depth, end effector opening, end effector moving to the screwdriver's position, end effector closing, end effector moving to a safe position, robot moving to the vicinity of the right table, target detection of the table's position and depth, end effector moving to the tabletop position, end effector opening, end effector moving to a safe position, end effector closing.

[0162] Through modular design, the functions of visual perception, natural language interaction, large language model intelligent decision-making, and motion control execution are decoupled and optimized and integrated. This not only enables the identification of object categories, bounding boxes, and depth information, but also meets the real-time requirements of industry. At the same time, it solves the problems of complex interaction and limited decision-making ability of traditional robots, as well as the complex construction and poor flexibility of embodied intelligent robots.

[0163] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A control method for an intelligent mobile grasping robot based on a modular architecture, characterized in that, Includes the following steps: S1: Real-time acquisition of image information of target objects, real-time processing of image data, identification of target objects and output of the position and category of target objects, and acquisition of depth information of detected target objects; Step S1 specifically includes the following steps: S11: Collect images through web crawlers and camera capture, manually annotate each image, and generate an annotation file for object detection; S12: Perform preprocessing and data augmentation operations on the original images generated in step S11 to expand the dataset, which is then divided into a training set, a validation set, and a test set. S13: Uses a depth camera to acquire RGB images and depth maps of the target object in real time; S14: Select the improved YOLOv10 model and train it on the training set obtained in step S12 on the cloud server. After training, the model file is obtained. The model file is quantized to obtain the quantized model, which is then deployed to the edge computing device to achieve local operation. At the same time, the RGB image collected in step S13 is detected in real time based on the quantized model to obtain the bounding box and classification label of the target object. The depth value of the object is obtained by combining the depth map collected in step S13. The detection result is output. Step S14 improves the YOLOv10 model by including the ResSPPF module, the OGCA module, and the C2f_ParNetAttention module. Among them, the ResSPPF module optimizes the multi-scale feature extraction capability by introducing a cascaded residual structure, thereby improving the detection performance of small targets and multi-scale targets. The OGCA module enhances the model's adaptability to complex scenes by introducing a global feature fusion mechanism and an occlusion awareness branch. The C2f_ParNetAttention module enhances detection performance by replacing labeled bottleneck blocks with bottleneck blocks improved based on ParNetAttention, thereby improving the spatial awareness of features and the ability to model inter-channel relationships. The OGCA module is designed with an occlusion-sensing branch, through... Convolution reduces the dimensionality of the input features to a single channel, generating a preliminary occlusion map. ,in, , , These are batch size, number of channels, and width, respectively. Convolution is used for local refinement to further improve the ability to perceive overlapping regions, and the occlusion probability is obtained by normalizing it to the range of [0, 1] using the Sigmoid function. Then, the occlusion map is adaptively averaged along the height and width dimensions to generate the occlusion distribution in the height direction. Distribution of occlusion in the width direction Attention weights are normalized using the Sigmoid function and then combined with the occlusion distribution to adjust the attention weights, thereby reducing the attention weights in overlapping regions and enhancing attention to non-overlapping target regions. In step S14, the quantization method is to convert the model weights from floating-point numbers to integers, perform format conversion and quantization processing on the trained model, and generate a quantized model suitable for NPU operation. S2: The user sends natural language text, the cloud server receives the text information, uses a large language model to perform natural language interaction to generate language response information, and performs action decomposition to generate decomposed action instructions; the language response information is returned to the user through the cloud server and the decomposed action instructions are sent to the edge computing device, so that the decomposed action instructions are combined with the target object depth information obtained in step S1 to obtain the final action instruction sequence. S3: Send the sequence of action instructions generated in step S2 to the IPC, and the IPC controls the mobile grasping robot to complete the movement and grasping related actions.

2. The intelligent mobile grasping robot control method based on modular architecture as described in claim 1, characterized in that, The specific steps of preprocessing and data augmentation in step S12 are as follows: automatically remove the background from some images, extract the target element map, and use random transformation and background synthesis to generate diverse synthetic images and corresponding annotation files.

3. The intelligent mobile grasping robot control method based on modular architecture as described in claim 1, characterized in that, Step S2 specifically includes the following steps: S21: The user inputs natural language text information through the public platform. The message interface of the public platform sends the instruction to the cloud server, and the cloud server receives and preprocesses it. S22: Call the large language model to perform natural language processing on the text information obtained in step S21. Utilize the semantic understanding capability of the large language model, generate language response information for natural language interaction based on CoT and few-shot prompting methods, and decompose actions to generate executable action instruction sequences. S23: Call the passive reply interface of the public platform to return the language reply information generated in step S22 to the user terminal; S24: The sequence of action instructions generated in step S22 is sent to the edge computing device through the cloud server, so that the decomposed action instructions are combined with the target category, bounding box and depth information of the target object obtained in step S1 to construct a complete sequence of action instructions. S24: For the issues of not detecting the target object in step S1 and the incorrect parameter of the instruction sequence generated in step S24, the message is sent to the cloud server through the edge computing device, and the cloud server actively sends it to the user terminal by calling the template message interface of the public platform.

4. The intelligent mobile grasping robot control method based on modular architecture as described in claim 1, characterized in that, Step S3 specifically includes the following steps: S31: Write the control program to realize the path planning and motion control of the mobile grasping robot; S32: The sequence of action instructions generated in step S2 is transmitted from the edge computing device to the IPC, and the IPC controls the mobile grasping robot to complete the movement and grasping related actions.

5. A modular architecture-based intelligent mobile grasping robot control system, characterized in that, The system comprising, using the method of any one of claims 1-4: The vision module acquires image information of the target object in real time, processes image data in real time, identifies the target object and outputs the position and category of the target object, and obtains the depth information of the detected target object. The language module involves the user sending natural language text, the cloud server receiving the text information, using a large language model to generate language response information through natural language interaction, and performing action decomposition to generate decomposed action instructions; the language response information is returned to the user through the cloud server; the decomposed action instructions are sent to the edge computing device, where they are combined with the target object depth information obtained by the vision module to obtain the final action instruction sequence. The motion module sends the sequence of motion instructions generated by the language module to the IPC, which then controls the mobile grasping robot to complete the movement and grasping actions.

6. The intelligent mobile grasping robot control system based on a modular architecture as described in claim 5, characterized in that, The vision module includes: The image acquisition unit collects images through web crawlers and camera capture, manually annotates each image, and generates an annotation file for target detection. The image preprocessing unit performs preprocessing and data augmentation operations on the raw images generated by the image acquisition unit, expands the dataset, and divides the dataset into training set, validation set and test set. The depth information acquisition unit uses a depth camera to acquire RGB images and depth maps of the target object in real time; The model training unit selects an improved YOLOv10 model and trains it on a cloud server. After training, a model file is obtained. The model file is quantized to obtain a quantized model, which is then deployed to an edge computing device for localized operation. Simultaneously, based on the quantized model, RGB images are detected in real time to obtain the bounding box and classification label of the target object. The depth value of the object is obtained by combining the depth map, and the detection results are output.

7. The intelligent mobile grasping robot control system based on a modular architecture as described in claim 5, characterized in that, The language module includes: The input unit allows users to input natural language text information through the public platform. The public platform's message interface sends the instructions to the cloud server, which then receives and preprocesses them. The processing unit calls the large language model to perform natural language processing on the text information, utilizes the semantic understanding capability of the large language model, generates language response information for natural language interaction based on CoT and few-shot prompting methods, and performs action decomposition to generate executable action instruction sequences. The interaction unit calls the passive response interface of the public platform to return the language response information obtained by the processing unit to the user; The action unit is decomposed, and the sequence of action instructions obtained by the processing unit is sent from the cloud server to the edge computing device. The decomposed action instructions are combined with the target category, bounding box and depth information of the target object obtained by the vision module to construct a complete sequence of action instructions. The error handling unit will send the error message to the cloud server via the edge computing device for issues such as the vision module failing to detect the target object or errors in the instruction sequence parameters generated by the decomposition action unit. The cloud server will then call the template message interface of the public platform to actively send the message to the user.

8. The intelligent mobile grasping robot control system based on a modular architecture as described in any one of claims 5-7, characterized in that, The action module includes: Control unit: Write control programs to realize path planning and motion control of the mobile grasping robot; The communication unit transmits the sequence of action commands generated by the language module to the IPC via the edge computing device. The IPC then controls the mobile grasping robot to complete the movement and grasping actions.

Images