Intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization

Abstract

This invention belongs to the fields of robotics and machine vision technology, and discloses an intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization. It includes a mobile unit and an intelligent visual guidance and grasping unit. Through a 3D visual positioning submodule, it achieves binocular visual positioning based on target recognition and feature segmentation, high-precision 3D positioning of target objects, and wide-area operation control. In the modal interaction submodule, a semantic-geometric bidirectional coupling cross-modal closed-loop optimization method is proposed: environmental physical feasibility is modeled using a 3D availability tensor field, and the attention of the language model is modulated to achieve geometric constraints on semantics; visual feature extraction is guided and joint reconstruction of the spatial and frequency domains is completed through task semantic topology modeling and topology matrix modulation; and closed-loop optimization of semantic and visual features is achieved through a cross-modal consistency loss function. Ultimately, the robot is driven to perform tasks, achieving closed-loop intelligent control of perception, understanding, and execution.

Description

Technical Field

[0001] This invention relates to the fields of robotics and machine vision, and in particular to an intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization. Background Technology

[0002] Industrial robots are multi-jointed manipulators or multi-degree-of-freedom machines widely used in industrial fields. With their high levels of automation, high production efficiency, and high safety, they are widely applied in various industrial sectors such as electronics, logistics, and chemicals. As intelligent manufacturing advances, the operational requirements for industrial robots are becoming increasingly precise, and the dynamic changes in the working environment are becoming more complex and frequent. They are gradually evolving from the traditional operation mode of pre-programmed, fixed workstations towards intelligent operation with environmental perception, autonomous decision-making, and flexible movement capabilities.

[0003] Traditional industrial robots are mostly fixed in place, and their workspace is limited by the length of the robotic arm and the range of motion of the joints, making it difficult to adapt to the needs of large-scale, multi-workstation operations. They also lack flexibility when facing cross-workstation operations or flexible production layouts. Furthermore, they can only execute preset programs, making them ill-suited to the complex demands of flexible production lines with their cumbersome operations and dynamic process adjustments. To address the limited workspace of robotic arms, the industry has introduced mobile platforms such as AGVs (Automated Guided Vehicles) to combine with robotic arms, expanding their workspace. However, in complex industrial scenarios, existing mobile robots lack the ability to perceive, understand, and respond in real time to unknown or dynamic environments, and the challenge of autonomous operation in dynamic environments remains unresolved.

[0004] Currently, some industrial solutions use depth cameras to indirectly locate objects by photographing targets. However, these solutions have the following shortcomings in practical applications: First, they lack robustness in complex environments; the system's recognition and positioning accuracy significantly decreases when lighting changes, occlusion occurs, or the background becomes cluttered. Second, they rely heavily on repetitive manual teaching, resulting in a large workload, low efficiency, and difficulty in adapting to rapid switching between different production lines. Third, they lack the ability to autonomously perceive and make decisions in dynamically changing environments, failing to achieve flexible and intelligent operation in complex and ever-changing industrial scenarios. Therefore, research on intelligent industrial robot technology based on 3D visual positioning and modal interaction is urgently needed.

[0005] A smart industrial robot (patent number CN202223231827.2) achieves left-right translation, lifting, and overall movement of the robot body by setting up a slider-moving groove mechanism driven by a first telescopic rod, a moving column lifting mechanism driven by a third telescopic rod, and bottom rollers. This effectively solves the problems of limited working range and difficulty in grasping workpieces at distant or different locations by traditional fixed robots. However, the above technology has not yet given the industrial robot autonomous navigation capabilities, and the system lacks real-time environmental perception, autonomous positioning and navigation, and task planning capabilities, thus failing to achieve intelligent operation.

[0006] "An Intelligent Industrial Robot" (Patent No. CN202320749654.9) provides an industrial robot system integrated into an AGV. Through a built-in power supply system and the combination of a mobile chassis and a robotic arm, the robot achieves free movement and rapid positioning within a large area of ​​the workshop; it also incorporates a 3D vision camera to achieve material positioning and grasping functions. However, it fails to utilize the vision module for mobile navigation and environmental perception, resulting in insufficient intelligent decision-making capabilities for responding to complex environmental changes. Summary of the Invention

[0007] The technical challenge to be solved by this invention is to address the problems of poor operational stability and low efficiency caused by existing mobile robots relying on indirect target positioning, cumbersome manual teaching operations, and insufficient perception and decision-making capabilities in complex and dynamic industrial environments. The invention proposes to design an intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization.

[0008] This invention constructs a training dataset of target objects encompassing multiple poses and scenarios, and achieves real-time autonomous recognition and fine segmentation of target objects based on deep learning recognition and segmentation models. It combines binocular vision and optical triangulation binocular localization methods to achieve high-precision 3D positioning of target objects in space. Through hand-eye calibration and multi-coordinate system transformation algorithms, it enables precise operation control of the robot over a wide area. Furthermore, under an edge computing architecture, this invention introduces a semantic-geometric bidirectional coupled cross-modal closed-loop optimization method: by constructing a 3D availability tensor field, it extracts physical feasibility information from the environment and uses it to modulate the attention distribution of the language model; by performing structured parsing of natural language to generate a task topology matrix, it conditionally modulates the visual feature extraction process and achieves joint perception reconstruction in the spatial and frequency domains; by constructing a cross-modal consistency loss function, it achieves closed-loop optimization between language semantics and visual features, thereby improving the system's autonomous decision-making ability and execution reliability in complex dynamic environments.

[0009] This enables closed-loop intelligent control of the robot throughout the entire process, from environmental perception and semantic understanding to task execution, thereby improving the flexibility, autonomy, and intelligence of industrial robots in complex scenarios.

[0010] The technical solution of this invention is as follows: An intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization, comprising: The mobile unit is used to enable the overall movement of the robot. A robotic arm, the base of which is fixedly connected to the moving unit; The end effector unit is fixed to the robotic arm and includes an end gripper and a robot end effector connected to it. The 3D camera is fixed to the end effector in a "eye in hand" manner and moves with the robotic arm; An edge computing module is integrated inside the 3D camera; the edge computing module integrates a 3D visual positioning submodule and a modal interaction submodule. The communication module is used to enable wireless communication between the mobile unit, the robotic arm, the 3D camera, and the edge computing module.

[0011] The robotic arm, communication module, end effector, robot end effector, 3D camera, and edge computing module form an intelligent vision guidance and grasping unit.

[0012] Through the collaboration of mobile units and intelligent vision guidance and grasping units, industrial-grade edge cross-modal real-time closed-loop control is achieved, encompassing the entire process from perception and understanding to execution.

[0013] The 3D visual positioning submodule includes the following steps: S1. Establish an image dataset; collect images including all poses of the target object, the entire working range of the 3D camera, and all complex environments of the industrial scene. Perform image enhancement, classification labeling, and feature extraction on the collected images, and divide the feature-extracted images and classification labeling data into training set, test set, and validation set according to the proportion to obtain the image dataset. S2. The deep learning recognition model is trained using an image dataset. The deep learning recognition model is used to automatically identify and locate objects based on the input image or video stream and generate corresponding recognition boxes. S3. The deep learning segmentation model is trained using the recognition box, and the deep learning segmentation model is used to perform fine feature local segmentation of the content of the recognition box; S4. Perform two-stage object localization based on a progressive fine-tuning localization algorithm that combines macroscopic target recognition and local feature segmentation, including preliminary localization and fine localization. The preliminary positioning is as follows: A deep learning recognition model is used to perform macroscopic target recognition on the left and right camera images of a 3D camera and generate recognition boxes. The centers of the recognition boxes are matched with each other, and the macroscopic targets are initially located using an optical triangulation binocular localization method. The principle of the optical triangulation binocular localization method is to calculate the three-dimensional coordinates by combining the disparity of the same point in the left and right camera images with the binocular localization matrix. in, , , Here are the coordinates of the target object in the right camera coordinate system. , , This represents the position of the target object in the left camera coordinate system. r ij These represent the axes of the left camera coordinate system mapped to the right camera coordinate system. i =1, 2, 3; j =1, 2, 3; This represents the translation of the left camera coordinate system along each x-axis within the right camera coordinate system. This represents the amount of translation of the left camera coordinate system along the y-axis within the right camera coordinate system. The matrix represents the translation of the left camera coordinate system along the z-axis in the right camera coordinate system. sum matrix These represent the coordinate system of the left camera. and right camera coordinate system The rotation matrix and left camera coordinate system of the transformation relationship between them and right camera coordinate system The translation vector that represents the transformation relationship between them is obtained directly through the camera calibration process; Precise positioning specifically refers to: Using the center point of the recognition box as the center, the left and right camera images are cropped, and a deep learning segmentation model is used for fine-grained local feature segmentation. The centroid coordinates of the feature mask obtained from the local segmentation are calculated. The centroid calculation method of the feature mask is as follows: in, A This represents the area of ​​the feature mask polygon. n This represents the number of vertices in the feature mask polygon. x k , y k ) represents the feature mask polygon of the first k The coordinates of the vertices, G x , G y Representing the center of gravityx The coordinates of the orientation in the image coordinate system, and the centroid y The coordinates of the orientation in the image coordinate system; Match the centroids of the left and right camera images and perform optical triangulation binocular positioning to achieve precise target localization and obtain accurate coordinates of the target object in the camera coordinate system. S5. Transform the coordinates of the target object in the camera coordinate system to the robot base coordinate system using a coordinate transformation algorithm to obtain the spatial coordinates of the target object: in, This represents the coordinates of the target object in the robot arm's base coordinate system; , These represent the rotation matrix from the end-effector coordinate system to the base coordinate system of the robotic arm, and the translation vector from the end-effector coordinate system to the base coordinate system of the robotic arm, respectively. These are calculated by combining the current joint angles of the robotic arm with the DH table of the robotic arm. , These represent the rotation matrix from the camera coordinate system to the robot arm end effector coordinate system and the translation vector from the camera coordinate system to the robot arm end effector coordinate system, respectively, obtained through hand-eye calibration. This indicates the coordinates of the target object in the camera coordinate system, obtained through binocular measurement.

[0014] The modal interaction submodule is implemented using a semantic-geometric bidirectional coupling cross-modal closed-loop optimization method, specifically including the following steps: Step 1: Multimodal data acquisition and edge preprocessing; Multimodal input information is acquired from the outside world, including images from the left camera, images from the right camera, and natural language. This information is then transmitted to the modal interaction submodule for data preprocessing. Stereo matching and point cloud reconstruction are performed on the left and right camera images, and speech recognition and text normalization are performed on the natural language to obtain standardized natural language data and visual image data. The modal interaction submodule deploys a deep learning-based affordance prediction model, language model, local language parsing model, topological graph encoding network, visual feature extraction network, closed-loop optimizer, local language parsing model, and local offset generation network. Step 2: The visual image data is used to construct a three-dimensional availability tensor field based on edge computing using a deep learning-based availability prediction model; Step 3: The three-dimensional availability tensor field is used to generate a linguistic semantic representation through a language model and geometric prior attention modulation based on edge reasoning. ; Step 4: The natural language data is processed by a local language parsing model and a topology graph coding network to construct a task topology based on edge semantic parsing, resulting in a topology matrix T; Step 5: The topology matrix T is subjected to semantic-driven joint reconstruction of visual perception via a visual feature extraction network to generate a visual feature representation. ; Step 6: Cross-modal consistency optimization based on edge closed loop: In the modal interaction submodule, the language semantic representation obtained in step 3 is used. Visual feature representation obtained in step 5 Calculate the cross-modal consistency function: It is a cross-modal consistency function; When cross-modal consistency falls below a threshold, the closed-loop optimizer in the modal interaction submodule is executed locally, along with the language semantic representation. Compared with visual feature representation Based on the differences between them, construct a cross-modal consistency loss function: in, Represents the cross-modal consistency loss function. , Indicates the weighting coefficient. This represents the semantic consistency loss between visual features and language semantic vectors. This represents the function for calculating cosine similarity. Represents geometric consistency loss. This represents the spatial location corresponding to the target action generated by the language model. This represents a feasible location constrained by a three-dimensional availability tensor field. With minimizing the aforementioned consistency loss function as the optimization objective, a gradient-based adaptive optimization algorithm is used to dynamically and jointly update the three-dimensional availability tensor field, geometric prior attention bias matrix, and topological matrix, and steps 3 to 6 are repeated until consistency convergence or the task execution conditions are met. Step 7: Task generation and execution control based on edge decision-making: Based on closed-loop optimization, the latest updated language semantic representation before exiting the loop is used. Visual feature representation The modal interaction submodule generates structured task instructions, which are then sent to the task scheduling module in the edge computing module for parsing. The robot control module then drives the mobile unit and the intelligent vision guidance and grasping unit to perform the task in coordination.

[0015] The construction of the three-dimensional availability tensor field based on edge computing is specifically as follows: The deep learning-based availability prediction model deployed in the modal interaction submodule is invoked to analyze and process the 3D point cloud corresponding to the visual image data, and to construct a 3D availability tensor field of the industrial scene. : in, Represents coordinates in three-dimensional space The three-dimensional availability tensor field defined at that location, n Represents the surface normal vector. Indicates the probability of crawling. This indicates the collision penalty weight.

[0016] The geometric prior attention modulation based on edge reasoning is specifically as follows: The three-dimensional availability tensor field is mapped to the language model decoding space in the modal interaction submodule, and a geometric prior attention bias matrix B is constructed to modulate the attention calculation process of the language model. Its definition is: in, This represents the element in the i-th row and j-th column of the geometric prior attention bias matrix. Represents a mapping function based on physical feasibility. , Indicates the semantic tagging of candidate actions. This represents the three-dimensional availability tensor field obtained from step 2; During the language model decoding process, the geometric prior attention bias matrix is ​​incorporated into the attention calculation: in, This represents the attention calculation function. Represents the normalization function. Represents the query matrix. Represents the key matrix. Represents a value matrix, This represents the feature dimension of the key vector; The output of attention computation is a context-weighted representation of each semantic tag, which is applied to the generation of the probability distribution of the next action semantics. Action semantics that do not conform to the physical environment constraints of the current industrial scenario are subjected to real-time probability suppression, resulting in a linguistic semantic representation constrained by geometric priors. .

[0017] The specific steps for constructing the task topology based on edge semantic parsing are as follows: The local language parsing model in the modal interaction submodule is invoked to perform structured parsing of standardized natural language data and generate a task semantic topology graph. Among them, nodes The edge E represents the operation relationship between task objects, and the edge weight represents the constraint condition. The task semantic topology graph is transformed into a topology matrix T by a topology graph encoding network.

[0018] The semantically driven joint reconstruction of visual perception specifically refers to: Using the topology matrix T as input, the visual feature extraction network is conditionally modulated, and joint perceptual reconstruction in the spatial and frequency domains is performed simultaneously, including spatial domain perceptual reconstruction and frequency domain adaptive modulation; the visual feature extraction network includes a backbone feature extraction structure and a multi-scale visual feature extraction structure. The spatial domain perception reconstruction specifically refers to: The topology matrix T is input into the local offset generation network to calculate the sampling offset of deformable convolution. : By applying the sampling offset of deformable convolution to the deformable convolutional layer in the backbone feature extraction structure, dynamic reconstruction of the visual receptive field based on language semantics is performed: in, This represents the response value of the output feature map at position p. Indicates the convolution kernel weights, Indicates the input feature map, This represents the k-th sampling position of the convolution kernel. This represents the offset of the corresponding sampling position; The frequency domain adaptive modulation specifically refers to: A semantic gating function is introduced into the multi-scale visual feature extraction structure. It is used to adjust the response of different frequency domain features, perform adaptive allocation of perceptual resources according to task requirements, and achieve collaborative adaptive optimization of visual information in terms of spatial structure and frequency domain features: in, This represents high-frequency features that contain edge and texture information. The low-frequency features that contain spatial structure and contour information are obtained by the multi-scale visual feature extraction structure at different scale layers. Finally, the visual features reconstructed jointly by the spatial and frequency domains are used as the visual feature representation. ; The beneficial effects of the present invention are as follows: In the present invention, based on binocular vision and optical triangulation binocular localization method, the combination of deep learning recognition model and deep learning segmentation model is used to realize the autonomous recognition of target objects, fine feature extraction and high-precision three-dimensional localization, thereby improving the localization accuracy, real-time performance and recognition robustness in complex scenes. In this invention, a three-dimensional availability tensor field is introduced to model the physical constraints in the environment, enabling the robot to judge physical feasibility during task execution. Based on this, a geometric prior attention bias matrix is ​​constructed to modulate the physical constraints of the language model reasoning process, thereby improving the rationality and safety of task planning. In this invention, task semantic topology modeling and topology matrix modulation are used to guide the visual feature extraction process through language semantics, thereby improving the consistency between perception and task objectives. Furthermore, through a joint spatial and frequency domain perception reconstruction mechanism, adaptive optimization of visual features is achieved, thereby improving perception accuracy in complex environments. In this invention, by constructing a cross-modal consistency loss function and performing closed-loop optimization, dynamic consistency correction between language semantics and visual perception is achieved, significantly improving the stability and decision accuracy of the system in dynamic environments. In this invention, a closed-loop intelligent control of the robot is achieved from perception and understanding to execution, thereby improving the flexibility, autonomy and intelligence of industrial robots under complex working conditions. Attached Figure Description

[0019] Figure 1 A schematic diagram of the overall structure of an intelligent industrial robot optimized for 3D visual positioning and cross-modal closed-loop operation.

[0020] Figure 2 This is a flowchart of a progressively refined localization algorithm based on macroscopic target recognition and local feature segmentation.

[0021] Figure 3 A flowchart for 3D visual positioning.

[0022] Figure 4 This is a flowchart of a semantic-geometric bidirectional coupled cross-modal closed-loop optimization method.

[0023] In the diagram: 1-Mobile unit, 2-Robotic arm, 3-Communication module, 4-End gripper, 5-Robot end effector, 6-3D camera, 7-Edge computing module. Detailed Implementation

[0024] The specific embodiments of the present invention are described in detail below with reference to the technical solutions and accompanying drawings.

[0025] The first step is to design and build an intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization. The overall structure of the robot is referenced from... Figure 1 The detailed structure is as follows: This example uses a composite robot with a maximum load of 5kg, an arm span of 1008mm, and a working accuracy of 0.02mm. An electric gripper is used as the robot's end effector 5, which has a gripping force of 45N~160N and a repeatability of 0.02mm. By adding 3D-printed flexible parts and adjusting the gripping force, it can reliably grasp different target objects (including brittle objects such as beakers). The 3D camera 6 is a binocular camera integrating two cameras. The camera sensor has a resolution of 3072×2048, a working distance of 120~900mm, and a pixel size of 2.4μm×2.4μm. This intelligent industrial robot, featuring 3D vision positioning and cross-modal closed-loop optimization, uses an edge computing module 7 as its control terminal and can be controlled in real-time via a computer. The 3D camera 6 outputs the acquired images to the edge computing module 7 for recognition, positioning, and compensation. The edge computing module 7 then outputs the compensated coordinates to the robot's control cabinet via the communication module 3. The control cabinet controls the electric gripper to move to the designated position. Finally, the robot connects to the electric gripper via an 8-pin aviation connector to achieve I / O control and thus realize target grasping.

[0026] The second step is to construct a 3D visual positioning submodule. The positioning process is referenced below. Figure 3 The detailed positioning method is as follows: First, a dataset is constructed. In this example, it includes a series of target objects such as complex machining tools, tool holders, buttons of various colors, placement areas, seven gestures, beakers, and round holes, ensuring the diversity of image types in the dataset. Images are acquired with a 2° gradient, ranging from a 45° overhead view to a 45° upward view, ensuring the diversity of image perspectives in the dataset. Images are also acquired with a 0.1m gradient, ranging from 0.2m to 4m, ensuring the dataset covers the entire working area, ensuring the diversity of image distances in the dataset. To enhance the generalization of the training model, the dataset images feature various complex scenes, using multiple shooting tools and employing multiple resolutions such as 720P, 1080P, 2K, and 4K. Then, deep learning recognition and segmentation models were trained. The Labelme plugin was used to label the maximum bounding box of the target object and to perform polygon segmentation on the object's outline. The collected images and labeling information were divided into training, testing, and validation sets in an 8:1:1 ratio. After the dataset was constructed, YOLO v8x and FCN models were used for deep learning training in the CUDA 12.1 environment. The training effect was optimized by setting training parameters to obtain deep learning recognition and segmentation models for the target object. Both the deep learning recognition and segmentation models were existing, readily available deep learning models. Subsequently, a progressively refined localization algorithm based on macroscopic target recognition and local feature segmentation is constructed. Frame-by-frame, the left and right camera images from the binocular camera are input into the trained deep learning recognition and segmentation models, allowing them to divide the images into a grid. By comparing the camera calibration results with the disparity of the two sets of images, the 3D coordinates of the target point are calculated using the optical triangulation binocular localization method. Then, the original image is cropped according to the recognition bounding box, and the above recognition and segmentation operations are repeated. The cropped image, after removing most of the complex background interference, undergoes secondary recognition and segmentation to achieve high-precision online localization of the target object's grasping position. The flowchart of the progressively refined localization algorithm based on macroscopic target recognition and local feature segmentation is provided below. Figure 2 .

[0027] Finally, a coordinate transformation algorithm is constructed. By performing hand-eye calibration in advance, a fixed relative relationship between the camera coordinate system and the robotic arm end-effector coordinate system is obtained. The transformation relationship between the robotic arm end-effector coordinate system and the robotic arm base coordinate system is obtained by solving the DH table. Finally, a coordinate transformation algorithm is constructed by using a multi-coordinate system serial transformation formula in the form of a homogeneous transformation matrix. This enables inverse kinematics calculation in the robotic arm base coordinate system, thereby transforming the spatial position of the target object in the camera coordinate system to the robotic arm base coordinate system.

[0028] The third step is to build the modal interaction submodule. See the method flow for reference. Figure 4 The detailed modal interaction methods are as follows: First, multimodal data acquisition and edge preprocessing are performed; Multimodal input information is acquired from the outside world, including images from the left and right cameras and natural language data. For the visual data, a binocular stereo matching method is used to calculate disparity, and combined with camera intrinsic parameters to complete the reconstruction of 3D point cloud, resulting in point cloud data containing spatial location and color information. For the natural language data, speech recognition and text normalization processing are performed to obtain a standardized natural language sequence. Then, a forward constraint mechanism is implemented to apply visual geometric information to the language semantic generation process; A 3D availability tensor field is constructed by inputting the 3D point cloud into a deep learning-based availability prediction model (e.g., an availability prediction model based on PointNet++), performing physical feasibility analysis on each spatial location in the scene, and constructing the 3D availability tensor field. in, Represents coordinates in three-dimensional space The three-dimensional availability tensor field defined at that location, n Represents the surface normal vector. Indicates the probability of crawling. Indicates the collision penalty weight; This three-dimensional availability tensor field is used to describe the distribution of operational feasibility at various locations in an industrial scenario; Geometric prior attention modulation is performed to map the three-dimensional availability tensor field to the language model decoding space, constructing the geometric prior attention bias matrix B: in, The expression represents the first element in the geometric prior attention bias matrix. i Line number j Column elements, Represents a mapping function based on physical feasibility. , Indicates the semantic tagging of candidate actions. This represents the three-dimensional availability tensor field obtained from step 2; And incorporate it into the attention calculation process: in, This represents the attention calculation function. Represents the normalization function. Represents the query matrix. Represents the key matrix. Represents a value matrix, The feature dimension of the key vector. This represents the geometric prior attention bias matrix obtained from step 3; The output of attention computation is a context-weighted representation of each semantic tag, which is applied to the probability distribution generation of the next action semantics. In this way, the language model is physically constrained when generating action semantics, thereby suppressing semantic outputs that do not conform to actual operational conditions, resulting in a language semantic representation constrained by geometric priors. .

[0029] Subsequently, a reverse modulation mechanism of language semantics on the visual feature extraction process is implemented; Perform task semantic topology construction by performing structured parsing of natural language to generate a task semantic topology graph: Among them, nodes The edge E represents the operation relationship between task objects, and the edge weight represents the constraint condition. Furthermore, the semantic topology graph of this task is transformed into a topology matrix T through a topology coding network for use in the subsequent visual modulation process; Semantic-driven visual joint reconstruction is performed. Based on the topology matrix T, the visual feature extraction network is conditionally modulated to achieve joint reconstruction in the spatial and frequency domains, which includes spatial domain perceptual reconstruction and frequency domain adaptive modulation. The visual feature extraction network includes a backbone feature extraction structure and a multi-scale visual feature extraction structure. Spatial domain perception reconstruction specifically refers to: Input the topology matrix T into the local offset generation network to obtain the sampling offset of the deformable convolution: By applying the sampling offset of deformable convolution to the deformable convolutional layer in the backbone feature extraction structure, dynamic reconstruction of the visual receptive field based on language semantics is performed: in, This represents the response value of the output feature map at position p. Indicates the convolution kernel weights, Indicates the input feature map, This represents the k-th sampling position of the convolution kernel. This represents the offset of the corresponding sampling position; The frequency domain adaptive modulation specifically refers to: A semantic gating function G_s is introduced into the multi-scale visual feature extraction structure to adjust the response of different frequency domain features, perform adaptive allocation of perceptual resources according to task requirements, and achieve collaborative adaptive optimization of visual information in spatial structure and frequency domain features: Multi-scale features are weighted and fused using a semantic gating function: in, This represents high-frequency features that contain edge and texture information. This represents low-frequency features containing spatial structure and contour information, both of which are extracted from different scale layers by the multi-scale visual feature extraction structure. Visual features reconstructed jointly by the spatial and frequency domains are used as visual feature representations. ; Finally, cross-modal closed-loop consistency optimization is performed; Based on language semantic representation Visual feature representation Calculate the cross-modal consistency function: When consistency falls below a set threshold, a cross-modal consistency loss function is constructed: in, Represents the cross-modal consistency loss function. , Indicates the weighting coefficient. This represents the semantic consistency loss between visual features and language semantic vectors. This represents the function for calculating cosine similarity. This represents the visual feature representation after joint perception reconstruction. This represents the semantic representation of language after physical constraints. Represents geometric consistency loss. This represents the spatial location corresponding to the target action generated by the language model. This represents a feasible location constrained by a three-dimensional availability tensor field. With the goal of minimizing this loss, the three-dimensional availability tensor field, geometric prior attention bias matrix and topological matrix are jointly optimized, and the parameters are updated based on the adaptive gradient optimization method to form a closed-loop optimization process; the forward constraint mechanism of visual geometric information on the language semantic generation process and the reverse modulation mechanism of language semantics on the visual feature extraction process are iteratively executed until consistent convergence or the task execution conditions are met. Task generation and execution control are performed based on optimized language semantic representation. Visual feature representation The system generates structured task instructions and sends them to the robot control module. The control module then drives the mobile unit and the grasping unit to perform the task in coordination and dynamically corrects the execution results through visual feedback.

Claims

1. An intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization, characterized in that, include: The mobile unit (1) is used to realize the overall movement of the robot; The robotic arm (2) has its base fixed to the moving unit (1); The end effector unit is fixed to the robotic arm (2) and includes an end gripper (4) and a robot end effector (5) connected to the end effector. The 3D camera (6) is fixed to the end gripper (4) in an "eye in hand" manner and moves with the robotic arm (2); An edge computing module (7) is integrated inside the 3D camera (6); a 3D visual positioning submodule and a modal interaction submodule are integrated on the edge computing module (7); The communication module (3) is used to realize wireless communication between the mobile unit (1), the robotic arm (2), the 3D camera (6) and the edge computing module (7); The modal interaction submodule is implemented using a semantic-geometric bidirectional coupling cross-modal closed-loop optimization method, specifically including the following steps: Step 1: Multimodal data acquisition and edge preprocessing; Multimodal input information is acquired from the outside world, including images from the left camera, images from the right camera, and natural language. This information is then transmitted to the modal interaction submodule for data preprocessing. Stereo matching and point cloud reconstruction are performed on the left and right camera images, and speech recognition and text normalization are performed on the natural language to obtain standardized natural language data and visual image data. The modal interaction submodule deploys a deep learning-based affordance prediction model, language model, local language parsing model, topological graph encoding network, visual feature extraction network, closed-loop optimizer, local language parsing model, and local offset generation network. Step 2: The visual image data is used to construct a three-dimensional availability tensor field based on edge computing using a deep learning-based availability prediction model; Step 3: The three-dimensional availability tensor field is used to generate a linguistic semantic representation through a language model and geometric prior attention modulation based on edge reasoning. ; Step 4: The natural language data is processed by a local language parsing model and a topology graph coding network to construct a task topology based on edge semantic parsing, resulting in a topology matrix T; Step 5: The topology matrix T is subjected to semantic-driven joint reconstruction of visual perception via a visual feature extraction network to generate a visual feature representation. ; Step 6: Cross-modal consistency optimization based on edge closed loop: In the modal interaction submodule, the language semantic representation obtained in step 3 is used. Visual feature representation obtained in step 5 Calculate the cross-modal consistency function: It is a cross-modal consistency function; When cross-modal consistency falls below a threshold, the closed-loop optimizer in the modal interaction submodule is executed locally, along with the language semantic representation. Compared with visual feature representation Based on the differences between them, construct a cross-modal consistency loss function: in, Represents the cross-modal consistency loss function. , Indicates the weighting coefficient. This represents the semantic consistency loss between visual features and language semantic vectors. This represents the function for calculating cosine similarity. Represents geometric consistency loss. This represents the spatial location corresponding to the target action generated by the language model. This represents a feasible location constrained by a three-dimensional availability tensor field. With minimizing the aforementioned consistency loss function as the optimization objective, a gradient-based adaptive optimization algorithm is used to dynamically and jointly update the three-dimensional availability tensor field, geometric prior attention bias matrix, and topological matrix, and steps 3 to 6 are repeated until consistency convergence or the task execution conditions are met. Step 7: Task generation and execution control based on edge decision-making: Based on closed-loop optimization, the latest updated semantic representation of the language before exiting the loop is optimized. Visual feature representation In the modal interaction submodule, structured task instructions are generated and sent to the task scheduling module in the edge computing module (7) for parsing. The robot control module drives the mobile unit (1) and the intelligent vision guidance and grasping unit to perform the operation in coordination.

2. The intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization according to claim 1, characterized in that, The robotic arm (2), communication module (3), end effector (4), robot end effector (5), 3D camera (6), and edge computing module (7) form an intelligent visual guidance and grasping unit.

3. The intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization according to claim 2, characterized in that, Through the collaboration of the mobile unit (1) and the intelligent vision guidance and grasping unit, industrial-grade edge cross-modal real-time closed-loop control is achieved from perception and understanding to execution.

4. The intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization according to claim 1, characterized in that, The 3D visual positioning submodule includes the following steps: S1. Establish an image dataset; collect images including all poses of the target object, the entire working range of the 3D camera (6), and all complex environments of the industrial scene. Perform image enhancement, classification labeling, and feature extraction on the collected images, and divide the feature-extracted images and classification labeling data into training set, test set, and validation set according to the proportion to obtain the image dataset. S2. The deep learning recognition model is trained using an image dataset. The deep learning recognition model is used to automatically identify and locate objects based on the input image or video stream and generate corresponding recognition boxes. S3. The deep learning segmentation model is trained using the recognition box, and the deep learning segmentation model is used to perform fine feature local segmentation of the content of the recognition box; S4. Perform two-stage object localization based on a progressive fine-tuning localization algorithm that combines macroscopic target recognition and local feature segmentation, including preliminary localization and fine localization. The preliminary positioning is as follows: A deep learning recognition model is used to perform macroscopic target recognition on the left and right camera images of the 3D camera (6) and generate recognition boxes. The centers of the recognition boxes are matched with each other, and the macroscopic target is initially located by the optical triangulation binocular positioning method. The principle of the optical triangulation binocular positioning method is to obtain the three-dimensional coordinates by combining the disparity of the same point in the left and right camera images with the binocular positioning matrix. in, , , Here are the coordinates of the target object in the right camera coordinate system. , , This represents the position of the target object in the left camera coordinate system. r ij These represent the axes of the left camera coordinate system mapped to the right camera coordinate system. i =1, 2, 3; j =1, 2, 3; This represents the translation of the left camera coordinate system along each x-axis within the right camera coordinate system. This represents the amount of translation of the left camera coordinate system along the y-axis within the right camera coordinate system. The matrix represents the translation of the left camera coordinate system along the z-axis in the right camera coordinate system. sum matrix These represent the coordinate system of the left camera. and right camera coordinate system The rotation matrix and left camera coordinate system of the transformation relationship between them and right camera coordinate system The translation vector that transforms the relationship between them is obtained directly through the camera calibration process; Precise positioning specifically refers to: Using the center point of the recognition box as the center, the left and right camera images are cropped, and a deep learning segmentation model is used for fine-grained local feature segmentation. The centroid coordinates of the feature mask obtained from the local segmentation are calculated. The centroid calculation method of the feature mask is as follows: in, A This represents the area of ​​the feature mask polygon. n This represents the number of vertices in the feature mask polygon. x k , y k ) represents the feature mask polygon of the first k The coordinates of the vertices, G x , G y Representing the center of gravity x The coordinates of the orientation in the image coordinate system, and the centroid y The coordinates of the orientation in the image coordinate system; Match the centroids of the left and right camera images and perform optical triangulation binocular positioning to achieve precise target localization and obtain accurate coordinates of the target object in the camera coordinate system. S5. Transform the coordinates of the target object in the camera coordinate system to the robot base coordinate system using a coordinate transformation algorithm to obtain the spatial coordinates of the target object: in, This represents the coordinates of the target object in the robot arm's base coordinate system; , These represent the rotation matrix from the end-effector coordinate system to the base coordinate system of the robotic arm, and the translation vector from the end-effector coordinate system to the base coordinate system of the robotic arm, respectively. These are calculated by combining the current joint angles of the robotic arm with the DH table of the robotic arm. , These represent the rotation matrix from the camera coordinate system to the robot arm end effector coordinate system and the translation vector from the camera coordinate system to the robot arm end effector coordinate system, respectively, obtained through hand-eye calibration. This indicates the coordinates of the target object in the camera coordinate system, obtained through binocular measurement.

5. The intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization according to claim 1, characterized in that, The construction of the three-dimensional availability tensor field based on edge computing is specifically as follows: The deep learning-based availability prediction model deployed in the modal interaction submodule is invoked to analyze and process the 3D point cloud corresponding to the visual image data, and to construct a 3D availability tensor field of the industrial scene. : in, Represents coordinates in three-dimensional space The three-dimensional availability tensor field defined at that location, n Represents the surface normal vector. Indicates the probability of crawling. This indicates the collision penalty weight.

6. The intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization according to claim 5, characterized in that, The geometric prior attention modulation based on edge reasoning is specifically as follows: The three-dimensional affordance tensor field is mapped to the language model decoding space in the modal interaction submodule to construct a geometric prior attention bias matrix. B The attention calculation process used to modulate the language model is defined as follows: in, This represents the element in the i-th row and j-th column of the geometric prior attention bias matrix. Represents a mapping function based on physical feasibility. , Indicates the semantic tagging of candidate actions, Represents a three-dimensional availability tensor field; During the language model decoding process, the geometric prior attention bias matrix is ​​incorporated into the attention calculation: in, This represents the attention calculation function. Represents the normalization function. Represents the query matrix. Represents the key matrix, Represents a value matrix, This represents the feature dimension of the key vector; The output of attention computation is a context-weighted representation of each semantic tag, which is applied to the generation of the probability distribution of the next action semantics. Action semantics that do not conform to the physical environment constraints of the current industrial scenario are subjected to real-time probability suppression, resulting in a linguistic semantic representation constrained by geometric priors. .

7. The intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization according to claim 1, characterized in that, The specific steps for constructing the task topology based on edge semantic parsing are as follows: The local language parsing model in the modal interaction submodule is invoked to perform structured parsing of standardized natural language data and generate a task semantic topology graph. Among them, nodes The edge E represents the operation relationship between task objects, and the edge weight represents the constraint condition. The task semantic topology graph is transformed into a topology matrix T by a topology graph encoding network.

8. The intelligent industrial robot with 3D visual positioning and cross-modal closed-loop optimization according to claim 5, characterized in that, The semantically driven joint reconstruction of visual perception specifically refers to: Using the topology matrix T as input, the visual feature extraction network is conditionally modulated, and joint perceptual reconstruction in the spatial and frequency domains is performed simultaneously, including spatial domain perceptual reconstruction and frequency domain adaptive modulation; the visual feature extraction network includes a backbone feature extraction structure and a multi-scale visual feature extraction structure. The spatial domain perception reconstruction specifically refers to: The topology matrix T is input into the local offset generation network to calculate the sampling offset of deformable convolution. : By applying the sampling offset of deformable convolution to the deformable convolutional layer in the backbone feature extraction structure, dynamic reconstruction of the visual receptive field by language semantics is performed: in, This represents the response value of the output feature map at position p. Indicates the convolution kernel weights. Indicates the input feature map, This represents the k-th sampling position of the convolution kernel. This represents the offset of the corresponding sampling position; The frequency domain adaptive modulation specifically refers to: A semantic gating function is introduced into the multi-scale visual feature extraction structure. It is used to adjust the response of different frequency domain features, perform adaptive allocation of perceptual resources according to task requirements, and achieve collaborative adaptive optimization of visual information in terms of spatial structure and frequency domain features: in, This represents high-frequency features that contain edge and texture information. The low-frequency features that contain spatial structure and contour information are obtained by the multi-scale visual feature extraction structure at different scale layers. Finally, the visual features reconstructed jointly by the spatial and frequency domains are used as the visual feature representation. .

Images