Robot target grasping recognition and positioning method and system based on knowledge graph

Abstract

The application discloses a kind of robot grasping target identification and positioning method and system based on knowledge graph, and is constructed by multi-source data integration including object appearance feature and grasping point pose parameter triple knowledge graph, realize structured node network;Further, the environment perception data of robot vision module is acquired, the vector representation of target object is determined after processing, if similarity exceeds threshold value, then use entity linking technology to query matching node, solve recognition ambiguity and obtain accurate result;Based on result retrieval pose parameter in knowledge graph, carry out coordinate conversion and generate fine-grained grasping sequence, combine real-time visual feedback and use dynamic adjustment algorithm to optimize path, and update knowledge graph by simulation evaluation to adapt to new scene.The application highlights the core role of knowledge graph in grasping optimization, improves the accuracy, robustness and self-adaptive ability of robot operation, and is suitable for industrial automation and other fields.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and particularly discloses a robot target grasping recognition and positioning method and system based on a knowledge graph. BACKGROUND

[0002] In modern industry and logistics fields, robot technology has become an important pillar to improve efficiency and reduce costs, especially in automated grasping and carrying tasks, its role cannot be ignored. Robots complete the recognition and grasping of target objects through visual perception and mechanical operation, and have been widely applied to production lines and warehouse management scenarios. However, despite the rapid development of this field, there are still many challenges, and innovative methods are needed to break through existing bottlenecks.

[0003] Currently, the recognition and positioning method of robot grasping targets mainly relies on single visual perception technology, which can still perform well in simple environments, but has obvious shortcomings in complex scenarios. Especially in the case of multi-object stacking, light changes or occlusion, existing methods often have difficulty in accurately distinguishing target objects from the background, and even cannot determine the specific state of the object and the appropriate operation mode. This problem not only stems from the lack of environmental adaptability, but also is caused by the lack of comprehensive understanding and use of multi-dimensional information of objects, which leads to the inability of robots to make flexible responses in dynamic environments.

[0004] More importantly, a key technical difficulty in robot grasping tasks is how to effectively integrate the relevance between the appearance features of objects and the operation parameters. Appearance features refer to the shape, color, texture and other intuitive information of objects, while operation parameters include the position of the grasping point and the adjustment of the robot's posture and other execution details. The relevance between the two has not been fully explored, making it difficult for robots to quickly determine the most appropriate grasping method after identifying the target. For example, in a logistics warehouse, when facing packages of different shapes, the robot may repeatedly try different angles because it cannot accurately determine the grasping point of a package, resulting in low efficiency and even operation failure. This gap from recognition to execution runs through the entire task flow and becomes a technical obstacle that needs to be solved.

[0005] Therefore, how to integrate the relevance between the appearance features of objects and the operation parameters in complex environments to achieve seamless connection from target recognition to accurate grasping has become a key problem in the application of robot automation. SUMMARY

[0006] The present application provides a robot target grasping recognition and positioning method and system based on a knowledge graph, which aims to solve at least one of the defects in the prior art.

[0007] An aspect of the present application relates to a knowledge graph-based robot target recognition and positioning method, comprising the following steps:

[0008] S100, a multi-modal knowledge graph is constructed by multi-source data integration, the multi-modal knowledge graph models the relationship between object appearance features and grasping point pose parameters in the form of triples, and a structured node network containing object appearance description and optimal grasping trajectory is obtained;

[0009] S200, according to the multi-modal knowledge graph, the environment perception data collected by the robot vision module is obtained, the environment perception data is processed by a data processing method, and the vector representation corresponding to the target object is determined;

[0010] S300, if the similarity of the vector representation and the node of the multi-modal knowledge graph exceeds a preset threshold, the matching object node is queried by entity linking technology, the object recognition ambiguity in complex scenes is judged and solved, and the accurate object recognition result is obtained;

[0011] S400, the object recognition result is used to retrieve the grasping point pose parameters in the multi-modal knowledge graph, the related triple data is obtained, and the pose mapping relationship for coordinate conversion is obtained;

[0012] S500, the coordinate conversion operation is performed through the pose mapping relationship, the perception coordinate system parameter is converted into the robot coordinate system value, and the fine-grained grasping operation sequence is determined;

[0013] S600, combining the fine-grained grasping operation sequence and the real-time visual feedback data, the object posture deviation is processed by a dynamic adjustment algorithm, and the optimized grasping execution path is obtained;

[0014] S700, based on the optimized grasping execution path, the robot operation environment is simulated, the path effectiveness is evaluated, and the multi-modal knowledge graph is updated to adapt to new scenes.

[0015] Further, step S100 comprises:

[0016] S110, the appearance feature data of the target object is obtained from the image sensor and the depth sensor by a multi-source data acquisition tool, the pre-processing operation is performed on the appearance feature data, the denoising tool is used to smooth the pre-processed appearance feature data, and the clear appearance feature description information is obtained;

[0017] S120, according to the appearance feature description information, the three-dimensional reconstruction tool is used to generate the three-dimensional structure data of the target object, when the three-dimensional structure data is generated, if the data missing area is detected, the interpolation tool is used to complete the data missing area, and the complete three-dimensional structure description is determined;

[0018] S130, for the stereoscopic structure description, the path planning tool is used to calculate the grasping point pose parameters, a plurality of candidate grasping trajectory points corresponding to the stereoscopic structure description are obtained, the candidate grasping trajectory points are sorted by comparison tool, and the optimal grasping trajectory parameters are judged out;

[0019] S140, the appearance feature description information is associated and mapped with the optimal grasping trajectory parameters through the knowledge graph construction tool, a triple relationship network containing structured nodes is generated, and a multi-modal knowledge graph is obtained.

[0020] Further, step S200 comprises:

[0021] S210, the visual module is used to obtain the original perception data stream from the environment, the denoising tool is used for preliminary processing on the original perception data stream, and the cleaned basic perception data is obtained;

[0022] S220, according to the basic perception data, the contour information of the target object is separated out by using the feature extraction tool, if the clarity of the contour information is lower than the preset threshold, the enhancement tool is used for optimization, and the complete contour description information is determined;

[0023] S230, for the contour description information, the data fusion tool is used to generate the spatial position data of the target object in combination with the depth information, the corresponding three-dimensional coordinate information is obtained, and the preliminary vector representation of the target object is judged out;

[0024] S240, the coordinate mapping tool is used for calibration processing on the preliminary vector representation, if the deviation of the calibrated vector representation is detected to be greater than the preset threshold during calibration, the adjustment tool is used for correction, and the vector representation form corresponding to the target object is obtained.

[0025] Further, step S300 comprises:

[0026] S310, according to the vector representation form, at least one candidate node data is obtained from the pre-established multi-modal knowledge graph, the feature comparison tool is used for preliminary screening, and a node set close to the characteristics of the target object is obtained;

[0027] S320, if there are a plurality of similar nodes in the node set, the data fusion tool is used to combine the environmental context information to perform secondary filtering on the node set, and a single node data matched with the current scene is determined;

[0028] S330, the link verification tool is used to detect the association reliability of the single node data and the target object, if the association reliability of the single node data and the target object is lower than the preset threshold, the standby node data is obtained from the multi-modal knowledge graph for supplementary comparison, and the final matching node result is judged out;

[0029] S340, generating an identification label of the target object by using a mapping tool according to the final matching node result, and correcting the identification label by using a multi-modal data consistency verification tool to obtain an accurate object identification result.

[0030] Further, the step S400 includes:

[0031] S410, extracting at least one related grasping point pose parameter data from the multi-modal knowledge graph according to the object identification result, and preliminarily classifying the grasping point pose parameter data by using a data screening tool to obtain a pose parameter set matched with the target object;

[0032] S420, if there are multiple candidate data in the pose parameter set, refining and filtering the pose parameter set by using a context environment matching tool combined with scene information to determine a single grasping point pose parameter data matched with the current grasping task;

[0033] S430, for the single grasping point pose parameter data, obtaining corresponding triple data from the multi-modal knowledge graph by using a triple extraction tool, and structuring the triple data by using a data mapping tool to obtain a preliminary pose mapping relationship for coordinate conversion;

[0034] S440, correcting the preliminary pose mapping relationship by using a consistency verification tool, and if the verification result does not match a preset threshold, obtaining backup data from the multi-modal knowledge graph for supplementary comparison to determine an accurate pose mapping relationship.

[0035] Further, the step S500 includes:

[0036] S510, performing a coordinate conversion operation by using the pose mapping relationship to convert the perception coordinate system parameter into a robot coordinate system value;

[0037] S520, determining a fine-grained grasping operation sequence according to the converted robot coordinate system value.

[0038] Further, the step S600 includes:

[0039] S610, obtaining real-time visual feedback data by using a visual sensor, continuously capturing images of the target object, and extracting target object features of the real-time visual feedback data by using an image processing tool to obtain preliminary position and attitude information of the target object;

[0040] S620, judging a deviation between an actual attitude and an expected attitude of the target object by using a comparison tool according to the preliminary position and attitude information of the target object combined with a pre-established deviation database, and generating a deviation correction parameter if the deviation exceeds a preset threshold;

[0041] S630, data fusion is performed on the bias correction parameter and the fine-grained operation sequence, a path planning tool is used to dynamically adjust a grabbing execution path, and adjusted path node data is determined;

[0042] S640, according to the adjusted path node data, real-time visual feedback data is combined, a data integration tool is used for secondary verification, and an optimized grabbing execution path is obtained.

[0043] Further, step S700 comprises:

[0044] S710, based on the optimized grabbing execution path, an environment simulation tool is used to digitally reconstruct the target operation environment, data extraction is performed on the spatial constraints in the scene, and a virtual operation environment data set is obtained;

[0045] S720, according to the virtual operation environment data set, a path verification tool is used to simulate the running of the grabbing execution path, collision detection is performed on the path nodes, and an executable judgment result of the path is determined;

[0046] S730, if the executable judgment result of the path shows that there is an obstacle, a path correction tool is used to locally adjust the blocked node, real-time monitoring data is combined for verification, and corrected path data is obtained;

[0047] S740, combined with the corrected path data and the scene matching data, a knowledge graph updating tool is used to incrementally adjust the multi-modal knowledge graph, data fusion is performed on the new scene features, and an adaptability updating result of the adjusted multi-modal knowledge graph is determined.

[0048] Another aspect of the application relates to a robot target recognition and positioning system based on a knowledge graph, used to execute the robot target recognition and positioning method based on the knowledge graph, comprising:

[0049] A multi-modal knowledge graph construction module is used to construct a multi-modal knowledge graph through multi-source data integration, the multi-modal knowledge graph models the relationship between object appearance features and grasping point pose parameters in a triple form, and a structured node network containing object appearance description and optimal grasping trajectory is obtained;

[0050] A vector representation form determination module is used to obtain environment perception data collected by a robot vision module according to the multi-modal knowledge graph, process the environment perception data by using a data processing method, and determine the vector representation form corresponding to the target object;

[0051] The object recognition result acquisition module is configured to query a matching object node by using an entity linking technology if the similarity of the vector representation form to the nodes of the multi-modal knowledge graph exceeds a preset threshold, to determine and solve object recognition ambiguity in a complex scene, and to obtain an accurate object recognition result.

[0052] The pose mapping relationship acquisition module is configured to retrieve the grasp point pose parameters in the multi-modal knowledge graph by using the object recognition result, to acquire relevant triple data, and to obtain a pose mapping relationship for coordinate conversion.

[0053] The fine-grained grasp operation sequence determination module is configured to perform a coordinate conversion operation by using the pose mapping relationship, to convert the perception coordinate system parameters into robot coordinate system values, and to determine a fine-grained grasp operation sequence.

[0054] The grasp execution path acquisition module is configured to combine the fine-grained grasp operation sequence and real-time visual feedback data, to process object posture deviation by using a dynamic adjustment algorithm, and to obtain an optimized grasp execution path.

[0055] The multi-modal knowledge graph updating module is configured to simulate a robot operation environment based on the optimized grasp execution path, to evaluate the path effectiveness, and to update the multi-modal knowledge graph to adapt to a new scene.

[0056] The present application has the following beneficial effects:

[0057] The present application provides a robot target recognition and positioning method and system based on a knowledge graph, which aims to solve the logical correlation problem of object recognition ambiguity, posture deviation and poor scene adaptability leading to grasp failure in a complex environment, constructs a triple knowledge graph containing object appearance features and grasp point pose parameters by integrating multi-source data, realizes a structured node network, further acquires environment perception data of a robot vision module, determines the vector representation of a target object after processing, queries a matching node by using an entity linking technology if the similarity exceeds a threshold, solves recognition ambiguity to obtain an accurate result, retrieves pose parameters in the knowledge graph based on the result, performs coordinate conversion to generate a fine-grained grasp sequence, optimizes the path by using a dynamic adjustment algorithm in combination with real-time visual feedback, and updates the knowledge graph to adapt to a new scene by simulation and evaluation. The present application highlights the core role of the knowledge graph in grasp optimization, improves the accuracy, robustness and self-adaptation ability of robot operation, and is suitable for industrial automation and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 FIG. 1 is a flowchart of an embodiment of the robot target recognition and positioning method based on the knowledge graph of the present application;

[0059] Figure 2 FIG. 2 is a functional block diagram of an embodiment of the robot target recognition and positioning system based on the knowledge graph of the present application.

[0060] BRIEF DESCRIPTION OF DRAWINGS

[0061] 10, multi-modal knowledge graph construction module; 20, vector representation form determination module; 30, object recognition result acquisition module; 40, pose mapping relationship acquisition module; 50, fine-grained grasping operation sequence determination module; 60, grasping execution path acquisition module; 70, multi-modal knowledge graph update module. DETAILED DESCRIPTION

[0062] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the drawings in the specification and specific embodiments.

[0063] As shown in Figure 1 The first embodiment of the present application proposes a robot grasping target recognition and positioning method based on knowledge graph, which includes the following steps:

[0064] Step S100, a multi-modal knowledge graph is constructed by integrating multi-source data, the multi-modal knowledge graph models the relationship between object appearance features and grasping point pose parameters in the form of triples, and a structured node network containing object appearance description and optimal grasping trajectory is obtained.

[0065] First, integrate multi-source heterogeneous data: including object image data (two-dimensional visual features), three-dimensional model data (stereo structure features), grasping experiment data (historical successful grasping parameters), material attribute data (surface friction coefficient, etc.), clean up, normalize the data, and extract uniform dimension object appearance features (shape, color, texture, size) and grasping core parameters (grasping point three-dimensional coordinates, clamping angle, opening and closing amplitude, motion speed); adopt triple (subject-predicate-object) form to model the correlation, typical triples are, for example, "[object A]-[corresponding grasping pose]-[(X=5, Y=3, Z=2), clamping angle 30°]", "[object A]-[optimal grasping trajectory]-[smooth path from pre-grasping point to clamping point]"; through knowledge graph construction technology (such as ontology modeling, node association reasoning), organize massive triples into a structured node network, each node contains detailed appearance description of the object, and is also associated with all grasping point pose parameters and experimentally verified optimal grasping trajectory of the object, the matching accuracy of the graph node is required to be ≥95%, and the grasping parameter calling delay is ≤10ms, which provides comprehensive and accurate knowledge support for subsequent target recognition and grasping decision.

[0066] Step S200, according to the multi-modal knowledge graph, the environment perception data collected by the robot vision module is acquired, the environment perception data is processed by using a data processing method, and the vector representation form corresponding to the target object is determined.

[0067] With the feature specification of the multi-modal knowledge graph as a benchmark (such as the dimensions of appearance features and data format requirements), the robot vision module (such as a high-definition RGB (Red, Green, Blue) camera, a depth sensor, and a 3D (Three Dimensions) point cloud camera) is used to collect real-time perception data of the environment where the target object is located: including two-dimensional image data (carrying color, texture, and planar shape features) and three-dimensional point cloud data (carrying three-dimensional structure, size, and spatial position features); using a processing method suitable for multi-modal data, pre-processing (denoising, image distortion correction, and point cloud registration) is performed, and then feature extraction algorithms (such as CNN (Convolutional Neural Network) convolution coding, PointNet point cloud feature quantization, and HOG (Histogram of Oriented Gradient) feature extraction) are used to extract core features with consistent dimensions as the knowledge graph nodes; finally, through normalization and vector encoding processing, high-dimensional perception features are converted into low-dimensional and standardized vector representations, and the vector dimensions are required to be consistent with the knowledge graph node vectors (such as 128 dimensions / 256 dimensions), with a feature extraction matching degree of ≥92%, to ensure that the vector can be directly used for similarity comparison with the knowledge graph nodes, providing standardized input for subsequent target recognition.

[0068] Step S300, if the vector representation form has a similarity to the nodes of the multi-modal knowledge graph exceeding a preset threshold, the entity linking technology is used to query the matching object nodes, to judge and solve the object recognition ambiguity in complex scenes, and to obtain accurate object recognition results.

[0069] The similarity (using cosine similarity, Euclidean distance, etc.) between the target object vector representation obtained in step S200 and all object node vectors in the multi-modal knowledge graph is calculated first; a preset similarity threshold (such as ≥88%, which can be dynamically adjusted according to the object category) is set, and if the calculation result exceeds the threshold, it indicates that there is a matching candidate node, and the entity linking technology is started immediately to query the most suitable object node in the graph according to the vector feature correlation; for complex scenes such as occlusion, coexistence of similar objects, and environmental light interference, the correlation reasoning ability of the knowledge graph (such as object attribute cross verification and scene context matching) is used to judge and exclude recognition ambiguity (such as distinguishing between similar cups and bowls), and finally an accurate recognition result containing the object category, key appearance features, and material properties is output, with a recognition accuracy of ≥96% and an ambiguity solving response delay of ≤50 ms, to provide accurate target identification for subsequent grasp pose retrieval.

[0070] Step S400, retrieve the grasping point pose parameters in the multi-modal knowledge graph using the object recognition result, obtain related triple data, and obtain the pose mapping relationship for coordinate conversion.

[0071] The accurate object recognition result obtained in step S300 (including object category, key appearance feature, and material attribute) is used as a search keyword to perform directional query in the multi-modal knowledge graph, match all associated triple data corresponding to the object, and extract the triple of the type of "object-corresponding grasping pose" (such as "[ceramic cup]-[optimal grasping pose]-[middle-upper part of cup body three-dimensional coordinates (X=8, Y=4, Z=6), clamping angle 15°, opening and closing amplitude 5 cm]"). The retrieved triple data is filtered (preferably, high success rate data verified by experiments are selected), and the core parameters such as grasping point three-dimensional coordinates, clamping angle, and pose direction are integrated to establish the corresponding logic of "object recognition feature-grasping point pose parameter", and form the pose mapping relationship for subsequent coordinate conversion, with a parameter retrieval accuracy of ≥97% and data extraction delay of ≤10 ms, to ensure that the mapping relationship can accurately associate the recognition result with the grasping parameters, and provide a clear basis for the conversion of the perception coordinate system to the robot coordinate system.

[0072] Step S500, perform coordinate conversion operation through the pose mapping relationship to convert the perception coordinate system parameters into robot coordinate system values, and determine the fine-grained grasping operation sequence.

[0073] Based on the pose mapping relationship obtained in step S400, a preset coordinate conversion algorithm (such as hand-eye calibration matrix conversion or homogeneous coordinate transformation) is called to accurately convert the grasping point pose parameters (three-dimensional coordinates, clamping angle) in the perception coordinate system (such as camera coordinate system or point cloud coordinate system) into quantized values (such as joint angle, end effector spatial coordinates, and motion speed threshold) in the robot base coordinate system (or end effector coordinate system). According to the physical process of robot grasping (pre-grasping→pose adjustment→precise clamping→stable lifting), the converted coordinate system values are decomposed into a fine-grained grasping operation sequence, and each sequence node contains clear action instructions (such as "move to pre-grasping position (X=10, Y=6, Z=8)", "adjust the clamping angle to 15°", "approach the target at a speed of 5 cm / s", and "open and close amplitude 5 cm clamping"). The coordinate conversion error is ≤0.1 mm, and the operation sequence logical consistency is ≥99%, to ensure that the robot can accurately perform grasping actions according to the sequence, and avoid motion trajectory conflicts or operation omissions.

[0074] Step S600, combine the fine-grained grasping operation sequence and real-time visual feedback data, and use a dynamic adjustment algorithm to process the object pose deviation to obtain an optimized grasping execution path.

[0075] Based on the fine-grained grasping operation sequence generated in step S500, the framework is executed to synchronously acquire real-time feedback data of the robot vision module: including real-time position offset of the target object, posture twist (such as inclination angle, rotation deviation), dynamic change of environmental obstacles, etc.; the preset pose parameters in the fine-grained grasping operation sequence are compared with the real-time feedback data to identify the object posture deviation (such as the offset amount of the actual grasping point from the preset coordinates, the clamping angle deviation); an adjustment algorithm adapted to the dynamic scene (such as PID (Proportional-Integral-Differential, proportional-integral-differential control) adaptive control, reinforcement learning fine-tuning algorithm, trajectory smoothing optimization algorithm) is used to calculate and analyze the deviation data, and the motion parameters in the operation sequence are corrected in a targeted manner (such as adjusting the end effector movement trajectory, correcting the clamping angle, optimizing the motion speed); finally, an optimized grasping execution path that avoids deviation and adapts to the real-time environment is generated, the deviation correction response delay is required to be ≤30 ms, the grasping success rate after path optimization is required to be improved by ≥12%, and it is ensured that the robot can still accurately complete grasping in the scene where the object moves slightly and the posture changes.

[0076] In step S700, based on the optimized grasping execution path, the robot operation environment is simulated, the path effectiveness is evaluated, and the multi-modal knowledge graph is updated to adapt to the new scene.

[0077] Based on the grasping execution path optimized in step S600, the real operation environment is simulated by using a robot simulation platform (such as ROS Gazebo, PyBullet) to restore key scene elements such as target object position, environmental obstacle distribution, and robot motion constraints; from four dimensions of grasping success rate, collision risk, execution efficiency, and posture stability, the path effectiveness is evaluated (such as whether obstacles are avoided, whether clamping is stable, and whether execution time is within a reasonable range); if the path verification is passed, the object features of the new scene, real-time deviation data, optimized path parameters, verification results, etc. are supplemented to the multi-modal knowledge graph in the form of triples; if the verification fails, the failure reasons (such as insufficient deviation correction, unreasonable path planning) are extracted and the corresponding associated logic in the multi-modal knowledge graph is updated, realizing dynamic iteration of the multi-modal knowledge graph, requiring path evaluation accuracy ≥94%, and the grasping adaptation rate of the new scene after updating the multi-modal knowledge graph is improved by ≥15%, so that the robot can continuously accumulate scene experience and enhance the adaptability to complex new scenes.

[0078] Further, the robot grasping target recognition and positioning method based on the knowledge graph provided by the embodiment includes steps S100:

[0079] In step S110, the appearance feature data of the target object is acquired from the image sensor and the depth sensor by a multi-source data acquisition tool, pre-processing operation is performed on the appearance feature data, and the pre-processed appearance feature data is smoothed by a denoising tool to obtain clear appearance feature description information.

[0080] The pre-processed appearance feature data is obtained by the following formula:

[0081] (1)

[0082] In formula (1),

[0083] represents the pre-processed appearance feature data, represents the feature data acquired from the image sensor, represents the feature data acquired from the depth sensor, represents the original feature data, represents a pre-processing transformation function, , , respectively represent the weight coefficients of the image feature, the depth feature and the original feature.

[0084] The clear appearance feature description information is obtained by the following formula:

[0085] (2)

[0086] In formula (2), represents the clear appearance feature description information, represents the number of dimensions of the feature description, represents the saliency weight of the th feature component, represents the gradient intensity of the th feature component, represents the smoothed feature data, represents a feature description extraction function.

[0087] In the robot grasping system, first, the appearance feature data of the target object is acquired from the image sensor and the depth sensor by a multi-source data acquisition tool. The multi-source data acquisition tool here can be understood as an integrated software framework that combines an RGB camera as an image sensor to capture color and texture information, and a ToF (Time of Flight) sensor as a depth sensor to measure the distance from the object surface to the sensor, thereby forming a multi-modal data stream. Assuming that the target object is an apple, this acquisition process will scan the surface color distribution and three-dimensional contour data of the apple in real time, ensuring that the data covers multiple angles of the object.

[0088] Preprocessing operations are performed on the appearance feature data. This includes calibrating and aligning the collected image and depth data, for example, using calibration functions in the OpenCV (Open Source Computer Vision Library) library to eliminate biases between sensors, and then applying filters to remove noise. If there is uneven lighting in the image, the brightness distribution is adjusted through histogram equalization to make the red texture of the apple more uniform and visible. The goal of this preprocessing step is to improve the quality of the data and provide a reliable foundation for subsequent analysis. For example, a denoising tool is used to smooth the preprocessed appearance feature data to obtain clear appearance feature description information. Denoising tools such as Gaussian filters apply convolution operations to the data to smooth out random noise points without losing the sharp features of the apple edges. Through this processing, the system can extract description information such as the shape, size, and surface smoothness of the apple, forming a vectorized feature set.

[0089] In step S120, the appearance feature description information is used to generate the three-dimensional structure data of the target object using a three-dimensional reconstruction tool. When generating the three-dimensional structure data, if a data missing area is detected, an interpolation tool is used to complete the data missing area to determine the complete three-dimensional structure description.

[0090] The three-dimensional structure data of the target object is obtained by the following formula:

[0091] (3)

[0092] In formula (3), represents the three-dimensional structure data at coordinate point in three-dimensional space, represents the total number of appearance feature description information, represents the weight coefficient of the th feature, represents the reconstruction function of the th appearance feature in three-dimensional space.

[0093] The complete three-dimensional structure description is obtained by the following formula:

[0094] (4)

[0095] In formula (4), represents the complete three-dimensional structure description data set, represents the effective structure data area obtained by the original three-dimensional reconstruction, represents the data missing area completed by the interpolation tool, represents the union operation of the data set.

[0096] According to the appearance feature description information, a three-dimensional reconstruction tool is used to generate the three-dimensional structure data of the target object. The three-dimensional reconstruction tool is, for example, a framework based on a Structure from Motion (SfM) algorithm, which calculates parallax from multiple images and constructs a point cloud model. For the apple, if some angle data is missing, the system will detect the hollow area in the point cloud. For example, when generating the three-dimensional structure data, if a data missing area is detected, the data missing area is completed by an interpolation tool to determine a complete three-dimensional structure description. The interpolation tool, such as a nearest neighbor interpolation method, will infer the missing value according to the data of the adjacent points, for example, fill the gap at the bottom of the apple with the average depth of the surrounding point cloud, so as to obtain a complete mesh model. This completion process ensures the completeness of the three-dimensional description and helps the accuracy of subsequent grasping planning.

[0097] In step S130, a path planning tool is used to calculate the grasping point pose parameters for the three-dimensional structure description, obtain a plurality of candidate grasping trajectory points corresponding to the three-dimensional structure description, and sort the candidate grasping trajectory points by a comparison tool to determine the optimal grasping trajectory parameters.

[0098] The optimal grasping point pose parameters are obtained by the following formula:

[0099] (5)

[0100] In formula (5), represents the optimal grasping point pose parameters, including position coordinates , , and attitude angles , , , represents the grasping stability score, represents the reachability score, represents the grasping force score, , , respectively represent the weight coefficients of each evaluation index.

[0101] For the stereo structure description, the path planning tool is used to calculate the grasp pose parameters. The path planning tool can be a system based on RRT (Rapidly-exploring Random Tree) algorithm, which generates multiple paths from the initial position of the robot arm to the surface of the apple, and calculates the pose parameters such as position coordinates and rotation angles of each path. For example, multiple candidate grasp trajectory points corresponding to the stereo structure description are obtained, and the comparison tool is used to sort the candidate grasp trajectory points to determine the optimal grasp trajectory parameters. The comparison tool will evaluate the collision risk and efficiency of the trajectory, for example, calculate the score of 10 candidate points, and select the trajectory with the highest score that avoids the collision of the apple stem as the optimal parameter. This sorting can significantly improve the success rate of grasping and bring higher efficiency in orchard automatic picking.

[0102] In step S140, the appearance feature description information is associated and mapped with the optimal grasp trajectory parameters by the knowledge graph construction tool to generate a triple relationship network containing structured nodes, and a multi-modal knowledge graph is obtained.

[0103] The triple relationship structure between the appearance feature and the grasp trajectory parameters is defined by the following formula:

[0104] (6)

[0105] In formula (6), represents the multi-modal knowledge graph, represents the appearance feature description information node, represents the associated mapping relationship, represents the optimal grasp trajectory parameter node, represents the appearance feature description information set, represents the grasp trajectory parameter set.

[0106] The complete triple network structure composed of structured nodes is described by the following formula:

[0107] (7)

[0108] In formula (7), represents the triple relationship network, represents the total number of triples, represents the head entity node of the th triple, represents the relationship type of the th triple, represents the tail entity node of the th triple.

[0109] The mapping process of converting the multi-modal input into the multi-modal knowledge graph by constructing the tool is represented by the following formula:

[0110] (8)

[0111] In formula (8), represents the multi-modal knowledge graph construction process, represents the multi-modal knowledge graph construction tool function, represents the appearance feature description information input, represents the optimal grasping trajectory parameter input, represents the generated multi-modal knowledge graph output.

[0112] The appearance feature description information and the optimal grasping trajectory parameter are associated and mapped by the knowledge graph construction tool to generate a triple relationship network containing structured nodes, and a multi-modal knowledge graph is obtained. The knowledge graph construction tool, such as the Neo4j framework, will create nodes such as “apple appearance features” and “grasping trajectory”, connect them with relationships such as “corresponding to”, and form a network. For example, the triple between nodes can be (apple texture, association, optimal grasping angle), and this multi-modal knowledge graph facilitates the system to learn and reuse knowledge and quickly optimize the grasping strategy when processing similar objects.

[0113] Preferably, the knowledge graph-based robot grasping target recognition and positioning method provided in the embodiment comprises the following steps S200:

[0114] In step S210, the visual module obtains the original perception data stream from the environment, and the original perception data stream is preliminarily processed by using a denoising tool to obtain the cleaned basic perception data.

[0115] The cleaned basic perception data is obtained by the following formula:

[0116] (9)

[0117] In formula (9), represents the time of the cleaned basic perception data, represents the original perception data stream obtained by the visual module at time , and represents the processing function of the denoising tool, represents the parameter set of the denoising tool, represents the fusion weight coefficient of the original data and the denoised data.

[0118] The original perception data stream obtained by the visual module at time is obtained by the following formula:

[0119] (10)

[0120] In formula (10), denotes the total length of the perception data collection, denotes the weight function at time , denotes the image intensity value at spatial coordinates and time .

[0121] The raw perception data stream is obtained from the environment by the vision module, which captures real-time image sequences using an integrated camera system. For example, in an industrial assembly line, a vision module such as an RGB-D (Red Green Blue Depth) camera collects noisy video streams from the workshop environment, which include color information and preliminary depth measurements.

[0122] When the raw perception data stream is processed using a denoising tool, the denoising tool specifically refers to a median filter, which eliminates salt and pepper noise by replacing pixel values with neighborhood median values. For example, for a target object such as a screw object on an assembly line, the raw data stream may be blurred due to light interference. The median filter scans a 3x3 neighborhood of each pixel, calculates the median value and replaces it, thereby outputting clear image data to ensure the accuracy of subsequent steps.

[0123] In step S220, the contour information of the target object is separated from the basic perception data using a feature extraction tool. If the clarity of the contour information is lower than the preset threshold, the enhancement tool is used for optimization to determine the complete contour description information.

[0124] The following formula is used to quantify the clarity of the contour by calculating the gradient energy and comparing it with the preset threshold:

[0125] (11)

[0126] In formula (11), denotes the clarity evaluation value of the contour information, and denote the width and height of the contour region, respectively, denotes the gradient amplitude at position , denotes a small constant to prevent division by zero.

[0127] The following formula is used to combine multiple enhancement techniques to obtain a complete and clear contour description:

[0128] (12)

[0129] Equation (12) is: represents the complete contour description information after optimization by the enhancement tool, represents the histogram equalization enhancement component, represents the sharpening filter enhancement component, represents the frequency domain filter enhancement component, , , respectively represent the weight coefficients of the three enhancement methods, and the low definition contour is optimized by the weighted combination of multiple enhancement techniques.

[0130] Based on the basic perception data, the contour information of the target object is separated by using the feature extraction tool. The feature extraction tool can be a Canny edge detection algorithm, which first applies a Gaussian filter to smooth the image, then calculates the gradient strength and direction, and then uses a double threshold to connect the edges after non-maximum suppression, thereby extracting the boundary line of the target object screw.

[0131] If the definition of the contour information is lower than the preset threshold, for example, the definition is evaluated by calculating the edge continuity score, if the score is lower than 0.8, optimization is performed by the enhancement tool, such as a Laplace sharpening filter, which enhances the high frequency component to highlight the edge details, and determines the complete contour description information. After this optimization, the thread contour of the target object screw becomes sharp, which facilitates accurate identification.

[0132] Step S230, for the contour description information, a data fusion tool is used to combine depth information to generate spatial position data of the target object, to obtain corresponding three-dimensional coordinate information, and to determine the preliminary vector representation of the target object.

[0133] The three-dimensional coordinate information of the target object is obtained by the following formula:

[0134] (13)

[0135] In equation (13), represents the three-dimensional spatial coordinates of the target object, represents the inverse matrix of the camera intrinsic matrix, and represent the two-dimensional pixel coordinates in the contour description information, represents the depth information value of the corresponding pixel point.

[0136] The preliminary vector representation of the target object is obtained by the following formula:

[0137] (14)

[0138] In equation (14), represents the preliminary vector representation of the target object, denotes the total number of sampling points, denotes the rotation transformation matrix of the denotes the coordinate transformation matrix, denotes the rotation transformation matrix of the denotes the local coordinate vector of the

[0139] The process of calibrating the preliminary vector representation through the coordinate mapping tool, which is a least squares-based transformation matrix calculation, maps the preliminary vector to the world coordinate system. During calibration, if a deviation is detected that exceeds a pre-set threshold, such as a deviation of more than 5 millimeters, an adjustment tool, such as an iterative closest point algorithm, is used to correct the deviation by iteratively matching point pairs and minimizing the error to obtain the vector representation corresponding to the target object. In an assembly scenario, this correction adjusts the vector of a screw from the preliminary [2.5, 1.0, 0.3] to the accurate [2.4, 1.1, 0.2], thereby improving the grasping accuracy.

[0140] Step S240, calibrating the preliminary vector representation through the coordinate mapping tool. During calibration, if a deviation of the calibrated vector representation is detected that exceeds a pre-set threshold, an adjustment tool is used to correct the deviation to obtain the vector representation corresponding to the target object.

[0141] The process of linearly transforming and calibrating the preliminary vector through the coordinate mapping tool is described by the following formula:

[0142] (15)

[0143] In formula (15), denotes the calibrated vector representation, denotes the coordinate mapping transformation matrix, denotes the offset correction vector.

[0144] The following formula is used to detect whether the deviation of the vector representation exceeds the pre-set threshold, and the correction mechanism is triggered when the deviation is greater than the threshold:

[0145] (16)

[0146] In formula (16), denotes the deviation detection result, denotes the target vector value, denotes the pre-set deviation threshold.

[0147] The following formula describes the process of further correcting the calibrated vector using the adjustment tool to obtain the vector representation corresponding to the target object:

[0148] (17)​

[0149] In formula (17), This represents the vector representation of the target object. This indicates an adjustment to the strength coefficient. This indicates the adjustment utility function. This indicates the set of parameters to be adjusted.

[0150] When generating spatial location data for a target object using a data fusion tool that combines contour description information with depth information, the data fusion tool is specifically a Kalman filter framework. It integrates contour edge data with depth sensor data, fusing information through prediction and update steps to obtain the corresponding 3D coordinate information and determine the initial vector representation of the target object. For example, when processing a screw, the filter projects the 2D contour into 3D space, calculates the x, y, and z coordinates, forming a vector representing the object's pose, such as [2.5, 1.0, 0.3], which aids in robot localization.

[0151] Furthermore, in the knowledge graph-based robot grasping target identification and localization method provided in this embodiment, step S300 includes:

[0152] Step S310: Based on the vector representation, obtain at least one candidate node data from the pre-established multimodal knowledge graph, and use a feature comparison tool for preliminary screening to obtain a set of nodes that are close to the features of the target object.

[0153] The filtered set of nodes is obtained using the following formula:

[0154] (18)

[0155] In formula (18), This represents the set of nodes after filtering. Representing multimodal knowledge graphs The first in 1 node Represents the feature distance metric function. Represents a node eigenvectors, The feature vector representing the target object. This represents the similarity threshold parameter.

[0156] According to the vector representation of the target object, the process of obtaining candidate node data from the multi-modal knowledge graph can be understood as first querying the node library in the graph, which stores multi-modal features of various objects, such as visual, tactile, and semantic information. For example, in an industrial robot grasping scene, the vector representation may be a three-dimensional coordinate and pose data like [3.2, 1.5, 0.4], and the system will extract at least one matching candidate node from the multi-modal knowledge graph through a similarity matching algorithm. For example, for a gear object, the candidate nodes may include multiple entries similar to the gear, covering descriptions of different sizes and materials.

[0157] When using the feature comparison tool for preliminary screening, the feature comparison tool can be a cosine similarity calculation method, which evaluates similarity by comparing the angle between vectors. The process involves taking the dot product of each component of the target vector with the corresponding component of the node vector, then dividing by the product of the respective module lengths to obtain a similarity score, thereby screening out a node set with a score higher than 0.7. For example, for the vector of a gear, the feature comparison tool compares the shape features of the nodes in the graph, such as the number of teeth and diameter. If the scores of multiple nodes are close, a set containing 3 to 5 nodes will be formed, ensuring that the preliminary screening focuses on options with similar features.

[0158] Step S320, if there are multiple similar nodes in the node set, then a secondary filtering of the node set is performed by combining environmental context information through a data fusion tool to determine a single node data that matches the current scene.

[0159] The following formula is used to select a single node that best matches the current scene from multiple similar nodes:

[0160] (19)

[0161] In formula (19), represents the final matching single node, represents the candidate node set, represents the node matching probability with the current scene, represents the environmental adaptability score of the node , represents the reliability index of the node .

[0162] If there are multiple similar nodes in the node set, a secondary filtering process is performed by combining environmental context information through a data fusion tool, which is specifically a Bayesian fusion framework. The Bayesian fusion framework integrates visual data and context such as lighting conditions or surrounding object information, calculates the conditional probability of each node through a probability updating step, gradually excludes mismatched nodes, and finally determines a single node. For example, on an assembly line, the context information includes the temperature and light intensity in the workshop. After fusion, if a node corresponds to a gear in a high-temperature environment, but the current scene is at room temperature, it will be filtered out, and a single node data that matches the current scene will be selected.

[0163] In step S330, a link verification tool is used to detect the association reliability of the single node data and the target object. If the association reliability of the single node data and the target object is lower than a preset threshold, backup node data is obtained from the multi-modal knowledge graph for supplementary comparison to determine the final matching node result.

[0164] The association reliability of the single node data and the target object is obtained by the following formula:

[0165] (20)

[0166] In formula (20), represents the association reliability score of the single node data and the target object, represents the total number of multi-modal features, represents the weight coefficient of the th modal feature, represents the similarity calculation function, represents the feature representation of the th node in the th modal, represents the feature representation of the target object in the th modal.

[0167] The final matching node result is obtained by the following formula:

[0168] (21)

[0169] In formula (21), represents the final matching node result, represents the original single node, represents the preset reliability threshold, to represents backup nodes, represents the comprehensive matching evaluation function of node and target object .

[0170] When using a link verification tool to detect the reliability of the association between a single node's data and a target object, the link verification tool can be a graph link strength evaluation algorithm. It examines the edge weights between nodes, which represent semantic or spatial associations. Reliability is determined by accumulating path weights and comparing them with a threshold. If the weights are below a preset threshold (e.g., 0.85), the process of obtaining backup node data from the multimodal knowledge graph for supplementary comparison involves expanding the query scope, introducing adjacent nodes for re-matching, and finally determining the matching node result. For example, for a gear, if the initial node link is weaker than the threshold, the system will retrieve backup nodes, such as gear-like variants, perform feature overlay comparison, and confirm the final match.

[0171] Step S340: Based on the final matching node results, use the mapping tool to generate the identification label of the target object, and use the multimodal data consistency verification tool to verify the identification label to obtain accurate object identification results.

[0172] The identification label for the target object is derived using the following formula:

[0173] (twenty two)

[0174] In formula (22), Indicates the first The identification label generated for each target object This represents a mapping utility function. Indicates the first The final matching node, Indicates the mapping parameters. Indicates the candidate label category, Indicates the number of feature dimensions. Indicates the first The weights of each feature Indicates the first Mapping function of features, Indicates the first Feature templates for class tags.

[0175] Accurate object recognition results are obtained using the following formula:

[0176] (twenty three)

[0177] In formula (23), This represents the final, accurate object recognition result. Indicates the candidate identification results. Indicates the quantity of objects. Indicates the first The importance weight of each object Indicates the first Error evaluation function for an object, Indicates the first Initial identification label for an object, Indicates the first The verification results of each object.

[0178] Based on the final matching node results, the process involves generating identification labels for the target object using a mapping tool. This mapping tool is specifically a label mapping function that converts node attributes into standard labels, such as "Industrial Gear Type A". Then, a multimodal data consistency verification tool is used to calibrate the labels. This verification tool employs a cross-validation mechanism, adjusting the labels by comparing the consensus of visual, depth, and semantic data to ensure consistency. For example, when identifying gears, if the visual data shows a circular outline while the semantic data matches "transmission component", the verification will yield a precise result of "precision gear". This calibration process improves the accuracy of identification and provides reliable output in complex environments.

[0179] Preferably, in the knowledge graph-based robot grasping target identification and localization method provided in this embodiment, step S400 includes:

[0180] Step S410: Based on the object recognition results, extract at least one relevant grasping point pose parameter data from the multimodal knowledge graph, and use a data filtering tool to perform preliminary classification of the grasping point pose parameter data to obtain a set of pose parameters that match the target object.

[0181] The optimal pose parameters of the grab point extracted from the multimodal knowledge graph are obtained by the following formula:

[0182] (twenty four)

[0183] In formula (24), This represents the optimal grasping point pose parameters extracted from the multimodal knowledge graph. Representing a multimodal knowledge graph, This represents the total number of related objects in the multimodal knowledge graph. Indicates the first The weighting coefficient of each object, The function represents the similarity calculation between the target object and objects in the knowledge graph. Represents the features of the target object being identified. Representing the first in a multimodal knowledge graph The feature vector of an object.

[0184] The set of pose parameters matched with the target object is obtained by the following formula:

[0185] (25)

[0186] In formula (25), a final pose parameter set matched with the target object, a total number of classifications, a pose parameter subset after the first classification, a function representing the pose parameter a matching degree evaluation function of the target object, an attribute feature of the target object, a minimum threshold requirement of the matching degree.

[0187] According to the object recognition result, at least one related grasp point pose parameter data is extracted from the multi-modal knowledge graph, which can be understood as that the system first queries the pose database in the multi-modal knowledge graph, and these databases store the grasp coordinates and pose information of the object in different scenes. For example, after recognizing a bolt on the automobile assembly line, the system will pull the related pose parameter data from the multi-modal knowledge graph based on the identification label such as “standard M8 bolt”, and these data may include a three-dimensional coordinate point [0.5, 1.2, 0.3] and a rotation angle. The extraction process involves a matching algorithm scanning the graph nodes to ensure that the data is consistent with the object geometry, so as to obtain an initial grasp point set.

[0188] When the data filtering tool is used to preliminarily classify the grasp point pose parameter data and obtain a pose parameter set matched with the target object, the data filtering tool can be a clustering-based method, which groups the pose data according to similarity, such as classifying the coordinate points into top grasp or side grasp groups by calculating the Euclidean distance. For example, for a bolt, the data filtering tool will classify multiple poses, such as a set containing [0.4, 1.1, 0.2] suitable for vertical grasping and [0.6, 1.0, 0.4] for lateral grasping. The classification process first calculates the center distance of each data point, and then induces a set to ensure that the object size and shape are preliminarily matched.

[0189] In step S420, if there are multiple candidate data in the pose parameter set, the context environment matching tool is used to combine scene information to refine and filter the pose parameter set, and determine a single grasp point pose parameter data that fits the current grasping task.

[0190] The single grasp point pose parameter data that fits the current grasping task is obtained by the following formula:

[0191] (26)

[0192] In formula (26), represents the optimal single grasp point pose parameter determined after refinement and filtering, represents a set of all candidate pose parameters, ​a balance factor, a pose parameter, a quality assessment function, a pose parameter, an environment adaptability assessment function.

[0193] If there are multiple candidate data in the pose parameter set, the context environment matching tool is used to refine and filter the pose parameter set in combination with scene information to determine a single grasping point pose parameter data that fits the current grasping task. The context environment matching tool is a rule-based filtering framework that integrates scene factors such as robot arm length and object position and excludes inappropriate options by item-by-item comparison. For example, in an assembly line scene, if the set has three poses, the context environment matching tool will filter out the poses that exceed the range in combination with environmental information such as a workbench height of 20 cm and arm length restrictions, and finally select [0.5, 1.2, 0.3] as a single data. This filtering can improve grasping efficiency and avoid collision risks.

[0194] In step S430, a triple extraction tool is used to obtain corresponding triple data from the multi-modal knowledge graph for the single grasping point pose parameter data, and a data mapping tool is used to structure the triple data to obtain a preliminary pose mapping relationship for coordinate conversion.

[0195] The corresponding triple data is obtained by the following formula:

[0196] (27)

[0197] In formula (27), represents a set of grasping pose triple data extracted from the multi-modal knowledge graph, represents a grasping point pose corresponding to the th entity node, represents a spatial relationship attribute, represents a pose parameter value, represents a pose entity set, represents a spatial relationship set, represents a parameter value set. The preliminary pose mapping relationship for coordinate conversion is obtained by the following formula:

[0198]

[0199] (28)

[0200] In formula (28), represents a preliminary pose mapping relationship for coordinate conversion, represents a mapping relationship construction function,​​ denotes a source coordinate system parameter set, denotes a target coordinate system parameter set, denotes the total number of mapping parameters, denotes a weight coefficient of the th mapping parameter, denotes a mapping transformation function of the th parameter, and denote the th corresponding parameter in the source coordinate system and the target coordinate system, respectively.

[0201] When the triple extraction tool is used to obtain corresponding triple data from the multi-modal knowledge graph for a single grasping point pose parameter data, the triple extraction tool is a query mechanism based on an RDF framework, which extracts a subject-predicate-object structure from the graph, such as (bolt, hasPose, [0.5, 1.2, 0.3]), and the extraction process involves traversing node edges and collecting related triples to ensure data integrity.

[0202] When the data mapping tool is used to structure the triple data and obtain a preliminary pose mapping relationship for coordinate conversion, the data mapping tool can be a conversion matrix method, which converts the triple into a matrix form, such as mapping the pose data into a 4x4 transformation matrix, and the processing process includes combining the coordinate points and the rotation parameters to generate a preliminary relationship for subsequent robot coordinate system conversion.

[0203] Step S440, the preliminary pose mapping relationship is checked by a consistency checking tool, and if the checking result does not meet the preset threshold, backup data is obtained from the multi-modal knowledge graph for supplementary comparison to determine the accurate pose mapping relationship.

[0204] The following formula is used to determine whether the checking result meets the accuracy requirement:

[0205] (29)

[0206] In formula (29), denotes the deviation amount of the checking result from the preset threshold, denotes the checking calculation result of the preliminary pose, denotes the preset threshold vector.

[0207] The following formula is used to determine the optimal pose mapping relationship by fusing multi-source data:

[0208] (30)

[0209] In formula (30), denotes the final determined accurate pose mapping relationship, a set of parameter vectors representing pose mapping related parameters, a weight coefficient representing preliminary data, a weight coefficient representing backup data, an error function representing preliminary data, an error function representing backup data based on a multimodal knowledge graph.

[0210] The preliminary pose mapping relationship is proofread by a consistency checking tool. If the checking result does not match the preset threshold, backup data is obtained from the multimodal knowledge graph for supplementary comparison to determine the accurate pose mapping relationship. The consistency checking tool is specifically an error calculation framework that compares the deviation of the mapping relationship from the actual scene. For example, the root mean square error is evaluated. If the deviation exceeds 0.1, the backup triple such as the alternative pose [0.55, 1.25, 0.35] is queried for comparison, and the final adjustment is made to the accurate relationship. This proofreading process can ensure the grasping accuracy and improve the overall task success rate in a complex assembly environment.

[0211] Further, the robot target recognition and positioning method based on the knowledge graph provided in the embodiment includes the following steps.

[0212] In step S510, a coordinate conversion operation is performed through the pose mapping relationship to convert the perception coordinate system parameter into a robot coordinate system value.

[0213] The rigid body transformation of the position coordinates from the perception coordinate system to the robot coordinate system is realized by the following formula:

[0214] (31)

[0215] In formula (31), represents a position vector in the robot coordinate system, represents a rotation matrix from the perception coordinate system to the robot coordinate system, represents a position vector in the perception coordinate system, represents a translation vector from the perception coordinate system to the robot coordinate system.

[0216] The coordinate conversion operation is performed by the pose mapping relationship to convert the perception coordinate system parameters into robot coordinate system values. This can be understood as the system first loading a pre-established mapping matrix, which is essentially a mathematical representation based on affine transformation, aligning the coordinate system of perception devices such as cameras with the coordinate system of robot arms. Perception coordinate system parameters usually include three-dimensional point positions of objects in the camera field of view, such as the position coordinates of a car part on the assembly line [1.0, 2.5, 0.8]. The pose mapping relationship is a 4x4 transformation matrix containing translation and rotation components. The conversion process involves matrix multiplication, multiplying the perception coordinate point by the matrix to get the corresponding value in the robot coordinate system, for example, converting [1.0, 2.5, 0.8] to [0.7, 1.8, 0.5]. This process ensures accurate bridging from visual perception to mechanical execution. In actual business, such as in the automotive manufacturing scene, the system will first verify the validity of the matrix by comparing the conversion results of known calibration points. If the perception coordinates and robot coordinates have large deviations, the matrix parameters are iteratively optimized to achieve reliable coordinate conversion. For example, when handling a nut grabbing task, the perception coordinate system captures the position of the nut as [0.9, 1.1, 0.4]. Using the rotation components and scaling factors in the pose mapping relationship, the system gradually calculates the transformation of each coordinate axis. First, apply a translation offset to the x-axis, then apply a rotation matrix to the y-axis and z-axis, and gradually obtain the robot coordinates [0.6, 0.8, 0.3]. This step-by-step calculation helps to debug the business process.

[0217] Step S520, determining the fine-grained grasping operation sequence according to the converted robot coordinate system values.

[0218] The following formula defines a complete grasping operation sequence generation method based on the robot coordinate system:

[0219] (32)

[0220] In formula (32), represents the complete fine-grained grasping operation sequence, represents the th operation action in the sequence, represents the total length of the operation sequence, represents the operation generation function, represents the speed parameter of the th step, represents the force control parameter of the th step, represents the time parameter of the th step.

[0221] According to the transformed robot coordinate system value, a fine-grained grasping operation sequence is determined, which involves decomposing the coordinate value into a series of action instructions. This fine-grained grasping operation sequence is usually generated by a path planning algorithm, which considers the kinematic model of the robot joints and converts the end-point coordinates into a sequence of joint angles. Specifically, in the automobile assembly business, the transformed coordinates [0.6, 0.8, 0.3] will be input into an inverse kinematics solver, which is an iterative method based on the Jacobian matrix that back-propagates the angles of each joint from the target position. For example, first calculate the rotation angle of the base joint as 30 degrees, then extend the arm joint to 45 degrees, and generate a fine-grained grasping operation sequence such as "move to point A, rotate the gripper, close the claws". The entire process ensures that the sequence avoids obstacles and verifies the feasibility of the sequence through simulation. If a step in the fine-grained grasping operation sequence collides with the workbench, it is adjusted to a detour path, thereby forming a complete grasping instruction chain. For example, for a complex gearbox part grasping, the transformed coordinates [1.2, 3.0, 0.9] will trigger the sequence generation, including slow movement in the initial approach phase and fine-tuning actions in the precise alignment phase. The system will integrate force feedback data to refine the sequence to ensure the appropriate pressure during grasping.

[0222] Preferably, the robot grasping target recognition and positioning method based on the knowledge graph provided by the embodiment comprises the following steps:

[0223] In step S610, real-time visual feedback data is obtained through a visual sensor, continuous image capture is performed on the target object, image processing tools are used to extract the target object features of the real-time visual feedback data, and preliminary position and attitude information of the target object is obtained.

[0224] The preliminary position and attitude information of the target object is obtained by the following formula:

[0225] (33)

[0226] In formula (33), represents the six-degree-of-freedom position and attitude vector of the target object, , , represents the three-dimensional spatial position coordinates of the object, , , represents the rotation angle of the object around the three axes, represents the coordinate transformation matrix, represents the total number of feature points, represents the confidence weight of the th feature point, , , represents the coordinates of a feature point, 、 、 The second moment of the image is used to calculate the pose angle.

[0227] Real-time visual feedback data is acquired through a vision sensor, which is usually an RGB-D camera mounted at the end of a robotic arm, capable of capturing depth information and color images. Continuous image capture involves a shooting frequency of 30 frames per second to ensure the capture of dynamic changes in the object. For example, on an automotive assembly line, a vision sensor continuously captures image sequences from multiple angles for an engine block as the target object. These images are processed using image processing tools such as the OpenCV library, which first applies edge detection algorithms such as the Canny operator to identify the object's outline, then uses feature extraction methods such as the SIFT (Scale-invariant Feature Transform) algorithm to match key points, thus calculating the initial position of the cylinder such as coordinates [2.1, 3.4, 1.2] and attitude information including a rotation angle of 45 degrees. This extraction process essentially converts pixel-level data into a three-dimensional spatial representation, ensuring that the preliminary information accurately reflects the object's state in the working environment.

[0228] Step S620, according to the preliminary position and attitude information of the target object, combined with the pre-established deviation database, using comparison tools to judge the deviation between the actual attitude and the expected attitude of the target object, if it exceeds the preset threshold, generate deviation correction parameters.

[0229] The following formula is used to calculate the positional deviation between the actual attitude and the expected attitude of the target object:

[0230] (34)

[0231] In formula (34), represents the value of the actual attitude in the th dimension, represents the value of the expected attitude in the th dimension, represents the final calculated positional deviation Euclidean distance.

[0232] The following formula is used to calculate the angular deviation between the actual rotation attitude and the expected rotation attitude of the target object:

[0233] (35)

[0234] In formula (35), represents the rotation matrix of the actual attitude, a rotation matrix representing an expected pose, a trace of a matrix, a final calculated rotation angle deviation.

[0235] The deviation correction parameter is generated by the following formula:

[0236] (36)

[0237] In formula (36), and are preset thresholds for position deviation and rotation deviation, respectively, and are correction coefficients, represents the final generated correction parameter value, and the correction parameter is calculated when the deviation exceeds the threshold, otherwise it is 0.

[0238] According to the obtained preliminary position and pose information, combined with the pre-established deviation database, the deviation database is a collection of historical calibration data, for example, it contains records of the pose deviations of various objects under different lighting conditions, such as the database records that the expected rotation angle of the cylinder under the standard pose is 0 degrees, and the actual captured is 5 degrees deviation. A comparison tool such as a matching algorithm based on Euclidean distance is used to compare the actual pose with the expected pose, and the tool will calculate the position vector difference and the pose quaternion difference. If the deviation exceeds the preset threshold, such as a position difference of 0.05 meters or an angle difference of 3 degrees, a deviation correction parameter is generated, for example, a correction vector [0.03, 0.02, 0.01] and a rotation compensation matrix are generated. This process optimizes the parameters by iteratively comparing database entries to ensure that the correction parameter can effectively compensate for environmental disturbances such as light changes.

[0239] Step S630, data fusion is performed by the deviation correction parameter and the fine-grained operation sequence, and a path planning tool is used to dynamically adjust the grasping execution path to determine the adjusted path node data.

[0240] The adjusted path node data is obtained by the following formula:

[0241] (37)

[0242] In formula (37), represents the adjusted path node data, represents the original path node, represents the fusion weight coefficient, represents the deviation correction parameter vector, represents the fine-grained operation sequence, represents the element-by-element multiplication operation.

[0243] The deviation correction parameter is fused with the fine-grained operation sequence, where the fine-grained operation sequence refers to a pre-generated grabbing action chain, such as joint motion instructions from a starting point to a grabbing point, and the data fusion adopts a weighted average method to integrate the correction parameter into the sequence, for example, adding the correction vector to the path point coordinates of the sequence. A path planning tool such as the RRT (Rapidly-exploring Random Tree) algorithm is used to dynamically adjust the grabbing execution path, which simulates the robot arm motion trajectory, considers obstacle avoidance, and determines the adjusted path node data such as a series of intermediate points [1.5, 2.0, 0.8], [1.8, 2.5, 1.0]. This adjustment process starts from the initial sequence and gradually inserts the corrected nodes to form a smoother path.

[0244] Step S640, according to the adjusted path node data, combined with real-time visual feedback data, using data integration tools for secondary verification, get the optimized grabbing execution path.

[0245] The optimized grabbing execution path is obtained by the following formula:

[0246] (38)

[0247] In formula (38), represents the position vector of the optimized grabbing execution path at time , represents the position data of the adjusted path node at time , represents the correction vector of the real-time visual feedback data at time , represents the weight coefficient of the path node data, and the value range is between 0 and 1.

[0248] According to the adjusted path node data, combined with real-time visual feedback data for secondary verification, here the data integration tool such as Kalman filter will fuse the node data and the latest image feedback, for example, the path node and the real-time position [2.0, 3.3, 1.1] are integrated by weighted, the verification process involves prediction-update cycle, predict path stability and update deviation, finally get the optimized grabbing execution path such as complete instruction chain from approaching to grabbing sequence, this execution path ensures the accuracy and safety of the cylinder grabbing in the automobile assembly.

[0249] Further, the robot grabbing target recognition and positioning method based on the knowledge graph provided by the embodiment comprises the following steps:

[0250] Step S710: Based on the optimized capture execution path, use an environment simulation tool to digitally reconstruct the target operating environment, extract data for spatial constraints in the scene, and obtain a virtual operating environment dataset.

[0251] The virtual operating environment dataset is derived using the following formula:

[0252] (39)

[0253] In formula (39), This represents the final generated virtual operating environment dataset. This indicates the total number of simulation scenarios. Indicates the first Geometric structure data for each scene, Indicates the first Attribute information data for each scenario, Indicates the first Visualized data for each scenario This indicates that data filtering filters are used to ensure the quality of the dataset.

[0254] Based on the optimized grasping execution path, an environmental simulation tool is used to digitally reconstruct the target operating environment. Here, the environmental simulation tool usually refers to simulation software such as Gazebo, which can build a three-dimensional virtual model by inputting sensor data and extract data for spatial constraints in the scene. For example, in the case of grasping the engine block on an automobile assembly line, the environmental simulation tool will extract the boundary information of obstacles such as walls and conveyor belts from the point cloud data obtained by the laser scanner, and calculate the spatial constraints such as the volume of the usable area being 2.5 cubic meters and the minimum gap being 0.1 meters, thereby obtaining a virtual operating environment dataset. This virtual operating environment dataset includes a gridded map and a list of constraint parameters to ensure the accuracy of subsequent simulations.

[0255] Step S720: Based on the virtual operating environment dataset, use a path verification tool to simulate the execution path, perform collision detection on the path nodes, and determine the executability judgment result of the path.

[0256] The feasibility of a path is determined using the following formula:

[0257] (40)

[0258] In formula (40), This represents the path security score output by the path verification tool. Indicates the number of time steps in the simulation. Indicates the penalty coefficient. Indicates the first The collision risk value at each time step. The threshold representing the collision risk is used to penalize risks exceeding the threshold using an exponential function, and the average is calculated to obtain the path safety score.

[0259] Based on the virtual operating environment dataset, a path verification tool is used to simulate the grasping execution path. This path verification tool can be a component such as the MoveIt framework. It simulates the movement trajectory of the robot arm by loading the dataset and performs collision detection on the path nodes. For example, in the simulation, it checks whether the path node [1.2, 2.3, 0.9] overlaps with virtual obstacles. The detection process involves a ray casting algorithm to determine the intersection point. If the distance between nodes is less than the safety threshold of 0.05 meters, it is marked as a potential collision, thereby determining the executability judgment result of the path. For example, a boolean value is output to indicate whether the path is unobstructed. This result helps to identify potential problems early.

[0260] Step S730: If the feasibility judgment result of the path shows that there is an obstacle, the path correction tool is used to make local adjustments to the blocked nodes, and the correction path data is obtained by combining real-time monitoring data.

[0261] The validation metrics for the corrected path data are derived using the following formula:

[0262] (41)

[0263] In formula (41), This represents the validation metrics for the corrected path data. Indicates the verification time window. express Real-time monitoring data status values ​​at all times. express The predicted state value of the path is constantly adjusted.

[0264] If the feasibility assessment of the path indicates an obstacle, a path correction tool is used to make local adjustments to the obstructed nodes. This path correction tool, such as an optimizer based on the A* algorithm, will generate alternative sub-paths for the obstructed nodes, such as [1.5, 2.0, 1.0], and verify them by combining real-time monitoring data. For example, the current image feedback obtained from the camera is incorporated into the adjustment. By iteratively comparing the length and safety of the new path, the corrected path data is finally obtained, such as updating the node sequence [1.4, 1.9, 0.95]. This process ensures that the path remains feasible in a dynamic environment.

[0265] Step S740: Combining the corrected path data and scene matching data, use a knowledge graph update tool to incrementally adjust the multimodal knowledge graph, perform data fusion for new scene features, and judge the adaptive update result of the adjusted multimodal knowledge graph.

[0266] The updated multi-modal knowledge graph is obtained by the following formula:

[0267] (42)

[0268] In formula (42), denotes the updated multi-modal knowledge graph, denotes the knowledge graph at the current time, denotes the incremental update part based on the corrected path data, denotes the incremental update part based on the scene matching data, denotes the incremental fusion operation of the multi-modal knowledge graph.

[0269] The adaptive update result of the adjusted multi-modal knowledge graph is obtained by the following formula:

[0270] (43)

[0271] In formula (43), denotes the adaptive evaluation value of the adjusted multi-modal knowledge graph, denotes the total number of entity nodes evaluated, denotes the feature representation of the th entity after updating, denotes the feature representation of the th entity before updating, denotes the feature change tolerance parameter of the th entity, denotes the importance weight of the th entity in the multi-modal knowledge graph.

[0272] Combined with the corrected path data and the scene matching data, the knowledge graph updating tool is used to incrementally adjust the multi-modal knowledge graph. Here, the knowledge graph updating tool refers to a Neo4j database management system, which adds new nodes and relationships through incremental algorithms, and performs data fusion for new scene characteristics. For example, the node data of the corrected path is matched with the lighting change characteristics of the scene, and the fusion process involves semantic matching to calculate the similarity. If the similarity exceeds 0.8, the multi-modal graph is updated to the crawling strategy entity. The adaptive update result of the multi-modal knowledge graph is output as an adaptive score of 0.92, indicating that the multi-modal knowledge graph has been optimized to adapt to the new environment. This update can improve the long-term learning ability of the robot system.

[0273] See Figure 2The embodiment provides a robot grasping target recognition and positioning system based on a knowledge graph, which is used for executing the robot grasping target recognition and positioning method based on the knowledge graph, and comprises a multi-modal knowledge graph construction module 10, a vector representation form determination module 20, an object recognition result acquisition module 30, a pose mapping relationship acquisition module 40, a fine-grained grasping operation sequence determination module 50, a grasping execution path acquisition module 60 and a multi-modal knowledge graph updating module 70. The multi-modal knowledge graph construction module 10 is used for constructing a multi-modal knowledge graph through multi-source data integration. The multi-modal knowledge graph models the relationship between object appearance features and grasping point pose parameters in a triple form, and obtains a structured node network containing object appearance description and an optimal grasping trajectory. The vector representation form determination module 20 is used for acquiring environment perception data collected by a robot vision module according to the multi-modal knowledge graph, processing the environment perception data by using a data processing method, and determining a vector representation form corresponding to a target object. The object recognition result acquisition module 30 is used for querying a matching object node by using an entity linking technology if the similarity of the vector representation form and nodes of the multi-modal knowledge graph exceeds a preset threshold, judging and solving object recognition ambiguity in a complex scene, and obtaining an accurate object recognition result. The pose mapping relationship acquisition module 40 is used for retrieving grasping point pose parameters in the multi-modal knowledge graph by using the object recognition result, acquiring related triple data, and obtaining a pose mapping relationship used for coordinate conversion. The fine-grained grasping operation sequence determination module 50 is used for performing a coordinate conversion operation by using the pose mapping relationship, converting perception coordinate system parameters into robot coordinate system values, and determining a fine-grained grasping operation sequence. The grasping execution path acquisition module 60 is used for combining the fine-grained grasping operation sequence and real-time visual feedback data, processing object attitude deviation by using a dynamic adjustment algorithm, and obtaining an optimized grasping execution path. The multi-modal knowledge graph updating module 70 is used for simulating a robot operation environment based on the optimized grasping execution path, evaluating path effectiveness, and updating the multi-modal knowledge graph to adapt to a new scene.

[0274] Compared with the prior art, the robot grasping target recognition and positioning method and system based on the knowledge graph provided by the embodiment construct a triple knowledge graph containing object appearance features and grasping point pose parameters through multi-source data integration, realize a structured node network, further acquire environment perception data of a robot vision module, determine a vector representation of a target object after processing, use entity linking technology to query a matching node if the similarity exceeds a threshold, solve recognition ambiguity to obtain an accurate result, retrieve pose parameters in a knowledge graph based on the result, perform coordinate conversion to generate a fine-grained grasping sequence, optimize a path by using a dynamic adjustment algorithm in combination with real-time visual feedback, and update the knowledge graph by simulation and evaluation to adapt to a new scene. The embodiment highlights the core role of the knowledge graph in grasping optimization, improves the accuracy, robustness and self-adaptive ability of robot operation, and is suitable for industrial automation and other fields.

[0275] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such variations and modifications as fall within the scope of the present application. It is apparent that those skilled in the art can modify and adapt the present application in various ways without departing from the spirit and scope of the present application. It is therefore intended that the present application encompass all such modifications and variations as fall within the scope of the claims and their equivalents.

Claims

1. A knowledge graph-based method for target recognition and localization in robot grasping, characterized in that, Includes the following steps: S100. A multimodal knowledge graph is constructed by integrating multi-source data. The multimodal knowledge graph uses triples to model the relationship between the appearance features of an object and the pose parameters of the grasping point, resulting in a structured node network that includes the object's appearance description and the optimal grasping trajectory. S200. Based on the multimodal knowledge graph, obtain the environmental perception data collected by the robot vision module, process the environmental perception data using data processing methods, and determine the vector representation form corresponding to the target object. S300. If the similarity between the vector representation and the nodes of the multimodal knowledge graph exceeds a preset threshold, then the entity linking technology is used to query and match object nodes, judge and resolve object recognition ambiguities in complex scenarios, and obtain accurate object recognition results. S400. Using the object recognition result, retrieve the pose parameters of the grasping point in the multimodal knowledge graph, obtain relevant triplet data, and obtain the pose mapping relationship for coordinate transformation; S500: Perform coordinate transformation operation through the pose mapping relationship to convert the sensor coordinate system parameters into robot coordinate coefficient values ​​and determine the fine-grained grasping operation sequence. S600. Combining the fine-grained grasping operation sequence and real-time visual feedback data, a dynamic adjustment algorithm is used to process the object posture deviation and obtain the optimized grasping execution path. S700: Based on the optimized grasping execution path, simulate the robot's operating environment, evaluate the effectiveness of the path, and update the multimodal knowledge graph to adapt to the new scenario; Step S100 includes: S110. Obtain the appearance feature data of the target object from the image sensor and depth sensor through a multi-source data acquisition tool, perform preprocessing operation on the appearance feature data, and use a noise reduction tool to smooth the preprocessed appearance feature data to obtain clear appearance feature description information. S120. Based on the appearance feature description information, use a three-dimensional reconstruction tool to generate three-dimensional structural data of the target object. When generating the three-dimensional structural data, if a data missing area is detected, use an interpolation tool to fill in the data missing area to determine a complete three-dimensional structural description. S130. For the three-dimensional structure description, a path planning tool is used to calculate the pose parameters of the grasping point, obtain multiple candidate grasping trajectory points corresponding to the three-dimensional structure description, sort the candidate grasping trajectory points by a comparison tool, and determine the optimal grasping trajectory parameters. S140. Using a knowledge graph construction tool, the appearance feature description information is associated and mapped with the optimal crawling trajectory parameters to generate a triplet relationship network containing structured nodes, thereby obtaining a multimodal knowledge graph. Step S200 includes: S210. Obtain the raw perception data stream from the environment through the vision module, and perform preliminary processing on the raw perception data stream using a noise reduction tool to obtain cleaned basic perception data. S220. Based on the basic perception data, the contour information of the target object is separated using a feature extraction tool. If the clarity of the contour information is lower than a preset threshold, it is optimized using an enhancement tool to determine the complete contour description information. S230. Based on the contour description information, a data fusion tool is used to combine depth information to generate spatial position data of the target object, obtain the corresponding three-dimensional coordinate information, and determine the preliminary vector representation of the target object. S240. The preliminary vector representation is calibrated using a coordinate mapping tool. If the deviation of the calibrated vector representation exceeds a preset threshold during calibration, an adjustment tool is used to correct it, thereby obtaining the vector representation of the target object. Step S300 includes: S310. Based on the vector representation, at least one candidate node data is obtained from the pre-established multimodal knowledge graph, and a feature comparison tool is used for preliminary screening to obtain a set of nodes that are close to the features of the target object. S320. If there are multiple similar nodes in the node set, the node set is filtered a second time by a data fusion tool in combination with environmental context information to determine the single node data that matches the current scene. S330. Use a link verification tool to detect the reliability of the association between the single node data and the target object. If the reliability of the association between the single node data and the target object is lower than a preset threshold, obtain backup node data from the multimodal knowledge graph for supplementary comparison and determine the final matching node result. S340. Based on the final matching node result, generate the identification label of the target object using a mapping tool, and verify the identification label using a multimodal data consistency verification tool to obtain an accurate object identification result.

2. The robot grasping target identification and localization method based on knowledge graph according to claim 1, characterized in that, Step S400 includes: S410. Based on the object recognition result, extract at least one relevant grasping point pose parameter data from the multimodal knowledge graph, and use a data filtering tool to perform preliminary classification of the grasping point pose parameter data to obtain a set of pose parameters matching the target object. S420. If there are multiple candidate data in the pose parameter set, the pose parameter set is refined and filtered by the context environment matching tool in combination with scene information to determine the pose parameter data of a single grasping point that matches the current grasping task. S430. For the pose parameter data of the single grasping point, the corresponding triplet data is obtained from the multimodal knowledge graph using a triplet extraction tool, and the triplet data is structured using a data mapping tool to obtain a preliminary pose mapping relationship for coordinate transformation. S440. The preliminary pose mapping relationship is checked using a consistency verification tool. If the verification result does not match the preset threshold, backup data is obtained from the multimodal knowledge graph for supplementary comparison to determine the accurate pose mapping relationship.

3. The robot grasping target identification and localization method based on knowledge graph according to claim 1, characterized in that, Step S500 includes: S510. Perform coordinate transformation operation through pose mapping relationship to convert the sensor coordinate system parameters into robot coordinate coefficient values; S520. Determine the fine-grained grasping operation sequence based on the converted robot coordinate coefficient values.

4. The robot grasping target identification and localization method based on knowledge graph according to claim 1, characterized in that, Step S600 includes: S610. Real-time visual feedback data is acquired through a visual sensor, continuous image capture is performed on the target object, and the target object features of the real-time visual feedback data are extracted using an image processing tool to obtain the preliminary position and posture information of the target object. S620. Based on the preliminary position and attitude information of the target object, combined with the pre-established deviation database, a comparison tool is used to determine the deviation between the actual attitude and the expected attitude of the target object. If it exceeds the preset threshold, deviation correction parameters are generated. S630. Data fusion is performed using the deviation correction parameters and fine-grained operation sequence, and the path planning tool is used to dynamically adjust the capture execution path to determine the adjusted path node data. S640. Based on the adjusted path node data and the real-time visual feedback data, a data integration tool is used for secondary verification to obtain the optimized crawling execution path.

5. The robot grasping target identification and localization method based on knowledge graph according to claim 4, characterized in that, Step S700 includes: S710. Based on the optimized capture and execution path, an environmental simulation tool is used to digitally reconstruct the target operating environment, and data is extracted to address the spatial constraints in the scene to obtain a virtual operating environment dataset. The virtual operating environment dataset is derived using the following formula: ； in, This represents the final generated virtual operating environment dataset. This indicates the total number of simulation scenarios. Indicates the first Geometric structure data for each scene, Indicates the first Attribute information data for each scenario, Indicates the first Visualized data for each scenario This indicates that data filtering filters are used to ensure the quality of the dataset; S720. Based on the virtual operating environment dataset, a path verification tool is used to simulate the execution of the crawling path, and collision detection is performed on the path nodes to determine the executability judgment result of the path. S730. If the feasibility assessment result of the path shows that there is an obstacle, the path correction tool is used to make local adjustments to the blocked nodes, and the correction path data is obtained by combining real-time monitoring data. S740. Combining the corrected path data with the scene matching data, the knowledge graph update tool is used to incrementally adjust the multimodal knowledge graph, and data fusion is performed for new scene features to determine the adaptive update result of the adjusted multimodal knowledge graph.

6. The robot grasping target identification and localization method based on knowledge graph according to claim 5, characterized in that, In step S720, the executability determination result of the path is obtained by the following formula: ； in, This represents the path security score output by the path verification tool. Indicates the number of time steps in the simulation. Indicates the penalty coefficient. Indicates the first The collision risk value at each time step. The threshold representing the collision risk is used to penalize risks exceeding the threshold using an exponential function, and the average is calculated to obtain the path safety score.

7. A knowledge graph-based robot target recognition and localization system, used to execute the knowledge graph-based robot target recognition and localization method as described in any one of claims 1 to 6, characterized in that, include: The multimodal knowledge graph construction module (10) is used to construct a multimodal knowledge graph by integrating multi-source data. The multimodal knowledge graph adopts the form of triples to model the relationship between the appearance features of objects and the pose parameters of the grasping points, and obtains a structured node network containing the appearance description of objects and the optimal grasping trajectory. The vector representation determination module (20) is used to obtain environmental perception data collected by the robot vision module based on the multimodal knowledge graph, process the environmental perception data using data processing methods, and determine the vector representation of the target object. The object recognition result acquisition module (30) is used to query matching object nodes through entity linking technology if the similarity between the vector representation and the nodes of the multimodal knowledge graph exceeds a preset threshold, to judge and resolve object recognition ambiguities in complex scenarios, and to obtain accurate object recognition results. The pose mapping relationship acquisition module (40) is used to retrieve the pose parameters of the grasping point in the multimodal knowledge graph using the object recognition result, obtain relevant triplet data, and obtain the pose mapping relationship for coordinate transformation; The fine-grasping operation sequence determination module (50) is used to perform coordinate transformation operation through the pose mapping relationship, convert the sensor coordinate system parameters into robot coordinate coefficient values, and determine the fine-grasping operation sequence. The grasping execution path acquisition module (60) is used to combine the fine-grained grasping operation sequence and real-time visual feedback data, and use a dynamic adjustment algorithm to process the object posture deviation to obtain the optimized grasping execution path; The multimodal knowledge graph update module (70) is used to simulate the robot's operating environment based on the optimized grasping execution path, evaluate the effectiveness of the path, and update the multimodal knowledge graph to adapt to the new scenario.

Images