An industrial part knowledge graph construction method

Abstract

The application provides an industrial part knowledge graph construction method. First, a pre-trained visual language large model is used for global context understanding and semantic preliminary screening of complex industrial images. Second, a lightweight vertical target classification network is introduced as a classification model, and the semantic re-calibration and representation enhancement of the large model fuzzy prediction are combined with the industrial vertical prior knowledge base. Third, the extracted accurate entity information is fused with the spatial topology and physical gasket relationship between parts to construct a semantic rigorous professional field knowledge graph. Finally, based on the graph analysis of the mechanical coupling and operation time sequence constraints between parts, a dynamic grabbing logic reasoning model is established, and an optimal grabbing operation sequence that meets the industrial safety specifications is automatically generated. The method can be applied to high-reliability grabbing operations of industrial robots in complex stacking and occlusion environments.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph construction technology, specifically to a method for constructing industrial parts graphs based on the collaboration of large and small models. Background Technology

[0002] With the deepening development of industrial automation, deep learning-based visual perception technology has become the core of intelligent manufacturing. Scene graph generation and industrial knowledge graph construction aim to provide semantic support for robots' autonomous decision-making by identifying part entities and their spatial topological relationships.

[0003] Existing scene graph generation methods typically employ a general object detection framework (such as Faster R-CNN or YOLO series) combined with a relational reasoning module to generate scene graphs. However, these methods are highly dependent on high-quality labeled datasets to train the model to learn the bounding boxes and category information of objects. In other words, a massive number of samples are needed to cover the morphological changes of objects under different viewpoints, lighting, and backgrounds. Otherwise, the generalization performance of the model will be greatly limited, making it difficult to cope with the variability in real-world scenes.

[0004] For specific targets or cross-domain tasks, mainstream methods tend to utilize large-scale pre-trained general visual language models (VLMs) for zero-shot or few-shot transfer learning, attempting to capture semantic relationships between parts through cross-modal semantic alignment. However, in actual industrial operations, existing perception solutions still face the following key technical challenges: First, general-purpose large models face barriers in semantic alignment within vertical domains. While existing VLMs possess strong cross-modal generalization capabilities, they often lack deep prior knowledge of specific vertical categories when dealing with highly specialized industrial parts, resulting in semantic representations that are general but not precise, making it difficult to achieve high-accuracy knowledge matching. This directly limits the depth of understanding of complex visual relationships by large models in specific industrial scenarios, leading to gaps in the generated knowledge graph's professional logic.

[0005] Secondly, severe overlap and occlusion in industrial environments create "blind spots" in feature representation. In scenarios where parts are scattered and stacked or densely distributed on production lines, the global contour of the target is often significantly obscured, causing traditional detection and recognition algorithms to fail due to the lack of key global features. Deep neural networks typically rely on complete morphological features for modeling, but under occlusion conditions, the model struggles to maintain robust representation capabilities with only incomplete visual information, leading to the loss of graph nodes or misjudgment of relationships, failing to meet the high reliability requirements of industrial applications.

[0006] Finally, the generation of gripping sequences for complex assembly and sorting tasks is highly uncertain in logic. In multi-object stacking scenarios, the gripping sequence depends not only on spatial geometry but also on the deep coupling of physical overlap between parts, mechanical balance, and operational logic. Traditional heuristic algorithms or simple models struggle to accurately model this dynamic sequence relationship, leading to generated gripping paths prone to collisions or operational failures due to disruption of stack stability. This becomes a bottleneck restricting industrial robots from performing complex and delicate tasks.

[0007] Therefore, how to combine prior knowledge of vertical categories to improve the understanding of large models in professional fields, enhance the local representation ability of models under extreme occlusion conditions, and generate a scientific and reasonable grasping logic order are the pain points that urgently need to be solved in the field of industrial vision and intelligent decision-making. Summary of the Invention

[0008] To address the problems of low semantic alignment accuracy, high inference cost of large-scale pre-trained models, and uncertain grasping logic in complex stacked environments in existing scene graph generation methods, this invention proposes an industrial parts knowledge graph construction framework based on large- and small model collaborative driving. This framework enhances the model's understanding of vertical domains and the reliability of operational decisions through the following key components:

[0009] First, a matching strategy based on general-vertical cross-dimensional semantic alignment is proposed. This component utilizes the general VLM Qwen-vl-max, which has strong generalization capabilities, as a "teacher guide" to perform multi-dimensional image description, local object attribute description, and preliminary target localization on the original industrial images, extracting global semantic context and focusing on outputting the part categories of the industrial scene. Subsequently, a lightweight vertical object classification network based on ResNet18 is introduced as a student model, taking the local feature representation output by the large model as input and combining it with a pre-set industrial vertical prior knowledge base for semantic refinement. This strategy achieves accurate mapping from general semantics to professional domain categories through the recalibration of the large model's fuzzy predictions by the small model, effectively solving the problem of "semantic illusion" and category misclassification caused by the large model's lack of professional knowledge in specific industrial scenarios.

[0010] Furthermore, a knowledge graph construction mechanism based on global-local collaborative feature extraction is proposed. Addressing the pain point of feature loss caused by overlap and occlusion in industrial environments, this mechanism no longer relies solely on the global geometric contour of the target. Instead, it explicitly extracts weak features such as local textures and key edges of the parts by dividing the image into 2x2 sub-images and randomly masking one of these sub-images. When the global representation is impaired, this mechanism can dynamically activate local feature compensation logic, ensuring that the model maintains highly robust target representation even under extreme occlusion conditions. By fusing the extracted precise entity information with the physical constraints between parts, and combining this with the image description capabilities of the visual language model Qwen-vl-max, this invention can generate a semantically rigorous professional domain knowledge graph.

[0011] Finally, a dynamic grasping sequence generation algorithm based on physical logic constraints is proposed. Based on the generated knowledge graph, this algorithm establishes a dynamic temporal decision-making model by analyzing the spatial topology and physical overlapping relationships between entities in the graph. This algorithm solves the problem that traditional geometric rules cannot model the mechanical coupling between objects. By simulating the balance and operational conflicts during the grasping process, it automatically generates a grasping logical sequence that meets industrial safety standards, providing precise execution path guidance for the robot's high-reliability operations.

[0012] This invention provides a method for constructing and planning the atlas of industrial parts based on a collaboration between large and small models. The method comprises three stages: a VLM-guided target localization and semantic initial screening module, a small-model semantic matching and representation enhancement module guided by vertical category prior knowledge, and a knowledge graph-based physical logic constraint and grasping sequence generation module. The VLM-guided target localization and semantic initial screening module is responsible for the first stage task of determining "which regions to process." This module utilizes the powerful zero-shot generalization capability of a pre-trained visual language large model to perform global contextual understanding on the input complex industrial images, outputting preliminary image descriptions, target object bounding boxes, and coarse-grained attribute descriptions. This module, through the generalized knowledge of the large model, determines key candidate regions and preliminary semantic benchmarks for subsequent tasks, solving the problem of insufficient initial labeled data in vertical domains. The small-model semantic matching and representation enhancement module guided by vertical category prior knowledge is responsible for the core task of "how to perform fine representation." It introduces a lightweight vertical target classification network based on ResNet18 as a student model, receiving local feature representations output by the large model. This module employs a global-local collaborative extraction framework, combined with a vertical prior knowledge base for industrial scenarios, to calibrate and refine the general semantics output by the large model. In complex occlusion environments, this module explicitly utilizes local texture features to achieve robust entity semantic representation, eliminating semantic illusions in the general large model and ensuring accurate mapping of professional domain categories. The physical logic constraints and grasping sequence generation module based on knowledge graph reasoning is the decision-making stage for achieving task closure in this invention. It integrates the precise entity information and spatial topological relationships extracted by the aforementioned modules to construct a professional domain knowledge graph oriented towards the industrial environment. Based on this, the module establishes a dynamic grasping logic reasoning model by analyzing the physical occlusion, mechanical balance, and operational timing constraints between parts in the graph. This module overcomes the deficiency of traditional methods in lacking physical interaction understanding and can automatically generate highly reliable grasping operation sequences that meet the requirements of actual processes. Specifically, it includes the following steps:

[0013] S1. Determine what's in the scene: Use GPT4o or Qwen-vl-max from the pre-trained Visual Language Model (VLM) to describe the input image. The prompts require the VLM to clearly state that this is an industrial scene with its own parts. The VLM should describe the image as comprehensively as possible and list the names of all objects in the image. The image description and object names are stored in a dictionary with the format {"description":"A gear motor and a heat sink are on the table.","objects":["gear motor","heat sink"]}.

[0014] S2. Determine the target location: Use the pre-trained open-world object detection model Grounding-DINO to detect targets in the image. The cue words are provided by all object names output by VLM, that is, the values ​​corresponding to the "objects" key in the output dictionary ["gear motor", "heat sink"]. The output of Grounding-DINO is the detected target box.

[0015] S3. Determine the orientation of the object to be grasped: The open-world object segmentation algorithm SegmentAnything is used to perform fine-grained segmentation of the object within the target box detected by Grounding-DINO, and the shape approximation of the segmented pixels is performed to find the longest and shortest sides with approximate slopes. The target box detected by Grounding-DINO is rotated so that the rotated target box fits the longest and shortest sides of the segmented pixels. This step is to ensure that the target detection box fits the target as closely as possible and avoids feature interference caused by background interference and other parts occluding the target object.

[0016] S4. Determine the target object: In industrial application scenarios, we need to clearly define what each object is, such as the heat sink of model R1. However, the categories obtained using open-world image descriptions may not include this category. Therefore, we need to match the categories output in the open-world description with the known categories in the vertical application scenario. We use a small classification model for a professional domain to perform vertical semantic classification. This small model uses ResNet18 as the backbone network and outputs the confidence score of each category with fully connected layers + Softmax. Based on the required number of categories in the industrial scenario, the output categories are set to 25.

[0017] S5. Determine the spatial relationships and grasping order of objects: After clarifying the category, location and orientation of each part in the industrial scene, use the visual language model Qwen-vl-max to analyze the spatial position of the parts, mainly to clarify the stacking relationship, output the spatial position between each part, and input grasping prompt words to prioritize grasping upper-level parts and surrounding parts to avoid the actuator damaging the parts or grasping failure.

[0018] The advantages of this invention are that, given a specific industrial part grasping scenario, the model can understand the vertical semantics of each part, unlike the fuzzy definitions given by the general world model, enabling the robot to grasp specific parts. Furthermore, for occluded targets, the constructed global and local inference framework can effectively detect targets, avoiding missed detections and false detections. Finally, based on the obtained spatial relationships of the parts, a visual language model is used to determine the grasping order, which can effectively improve the success rate of industrial part grasping. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0020] Figure 1 This is a diagram of the industrial parts inspection method based on size model collaboration of the present invention;

[0021] Figure 2 This is a diagram of the industrial parts vertical category knowledge target classification small model method of the present invention;

[0022] Figure 3 This is a diagram illustrating the implementation effect of the present invention in an industrial setting for target localization, segmentation, and classification.

[0023] Figure 4 This is an implementation diagram of the grabbing sequence generation of the present invention in an industrial scenario. Detailed Implementation

[0024] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific examples. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0025] A flowchart of a method for constructing an industrial parts knowledge graph based on large-small model collaboration is shown below. Figure 1 As shown, the vertical knowledge target classification small network method in this method is as follows: Figure 2 As shown, the overall process consists of four steps: initial semantic screening and target localization using a large model, fitting and enhancing the representation of the rotating detection box, constructing a knowledge graph, and generating dynamically extracted sequences. Figure 1 The image shows a collaboratively driven generation framework. Figure 2 The semantic recalibration mechanism of small networks is revealed, as follows:

[0026] Step S1: First, the raw industrial images of complex scenes captured by industrial cameras are input into the pre-trained visual language model Qwen-vl-max, which has strong zero-shot generalization capabilities. By designing specific prompts, the VLM is guided to perform global contextual understanding of the images. The prompts require the VLM to clearly define the current vertical industrial work scene, describe the image as comprehensively as possible, and output a list of all parts appearing in the image. The global image description and coarse-grained object name list output by the large model are saved in dictionary format as the semantic benchmark for subsequent modules. This step utilizes the generalized knowledge of the large model to complete the initial semantic screening of the industrial scene, effectively solving the target recognition problem in the initial stage of vertical industrial fields when large-scale labeled data is lacking.

[0027] Step S2: After obtaining the list of object names output in Step S1, this list (i.e., the text values ​​corresponding to the "objects" key in the dictionary are prompt words) is input into the pre-trained open-world object detection model Grounding-DINO. Grounding-DINO performs multi-dimensional matching on the original industrial image based on the text prompts to achieve preliminary target localization of industrial parts. For each target detected in the image, this module outputs the corresponding horizontal detection box coordinates.

[0028] Step S3: To eliminate feature loss caused by overlapping, occlusion, and background interference in complex industrial stacking environments, this method introduces the open-world object segmentation algorithm SegmentAnything (SAM) for fine-tuning. The horizontal detection box output from Grounding-DINO in Step S2 is used as a cue word input to SAM to perform fine-grained pixel-level segmentation of the target object within the box, obtaining a binary mask image of the object. Subsequently, the segmented target pixel contour is geometrically approximated. By calculating the pixel distribution of the contour edges, the longest and shortest sides that fit the object's shape are found. The approximate slope is calculated based on the direction of the longest side. This slope is used to rotate and tightly fit the original horizontal target detection box, transforming it into a rotated target detection box with directional information. Through this rotational fitting step, the detection box fits the target part to the greatest extent, explicitly eliminating feature interference from the background and other stacked parts.

[0029] Step S4: Since the object categories given by the open-world large model are usually of general semantics (e.g., "heatsink"), they cannot directly meet the mapping requirements of industrial-grade high-precision operations for specific models (e.g., "R1 model heatsink"), and may even lead to "semantic illusions." Therefore, this step introduces a lightweight vertical object classification network as a subdivision model. This model uses ResNet18 as the backbone network, combining fully connected layers and Softmax for subdivision. Based on the actual industrial parts scenario, the Softmax layer finally outputs 25 categories. Figure 2 As shown, the local feature representation output by the large model is used as input to collaboratively extract weak features such as local texture and key edges of the parts, and semantic refinement and recalibration are performed in combination with the preset industrial vertical prior knowledge base. When the global representation is occluded and damaged, this mechanism can dynamically activate the local feature compensation logic, realizing the accurate mapping from general semantics to professional domain categories.

[0030] Step S5: After clarifying the precise vertical semantics, high-precision position, and orientation angle of each industrial part, this precise entity information is fused with the physical constraints between parts to construct a professional domain knowledge graph for the industrial environment. The knowledge graph is represented in a graph structure. Based on this, the graph information and grasping prompts are input into the visual language model Qwen-vl-max to establish a dynamic temporal decision model. This algorithm simulates and calculates the balance and operational conflicts during the grasping process by analyzing the spatial overlap, physical contact surface, and upper and lower mechanical balance relationships between entities in the graph. The temporal decision model follows the physical logic constraint of "prioritizing grasping unobstructed upper-layer parts and parts around the edges," dynamically updating the grasping priority score and generating a discrete optimal dynamic grasping logical sequence that meets industrial safety standards, providing precise path guidance for the robot's high-reliability operation.

[0031] like Figure 3 and Figure 4 As shown, the effectiveness of this method has been verified in a real-world scenario of complex industrial parts stacking. Figure 3 The system demonstrated its effectiveness in performing high-precision target positioning, fine-grained boundary segmentation, and accurate vertical classification of various industrial parts (such as different models of motors and heat sinks) under extreme occlusion and overlap conditions, with no missed detections or misclassifications. Figure 4 The study demonstrates the effectiveness of the system in guiding the robot's dynamic grasping sequence and execution path in real time, based on the constructed professional knowledge graph. Experimental results show that this method significantly improves semantic alignment accuracy in vertical industrial scenarios and greatly increases the robot's success rate in grasping specified parts.

[0032] To simplify the explanation of this method, the above figures and text are described as a series of steps. However, it should be understood and understood that these methods are not restricted by the order of operations, because some actions may occur in different orders when implemented according to one or more steps, but those skilled in the art can understand the principle behind the actions.

[0033] Although illustrative specific embodiments of the invention have been described step by step to enable those skilled in the art to understand them, the invention is not limited to the scope of the specific embodiments. Those skilled in the art can make various modifications or variations within the scope of the claims, as long as such variations are within the spirit and scope of the invention as defined and determined by the appended claims.

Claims

1. A method for constructing and generating a knowledge graph of industrial parts, characterized in that, Includes the following steps: S1. Use a pre-trained visual language large model to perform global context understanding and multi-dimensional description on the input original industrial parts scene image, extract the global semantic context of the image, and output all the identified candidate object names in list form and save them as the initial dictionary data structure. S2. Input the output list of object names as text prompts into the pre-trained open-world object detection model, perform object detection through text-visual cross-modal matching, and output the horizontal candidate object detection boxes of each industrial part in the image. S3. The open-world target segmentation algorithm is used to perform pixel-level fine-grained segmentation on the output horizontal candidate target detection box. The obtained object binary mask pixels are geometrically approximated to determine the slope of its long and short sides. The horizontal candidate target detection box is rotated and tightly fitted in this way to transform it into a rotated target detection box with orientation angle information. S4. Input the features within the rotating target detection box area into the lightweight vertical target classification network to collaboratively extract the global contour and local weak texture features of the target, and combine them with the preset industrial vertical prior knowledge base for semantic recalibration, so as to accurately map the fuzzy semantics output by the general large model to the professional part category model of the vertical field. S5. Integrate the high-precision spatial location, orientation, and professional category of the acquired parts to construct a professional domain knowledge graph; based on the knowledge graph, analyze the spatial topology and physical overlay constraints of the part entities, establish a dynamic temporal decision model, and automatically generate the optimal dynamic grasping logic operation sequence.

2. The method for constructing and generating an industrial parts knowledge graph according to claim 1, characterized in that: The overall framework is divided into a target localization and semantic screening module guided by a large model, a small model semantic matching and representation enhancement module guided by vertical category prior knowledge, and a physical logic constraint and crawling sequence generation module based on knowledge graph reasoning. The target localization and semantic screening module guided by the large model corresponds to steps S1 and S2, the small model semantic matching and representation enhancement module corresponds to steps S3 and S4, and the physical logic constraint and crawling sequence generation module corresponds to step S5.

3. The method for generating an industrial parts knowledge graph according to claim 2, characterized in that: In step S1, the initial dictionary data structure includes global text description keys for the industrial scene from the large model, and coarse-grained object name list keys output by the large model.

4. The method for generating an industrial parts knowledge graph according to claim 1, characterized in that: In step S2, the open-world object detection model is Grounding-DINO, which uses the list of object names in the dictionary as prompt words to output the pixel coordinates of the top left and bottom right corners of the horizontal candidate object detection boxes of each detected object in the image.

5. The method for generating an industrial parts knowledge graph according to claim 1, characterized in that: In step S3, the open-world object segmentation algorithm is SegmentAnything, which receives horizontal candidate object detection boxes and extracts binary mask images; by finding the longest side that fits the shape of the object mask, the orientation angle is calculated based on the angle between the longest side and the horizontal axis of the image; the original box is recalibrated using the orientation angle to generate a rotated object detection box containing the center point coordinates, the length of the long side, the length of the short side, and the orientation angle.

6. The method for generating an industrial parts knowledge graph according to claim 1, characterized in that: In step S4, the lightweight vertical target classification network acts as a student model, receiving global-local collaborative features within the rotated target detection box region. The network classifier calculates the probability distribution of the part belonging to each professional category in the preset industrial vertical prior knowledge base, and uses the maximum posterior probability to determine the professional model of the part. When the industrial part is partially occluded, the lightweight vertical target classification network dynamically activates the local feature compensation logic, using local texture and key edge features to perform semantic recalibration on the fuzzy prediction of the general large model.

7. The method for generating an industrial parts knowledge graph according to claim 1, characterized in that: In step S5, the professional domain knowledge graph is represented by a graph structure, where vertices represent part entities and edges represent the spatial topology and physical overlapping relationships between parts. The dynamic temporal decision model analyzes the physical occlusion and mechanical balance relationship of each part in the graph, comprehensively evaluates the level of the part in the stacking relationship, the mechanical stability of the surrounding stacking system during grasping, and the probability of conflict during actuator operation, calculates the grasping priority score, and automatically generates the optimal dynamic grasping logic operation sequence that meets industrial safety specifications according to the order of scores from high to low.

Images