Bolt missing detection method based on cross-media and knowledge reasoning

Abstract

The invention discloses a bolt missing detection method based on cross-media and knowledge reasoning. The method comprises the following steps: constructing a bolt-fitting image-text pair data set and a bolt knowledge graph; a cross-media bolt knowledge fusion model is created, and the cross-media bolt knowledge fusion model generates overall text description of the image by fusing global information extracted by the fine-tuned MiniGPT-4 and local information captured by the YOLOv8; and creating a big language reasoning model based on the knowledge graph, guiding the big language model to perform knowledge reasoning on output information of the cross-media bolt knowledge fusion model by using the bolt knowledge graph through the big language reasoning model based on the knowledge graph, and obtaining an interpretable bolt missing detection result. According to the method, the collected bolt image can be detected, the bolt missing detection precision is improved, the bolt missing detection result has interpretability, and compared with other advanced missing bolt detection methods, the AP value and the mAP value are optimal.

Description

Technical Field

[0001] The present invention relates to the technical field of bolt detection, and in particular to a bolt missing pin detection method based on cross-media and knowledge reasoning. Background Art

[0002] Bolts are common small targets in the engineering field. They are responsible for connecting and fixing various components and are crucial to the safety and stability of the overall structure. In order to ensure the safe and stable operation of transmission lines, it is necessary to regularly inspect and maintain the bolts on the transmission lines. Traditional bolt detection mainly relies on manual inspections. This method is simple to operate, but inefficient and affected by the experience and vision of manual inspectors, which may lead to inaccurate detection results. With the development of drones and deep learning, machine vision algorithms are applied to images taken by drones to dynamically identify problems such as missing bolts. The current bolt missing pin detection method mainly relies on basic target detection algorithms, including object detectors in the first-stage algorithm (such as YOLO and SSD) and the second-stage algorithm (such as Faster R-CNN). These advanced algorithms have made significant progress in the field of bolt detection and have the ability to automatically identify bolts in images.

[0003] However, although these algorithms have shown excellent performance, they still have certain limitations and room for optimization. Bolts are tiny targets and the background is complex. Traditional target detection algorithms perform poorly when dealing with tiny targets because they are usually designed for larger and obvious targets. Bolts usually have a low number of pixels and a fuzzy appearance, which makes them difficult to clearly distinguish and locate in the image. During the recognition process, they may be affected by image noise, illumination changes and background interference, further reducing the recognition accuracy. Traditional target detection algorithms mainly focus on local features in the image, but have limited understanding of the context and external knowledge around the target. This defect becomes particularly obvious when dealing with small targets, especially tiny targets such as bolts. Bolts are often located in complex backgrounds, and their recognition requires more contextual information, such as spatial orientation, positional relationships of bolts, etc. This information is crucial for correctly locating and classifying bolts, but traditional algorithms find it difficult to use them effectively. Summary of the invention

[0004] The present invention provides a method for detecting missing bolt pins based on cross-media and knowledge reasoning, which can effectively solve the problem mentioned in the above background technology that deep learning algorithms are currently mainly used to detect bolts, and research is only based on single image data. However, bolts are tiny targets, and image features are difficult to fully extract, which easily leads to the difficulty of detecting bolts in the image. Secondly, when processing data, traditional models mainly rely on local knowledge, and their global knowledge processing capabilities are insufficient, resulting in an incomplete understanding of the target and background, affecting the accuracy of bolt detection. At the same time, the black box problem of the deep learning model makes its detection results lack interpretability.

[0005] To achieve the above object, the present invention provides the following technical solution: a bolt missing pin detection method based on cross-media and knowledge reasoning, comprising the following steps:

[0006] Step 1: Build a bolt-fitting image-text pair dataset and a bolt knowledge graph;

[0007] Step 2: Create a cross-media bolt knowledge fusion model. The cross-media bolt knowledge fusion model generates an overall text description of the image by fusing the global information extracted by the fine-tuned MiniGPT-4 and the local information captured by YOLOv8.

[0008] Step three, create a big language reasoning model based on the knowledge graph. The big language reasoning model based on the knowledge graph uses the bolt knowledge graph to guide the big language model to perform knowledge reasoning on the output information of the cross-media bolt knowledge fusion model to obtain explainable bolt missing pin detection results.

[0009] According to the above technical solution, in the step 1, in the process of constructing the bolt-fitting image-text pair data set, specifically in the transmission line, the bolt state is closely related to the background fittings and spatial orientation in which it is located;

[0010] Specifically, the following situations are included: the bolts with anti-vibration hammer as background hardware do not have missing pins; the bolts with bag-type suspension wire clamp as background hardware are bolts without pins if the space orientation is vertical, bolts with pins if the space orientation is horizontal; the bolts with heavy hammer and connecting plate as background hardware are bolts with pins if the space orientation is horizontal;

[0011] Collect image data and text data of the above four types of hardware to build the hardware-bolt test dataset FBTD, the hardware-bolt image-text pair fine-tuning dataset BIFD and the bolt knowledge graph BoltKG;

[0012] The hardware-bolt test dataset, hardware-bolt image-text pair fine-tuning dataset, and bolt knowledge graph are used to deal with three different tasks:

[0013] FBTD is used to evaluate performance in missing pin bolt detection applications;

[0014] BIFD is used in the fine-tuning process of MiniGPT-4;

[0015] BoltKG is used in the LLMKG knowledge reasoning task.

[0016] According to the above technical solution, in the step 1, in the process of constructing the bolt knowledge graph, the bolt knowledge graph is constructed based on the association knowledge between the bolt state and the background hardware and spatial orientation. The specific construction process includes the following steps:

[0017] 1) Collect relevant data, including relevant information and expert database in the power field;

[0018] 2) Extract knowledge from the collected data, identify and determine the key entities in the knowledge graph;

[0019] 3) Based on the professional knowledge in the power field, define the logical relationship between entities;

[0020] 4) Construct the bolt knowledge graph in the form of structured triples.

[0021] According to the above technical solution, in the step 2, when creating a cross-media bolt knowledge fusion model, the cross-media bolt knowledge fusion model is mainly composed of two modules: a global information extraction module and a local detail recognition module. The outputs of the two modules are combined to jointly realize the cross-media processing process from image to text;

[0022] The global feature representation and preliminary recognition results output by the global information extraction module are directly used as the input of the local detail recognition module to locate the bolt area and guide the subsequent fine detection;

[0023] The detailed feature description and out-of-stock detection results output by the local detail recognition module are input into the cross-media fusion and reasoning module together with part of the output structure of the global information extraction module to perform deep fusion and reasoning analysis of cross-media knowledge.

[0024] According to the above technical solution, the global information extraction module uses its powerful feature extraction capability to capture the global information of the bolt from the image. The module mainly uses MiniGPT-4 to fine-tune the input bolt-fitting image and text data to construct FB-GPT to fuse the image and text information of the bolt-fitting. In the bolt detection process, the input bolt-fitting image and question are first processed by FB-GPT, and then the model generates a global text description of the global feature representation of the detection image;

[0025] When the background hardware in the global information is a connecting plate, a heavy hammer or a bag-type suspension clamp, the hardware-bolt image enters the local detail recognition module for detection. Since there is no missing pin on the bolt of the shock-absorbing hammer, target detection is not required.

[0026] According to the above technical solution, in the subsequent stage of bolt detection, the local detail recognition module uses the high-performance target detection model YOLOv8 to perform in-depth and fine detection on the input bolt-fitting detection image, accurately identify multiple detail features of the bolt, including the spatial orientation of the bolt and the presence of the pin, use natural language to describe the detection information of YOLOv8, and finally output the bolt information description.

[0027] According to the above technical solution, the cross-media bolt knowledge fusion model combines the global text description output by the global information extraction module and the bolt information description of the local detail recognition module to finally form an overall text description. The cross-media fusion model finally generates a complete status description of the bolt, including its background hardware, bolt status and spatial orientation description information, completing the cross-media processing task from image to text.

[0028] According to the above technical solution, in step 3, the large language reasoning model based on the knowledge graph completes the knowledge transfer through LangChain, connects ChatGLM with the bolt knowledge graph, and reasonably uses the associated knowledge in BoltKG to guide the reasoning process of the large language model. The large language reasoning model based on the knowledge graph includes an input processing module and a multi-stage knowledge reasoning module;

[0029] The large language reasoning model based on knowledge graph combines the real-time and interpretability of knowledge graph with the reasoning ability of large language model, effectively overcoming the limitations of large model in the field of bolt missing pin detection in terms of knowledge real-time and insufficient coverage, and making the bolt missing pin detection results interpretable. The model forms prompts consisting of questions, related triples and historical messages through knowledge transfer, and performs knowledge reasoning based on these prompts.

[0030] According to the above technical solution, the input processing module combines the overall text description output by the cross-media bolt knowledge fusion model with the question according to the question template to construct the original question, and then decomposes the original question into each stage question. By combining the overall description with the preset question according to the question template, the module can generate one or more specific and structured original questions;

[0031] The original question is complex and difficult to answer in one go, so the module needs to break it down into multiple sub-questions, which usually have clearer query goals and are easier to process later.

[0032] According to the above technical solution, the multi-stage knowledge reasoning module first receives the post-processing stage questions and query requests from the input processing module, and then establishes a connection with the bolt knowledge graph BoltKG through the knowledge transfer mechanism to accurately retrieve knowledge triples related to the question. These triples widely cover the types of bolts, the specific states of with and without pins, the identification features of missing pins, and the key information of the bolt's orientation in space;

[0033] After retrieving the knowledge triples, the multi-stage knowledge reasoning module deeply integrates them with historical messages, including previous query results and detection records, to carefully construct a structured prompt, aiming to provide comprehensive and rich context information for the large language model;

[0034] Next, the large language model conducts in-depth reasoning analysis based on the prompts and generates corresponding phased answers. These answers are not only direct responses to the current query, but are also recorded and fed back into the "memory" system as important historical information in the subsequent reasoning process.

[0035] In this way, the multi-stage knowledge reasoning module can gradually accumulate and utilize previous reasoning results, optimize the subsequent question-answering process, and ensure that the final output of bolt missing pin detection results is both accurate and explainable.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention constructs a hardware-bolt image-text pair dataset, selects image data and text data of bolts on 4 types of hardware as the source of the dataset, associates these images with the types of hardware, and classifies them according to the types of hardware, thereby forming a cross-media dataset. By collecting the pin-containing state of the bolts and the knowledge associated with their spatial orientation and position, and using the entity-relationship-entity triple form to represent their potential relationship, a bolt knowledge graph is constructed, and a cross-media bolt knowledge fusion model is constructed by using the "big model + small model" method. First, the big model completes the preliminary recognition of the image to obtain global knowledge, and then uses the small model to identify the local knowledge about the bolts in the image, thereby fusing the local knowledge and global knowledge of the image. In order to obtain an explainable bolt missing pin detection result, the bolt knowledge graph is combined with the large language model, and the bolt knowledge graph is reasonably used to guide the reasoning process of the large language model.

[0038] 2. The present invention adopts cross-media information of bolts to enhance the ability to process complex information of images. It effectively integrates cross-media information and global and local knowledge of images by combining large and small models to enhance the global knowledge processing ability and effectively integrate global knowledge and local knowledge. It uses the bolt knowledge graph to perform knowledge reasoning on the image description information output by the cross-media bolt knowledge fusion model to obtain explainable detection results. The bolt missing pin detection method can not only detect the collected bolt images and improve the accuracy of bolt missing pin detection, but also make the bolt missing pin detection results explainable. Compared with other advanced missing pin bolt detection methods, the AP and mAP values ​​are both optimal.

[0039] In summary, the present invention proposes to combine a large model with a target detection model so as to fully learn the image information, and through the relationship between the bolt state and the background hardware, use the bolt knowledge graph to guide the large language model to reason about the image information, thereby obtaining explainable bolt missing pin detection and improving the accuracy of bolt missing pin detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0041] In the attached picture:

[0042] Figure 1 is a flow chart of a method for detecting missing bolt pins of the present invention;

[0043] Figure 2 It is a flow chart of the process of constructing a data set of images and texts of the present invention;

[0044] Figure 3 It is a flow chart of the process of constructing the bolt knowledge graph of the present invention;

[0045] Figure 4 is a schematic diagram of the cross-media bolt knowledge fusion model of the present invention;

[0046] Figure 5 It is a schematic diagram of the large language reasoning model based on knowledge graph of the present invention. DETAILED DESCRIPTION

[0047] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0048] Example: Figure 1 As shown, the present invention provides a technical solution, a bolt missing pin detection method based on cross-media and knowledge reasoning, comprising the following steps:

[0049] Step 1: Build a bolt-fitting image-text pair dataset and a bolt knowledge graph;

[0050] Step 2: Create a cross-media bolt knowledge fusion model. The cross-media bolt knowledge fusion model generates an overall text description of the image by fusing the global information extracted by the fine-tuned MiniGPT-4 and the local information captured by YOLOv8.

[0051] Step three, create a big language reasoning model based on the knowledge graph. The big language reasoning model based on the knowledge graph uses the bolt knowledge graph to guide the big language model to perform knowledge reasoning on the output information of the cross-media bolt knowledge fusion model to obtain explainable bolt missing pin detection results.

[0052] Based on the above technical solution, in step 1, in the process of constructing the bolt-fitting image-text pair dataset, specifically in the transmission line, the bolt state is closely related to the background fittings and spatial orientation in which it is located;

[0053] Specifically, the following situations are included: the bolts with anti-vibration hammer as background hardware do not have missing pins; the bolts with bag-type suspension wire clamp as background hardware are bolts without pins if the space orientation is vertical, bolts with pins if the space orientation is horizontal; the bolts with heavy hammer and connecting plate as background hardware are bolts with pins if the space orientation is horizontal;

[0054] Collect image data and text data of the above four types of hardware to build the Fitting-Bolt Test Dataset, the Fitting-Bolt Image-Text Pair Fine-Tuning Dataset, and the Bolt Knowledge Graph;

[0055] The hardware-bolt test dataset, hardware-bolt image-text pair fine-tuning dataset, and bolt knowledge graph are used to deal with three different tasks:

[0056] FBTD is used to evaluate performance in missing pin bolt detection applications;

[0057] BIFD is used in the fine-tuning process of MiniGPT-4;

[0058] BoltKG is used in the LLMKG knowledge reasoning task. The specific construction process of FBTD and BIFD is as follows: Figure 2 shown.

[0059] Based on the above technical solution, in step 1, in the process of constructing the bolt knowledge graph, the bolt knowledge graph is constructed based on the association knowledge between the bolt state and the background hardware and spatial orientation, such as Figure 3 As shown, the specific construction process includes the following steps:

[0060] 1) Collect relevant data, including relevant information and expert database in the power field;

[0061] 2) Extract knowledge from the collected data, identify and determine the key entities in the knowledge graph, including bolts and anti-vibration hammers;

[0062] 3) Based on the professional knowledge in the power field, define the logical relationship between entities, including the background relationship between bolts and hardware, and the status attributes of bolts;

[0063] 4) Construct the bolt knowledge graph in the form of structured triples.

[0064] Based on the above technical solution, in step 2, when creating a cross-media bolt knowledge fusion model, the cross-media bolt knowledge fusion model is mainly composed of two modules: a global information extraction module and a local detail recognition module. The outputs of the two modules are combined to jointly realize the cross-media processing process from image to text. Specifically, Figure 4 As shown;

[0065] The global feature representation and preliminary recognition results output by the global information extraction module are directly used as the input of the local detail recognition module to locate the bolt area and guide the subsequent fine detection;

[0066] The detailed feature description and out-of-stock detection results output by the local detail recognition module and part of the output structure of the global information extraction module enter the cross-media fusion and reasoning module for deep fusion and reasoning analysis of cross-media knowledge.

[0067] Based on the above technical solution, the global information extraction module uses its powerful feature extraction capability to capture the global information of the bolt from the image. This module mainly uses MiniGPT-4 to fine-tune the input bolt-fitting image and text data to build FB-GPT to fuse the image and text information of the bolt-fitting. In the bolt detection process, the input bolt-fitting image and question are first processed by FB-GPT, and then the model generates a global text description of the global feature representation of the detection image;

[0068] When the background hardware in the global information is a connecting plate, a heavy hammer or a bag-type suspension clamp, the hardware-bolt image enters the local detail recognition module for detection. Since there is no missing pin on the bolt of the shock-absorbing hammer, target detection is not required.

[0069] Based on the above technical solution, in the subsequent stage of bolt detection, the local detail recognition module uses the high-performance target detection model YOLOv8 to perform in-depth and fine detection on the input bolt-fitting detection image, accurately identify multiple detail features of the bolt, including the spatial orientation of the bolt and the presence of the pin, use natural language to describe the detection information of YOLOv8, and finally output the bolt information description.

[0070] Based on the above technical solution, the cross-media bolt knowledge fusion model combines the global text description output by the global information extraction module and the bolt information description of the local detail recognition module to finally form an overall text description. The cross-media fusion model finally generates a complete status description of the bolt, including its background hardware, bolt status and spatial orientation description information, completing the cross-media processing task from image to text.

[0071] Based on the above technical solution, in step 3, the large language reasoning model based on the knowledge graph completes the knowledge transfer through LangChain, connects ChatGLM with the Bolt knowledge graph, and reasonably uses the associated knowledge in BoltKG to guide the reasoning process of the large language model. The large language reasoning model based on the knowledge graph includes an input processing module and a multi-stage knowledge reasoning module;

[0072] The large language reasoning model based on knowledge graph combines the real-time and interpretability of knowledge graph with the reasoning ability of large language model, effectively overcoming the limitations of large model in the field of bolt missing pin detection in terms of knowledge real-time and insufficient coverage, and making the bolt missing pin detection results interpretable. The model forms prompts consisting of questions, related triples and historical messages through knowledge transfer, and performs knowledge reasoning based on these prompts, as follows: Figure 5 shown.

[0073] Based on the above technical solution, the input processing module combines the overall text description output by the cross-media bolt knowledge fusion model with the question according to the question template to construct the original question, and then breaks the original question into each stage question. By combining the overall description with the preset question according to the question template, the module can generate one or more specific and structured original questions;

[0074] The original question is complex and difficult to answer in one go, so the module needs to break it down into multiple sub-questions, which usually have clearer query goals and are easier to process later.

[0075] Based on the above technical solution, the multi-stage knowledge reasoning module first receives the post-processing stage questions and query requests from the input processing module, and then establishes a connection with the bolt knowledge graph BoltKG through the knowledge transfer mechanism to accurately retrieve knowledge triples related to the question. These triples widely cover the types of bolts, the specific states of with and without pins, the identification features of missing pins, and the key information of the bolt's orientation in space;

[0076] After retrieving the knowledge triples, the multi-stage knowledge reasoning module deeply integrates them with historical messages, including previous query results and detection records, to carefully construct a structured prompt, aiming to provide comprehensive and rich context information for the large language model, which is ChatGLM.

[0077] Next, the large language model conducts in-depth reasoning analysis based on the prompts and generates corresponding phased answers. These answers are not only direct responses to the current query, but are also recorded and fed back into the "memory" system as important historical information in the subsequent reasoning process.

[0078] In this way, the multi-stage knowledge reasoning module can gradually accumulate and utilize previous reasoning results, optimize the subsequent question-answering process, and ensure that the final output of bolt missing pin detection results is both accurate and explainable.

[0079] In order to further verify the superiority of the proposed method in the detection performance of missing pin bolts, it is compared with some existing missing pin bolt detection methods;

[0080] Since the existing bolt missing pin detection method does not use image and text pair data sets for relevant training, it is necessary to compare with other bolt missing pin detection methods on different bolt data sets, that is, based on the bolt-fitting image and text pair data set for detection, and other methods are based on the fitting-bolt image data set for detection. The accuracy comparison results are shown in the following table:

[0081]

[0082] It can be concluded from the above table that the present invention performs excellently in the detection method of missing pin bolts, with the mAP value as high as 88.7%, which is significantly better than other detection methods. This shows the effectiveness of the proposed method. This is because the present invention not only fully learns the cross-media data of graphics and text, but also reasonably uses the relationship between the bolt structure and its background hardware and spatial orientation as the guiding knowledge for knowledge reasoning.

[0083] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A bolt missing pin detection method based on cross-media and knowledge reasoning, characterized by: The steps include: Step 1: Build a bolt-fitting image-text pair dataset and a bolt knowledge graph; Step 2: Create a cross-media bolt knowledge fusion model. The cross-media bolt knowledge fusion model generates an overall text description of the image by fusing the global information extracted by the fine-tuned MiniGPT-4 and the local information captured by YOLOv8. Step three, create a big language reasoning model based on the knowledge graph. The big language reasoning model based on the knowledge graph uses the bolt knowledge graph to guide the big language model to perform knowledge reasoning on the output information of the cross-media bolt knowledge fusion model to obtain explainable bolt missing pin detection results.

2. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 1 is characterized in that: In the step 1, in the process of constructing the bolt-fitting image-text pair dataset, specifically in the transmission line, the bolt state is closely related to the background fittings and spatial orientation thereof; Specifically, the following situations are included: the bolts with anti-vibration hammer as background hardware do not have missing pins; the bolts with bag-type suspension wire clamp as background hardware are bolts without pins if the space orientation is vertical, bolts with pins if the space orientation is horizontal; the bolts with heavy hammer and connecting plate as background hardware are bolts with pins if the space orientation is horizontal; Collect image data and text data of the above four types of hardware to build the hardware-bolt test dataset FBTD, the hardware-bolt image-text pair fine-tuning dataset BIFD and the bolt knowledge graph BoltKG; The hardware-bolt test dataset, hardware-bolt image-text pair fine-tuning dataset, and bolt knowledge graph are used to deal with three different tasks: FBTD is used to evaluate performance in missing pin bolt detection applications; BIFD is used in the fine-tuning process of MiniGPT-4; BoltKG is used in the LLMKG knowledge reasoning task.

3. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 1 is characterized in that: In the step 1, in the process of constructing the bolt knowledge graph, the bolt knowledge graph is constructed based on the association knowledge between the bolt state and the background hardware and spatial orientation. The specific construction process includes the following steps: 1) Collect relevant data, including relevant information and expert database in the power field; 2) Extract knowledge from the collected data, identify and determine the key entities in the knowledge graph; 3) Based on the professional knowledge in the power field, define the logical relationship between entities; 4) Construct the bolt knowledge graph in the form of structured triples.

4. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 1 is characterized in that: In the step 2, when creating a cross-media bolt knowledge fusion model, the cross-media bolt knowledge fusion model is mainly composed of two modules: a global information extraction module and a local detail recognition module. The outputs of the two modules are combined to jointly realize the cross-media processing process from image to text; The global feature representation and preliminary recognition results output by the global information extraction module are directly used as the input of the local detail recognition module to locate the bolt area and guide the subsequent fine detection; The detailed feature description and out-of-stock detection results output by the local detail recognition module are input into the cross-media fusion and reasoning module together with part of the output structure of the global information extraction module to perform deep fusion and reasoning analysis of cross-media knowledge.

5. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 4 is characterized in that: The global information extraction module uses its powerful feature extraction capability to capture the global information of the bolt from the image. This module mainly uses MiniGPT-4 to fine-tune the input bolt-fitting image and text data to build FB-GPT to fuse the image and text information of the bolt-fitting. In the bolt detection process, the input bolt-fitting image and question are first processed by FB-GPT, and then the model generates a global text description of the global feature representation of the detection image; When the background hardware in the global information is a connecting plate, a heavy hammer or a bag-type suspension clamp, the hardware-bolt image enters the local detail recognition module for detection. Since there is no missing pin on the bolt of the shock-absorbing hammer, target detection is not required.

6. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 4 is characterized in that: In the subsequent stage of bolt detection, the local detail recognition module uses the high-performance target detection model YOLOv8 to perform in-depth and fine detection on the input bolt-fitting detection image, accurately identify multiple detail features of the bolt, including the spatial orientation of the bolt and the presence of the pin, use natural language to describe the detection information of YOLOv8, and finally output the bolt information description.

7. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 4 is characterized in that: The cross-media bolt knowledge fusion model combines the global text description output by the global information extraction module and the bolt information description of the local detail recognition module to finally form an overall text description. The cross-media fusion model finally generates a complete state description of the bolt, including its background hardware, bolt state and spatial orientation description information, completing the cross-media processing task from image to text.

8. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 1 is characterized in that: In the step 3, the large language reasoning model based on the knowledge graph completes the knowledge transfer through LangChain, connects ChatGLM with the bolt knowledge graph, and reasonably utilizes the associated knowledge in BoltKG to guide the reasoning process of the large language model. The large language reasoning model based on the knowledge graph includes an input processing module and a multi-stage knowledge reasoning module; The large language reasoning model based on knowledge graph combines the real-time and interpretability of knowledge graph with the reasoning ability of large language model, effectively overcoming the limitations of large model in the field of bolt missing pin detection in terms of knowledge real-time and insufficient coverage, and making the bolt missing pin detection results interpretable. The model forms prompts consisting of questions, related triples and historical messages through knowledge transfer, and performs knowledge reasoning based on these prompts.

9. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 8 is characterized in that: The input processing module combines the overall text description output by the cross-media bolt knowledge fusion model with the question according to the question template to construct the original question, and then breaks down the original question into each stage question. By combining the overall description with the preset question according to the question template, the module can generate one or more specific and structured original questions; The original question is complex and difficult to answer in one go, so the module needs to break it down into multiple sub-questions, which usually have clearer query goals and are easier to process later.

10. The bolt missing pin detection method based on cross-media and knowledge reasoning according to claim 8 is characterized in that: The multi-stage knowledge reasoning module first receives the post-processing stage questions and query requests from the input processing module, and then establishes a connection with the bolt knowledge graph BoltKG through the knowledge transfer mechanism to accurately retrieve knowledge triples related to the question. These triples widely cover the types of bolts, the specific states of with and without pins, the identification features of missing pins, and the key information of the bolt's orientation in space; After retrieving the knowledge triples, the multi-stage knowledge reasoning module deeply integrates them with historical messages, including previous query results and detection records, to carefully construct a structured prompt, aiming to provide comprehensive and rich context information for the large language model; Next, the large language model conducts in-depth reasoning analysis based on the prompts and generates corresponding phased answers. These answers are not only direct responses to the current query, but will also be recorded and fed back to the "memory" system as important historical information in the subsequent reasoning process; In this way, the multi-stage knowledge reasoning module can gradually accumulate and utilize previous reasoning results, optimize the subsequent question-answering process, and ensure that the final output of bolt missing pin detection results is both accurate and explainable.

Images