Equipment inspection method and device

Abstract

The invention discloses an equipment inspection method and device, and relates to the technical field of equipment inspection. The method comprises the following steps: acquiring an image of equipment; performing scene analysis on equipment in the image according to the first multi-modal large model and feature knowledge of different scenes to obtain equipment and scene information; acquiring corresponding inspection related knowledge from an inspection knowledge base according to the equipment and scene information; inputting the equipment and scene information and the inspection related knowledge into a second multi-mode large model to obtain an automatic analysis workflow output by the second multi-mode large model; performing feature extraction on the image based on the automatic analysis workflow to obtain features of the image; and inputting the image, the features of the image and the inspection related knowledge into a third multi-mode large model to obtain an equipment inspection result output by the third multi-mode large model. According to the method, the problems that the modeling workload is large, the false detection rate is increased and a new scene cannot be quickly adapted in the related technology are solved.

Description

Technical Field

[0001] This application relates to the field of equipment inspection technology, and in particular to an equipment inspection method and apparatus. Background Technology

[0002] Currently, equipment inspection in industrial production workshops primarily relies on visual inspection solutions. These solutions acquire images of the equipment using industrial cameras, and then employ image processing algorithms and artificial intelligence technologies to perform visual analysis, detecting surface defects, component locations, and conditions.

[0003] Visual analysis mainly relies on traditional image processing methods and visual small model analysis methods, both of which require separate modeling and analysis for specific equipment. When there are many types of equipment in the workshop and the form of the same type of equipment varies greatly, this method will have problems such as large modeling workload, increased false detection rate, and inability to quickly adapt to new scenarios. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for equipment inspection, so as to solve the problems in related technologies where separate modeling and analysis of specific equipment leads to a large workload for modeling, an increased false detection rate, and the inability to quickly adapt to new scenarios.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions: In a first aspect, embodiments of this application provide a device inspection method, comprising: acquiring an image of a device; performing scene analysis on the device in the image based on a first multimodal large model and feature knowledge of different scenes to obtain device and scene information; obtaining corresponding inspection-related knowledge from an inspection knowledge base based on the device and scene information; inputting the device and scene information and the inspection-related knowledge into a second multimodal large model to obtain an automatic analysis workflow output by the second multimodal large model; performing feature extraction on the image based on the automatic analysis workflow to obtain the features of the image; and inputting the image, the features of the image, and the inspection-related knowledge into a third multimodal large model to obtain the inspection result of the device output by the third multimodal large model.

[0006] Secondly, embodiments of this application provide an equipment inspection device, comprising: a first acquisition module for acquiring an image of a device; an analysis module for performing scene analysis on the device in the image based on a first multimodal large model and feature knowledge of different scenes to obtain device and scene information; a second acquisition module for acquiring corresponding inspection-related knowledge from an inspection knowledge base based on the device and scene information; a first input module for inputting the device and scene information and the inspection-related knowledge into a second multimodal large model to obtain an automatic analysis workflow output by the second multimodal large model; an extraction module for extracting features from the image based on the automatic analysis workflow to obtain features of the image; and a second input module for inputting the image, the features of the image, and the inspection-related knowledge into a third multimodal large model to obtain the inspection result of the device output by the third multimodal large model.

[0007] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: This application employs a multimodal large model to perform scene analysis and feature extraction on equipment images, and combines this model with inspection-related knowledge from an inspection knowledge base to generate equipment inspection results. For image detection scenarios of different workshops and equipment, end-to-end automatic adaptation can be achieved without extensive manual debugging and intervention. Building a multimodal large model once enables rapid adaptation to multiple scenarios, solving the problems of large modeling workload, increased false detection rate, and inability to quickly adapt to new scenarios in related technologies. Attached Figure Description

[0008] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating an equipment inspection method provided in one embodiment of this application; Figure 2 A schematic diagram illustrating the principle of an equipment inspection method provided in one embodiment of this application; Figure 3 A schematic diagram illustrating the presentation of annotation software according to an embodiment of this application; Figure 4 A schematic diagram of a labeling method (frame drawing and segmentation labeling) provided for one embodiment of this application; Figure 5 A schematic diagram illustrating a region description provided for one embodiment of this application; Figure 6 A schematic diagram illustrating an implementation of an inspection scenario system according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an equipment inspection device provided in one embodiment of this application. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, "and / or" in this application indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. It should be noted that all data involved in this application was obtained with the user's authorization.

[0011] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0012] Figure 1 This is a flowchart illustrating an equipment inspection method according to one embodiment of this application. Figure 1 As shown, the equipment inspection method of this application embodiment may specifically include the following steps: S101, acquire an image of the device.

[0013] In this embodiment of the application, the device that performs the equipment inspection method can be the equipment inspection device provided in this embodiment of the application.

[0014] This application embodiment combines the industrial inspection processes of multiple factories to abstract the equipment inspection process, and fits the process of factory inspection technicians judging abnormalities of on-site equipment.

[0015] To acquire images of equipment in industrial production workshops (such as alumina, electrolytic aluminum, and carbon workshops), specific methods include using fixed video capture devices (such as cameras) to obtain video footage of the equipment in the workshop, extracting frames from the video at fixed time intervals to obtain images of the equipment. Alternatively, FFmpeg can be used to parse the equipment video in real time and extract images using dynamic keyframe algorithms (e.g., image extraction triggered by frames of equipment status changes or scene transitions).

[0016] S102, based on the first multimodal large model and the feature knowledge of different scenes, perform scene analysis on the device in the image to obtain device and scene information.

[0017] In this embodiment, feature knowledge of different scenarios is acquired. The feature knowledge of different scenarios and the images of the devices acquired in step S101 are input into a first multi-modal large language model (MLLM). The first multi-modal large language model performs scene analysis on the devices in the images to obtain device and scene information. The device and scene information may specifically include, but is not limited to, at least one of the following: the workshop where the device is located, the location of the device in the workshop, the name of the device, and the process to which the device belongs.

[0018] In the initial construction of feature knowledge for different scenarios, professional technical personnel are required to assist, that is, human participation is needed in the construction and analysis.

[0019] In this application embodiment, scene equipment knowledge, process characteristics of each workshop, and technician knowledge are manually summarized and compiled to form a context for describing process equipment, which assists the multimodal large model in judging the equipment in the scene.

[0020] S103 retrieves relevant inspection knowledge from the inspection knowledge base based on equipment and scenario information.

[0021] In this embodiment of the application, based on the equipment and scenario information obtained in step S102, a search is performed in a pre-built inspection knowledge base to obtain inspection-related knowledge corresponding to the equipment and scenario information. Specifically, the inspection-related knowledge may include, but is not limited to, at least one of the following: a description of the equipment's normal state (i.e., equipment knowledge), relevant parameters of the equipment, the equipment's inspection methods (and inspection procedures), and expert knowledge such as judgment criteria for various abnormal states of the equipment.

[0022] The initial construction of the inspection knowledge base requires the assistance of professional technicians to create descriptions of the normal status of equipment, knowledge of anomaly determination, and process flow. This means that manual participation in the construction and analysis is necessary.

[0023] S104: Input the equipment and scenario information and inspection-related knowledge into the second multimodal large model to obtain the automatic analysis workflow output by the second multimodal large model.

[0024] In this embodiment of the application, the device and scene information obtained in step S102 and the inspection-related knowledge obtained in step S103 are input into the second multimodal large model, and the second multimodal large model constructs the automatic analysis workflow corresponding to the device.

[0025] The anomaly analysis workflow for different devices required in the initial stage of the automated analysis workflow needs the assistance of professional technicians, that is, manual participation in the construction and analysis is required.

[0026] S105, Based on the automatic analysis workflow, feature extraction is performed on the image to obtain the image features.

[0027] In this embodiment of the application, the image is analyzed and features are extracted step by step through the automatic analysis workflow generated in step S104 to obtain the relevant features of the device in the image, and the workflow analysis results are summarized to obtain the feature set of the device in the image.

[0028] Furthermore, before step S105 "extracting features from the image based on the automatic analysis workflow to obtain the features of the image", the equipment inspection method of this application embodiment may also include the following steps: calling tools in the relevant tool library to perform at least one of the following processing on the image: preprocessing, region detection and region size analysis, etc.

[0029] The basic information of the equipment is judged. For example, for the pumps in the alumina raw material workshop, the only detection items are oil leakage, water leakage, and material leakage. Then, the feature items required for judging water leakage, oil leakage, and material leakage in the image are processed accordingly using the tool library. For example, the presence or absence of water, the location of water, the form of water, and the area and size of water stains are used to provide a comprehensive analysis of subsequent equipment anomalies.

[0030] S106, input the image, image features and inspection-related knowledge into the third multimodal large model to obtain the equipment inspection results output by the third multimodal large model.

[0031] In this embodiment, the image of the device obtained in step S101, the features of the image generated in step S105, and the inspection-related knowledge obtained in step S103 are input into a third multimodal large model for comprehensive analysis and judgment. The third multimodal large model outputs the inspection result of the device. The inspection result may include, but is not limited to, at least one of the following information: the device's anomaly determination result (e.g., whether the device is abnormal and the type of anomaly), the basis for the anomaly determination, and the anomaly resolution suggestion.

[0032] Specifically, through the prompt word engineering, the image of the device obtained in step S101, the features of the image generated in step S105, and the inspection-related knowledge obtained in step S103 are integrated to generate prompt words (i.e., judgment instructions) for determining equipment anomalies; the prompt words are then input into the third multimodal large model to obtain the inspection results output by the third multimodal large model.

[0033] The schematic diagram of the embodiment of this application can be seen as follows: Figure 2As shown, the process includes the following steps: acquiring images of workshop equipment; inputting feature knowledge of different scenarios and equipment images into a multimodal inspection model for scenario analysis to obtain equipment and scenario information; retrieving relevant inspection knowledge (such as equipment knowledge, inspection process, and anomaly judgment criteria) from the equipment inspection knowledge base (built based on equipment knowledge, process flow, and anomaly judgment knowledge) based on the equipment and scenario information; inputting inspection-related knowledge and equipment and scenario information into the multimodal inspection model to build an automated analysis workflow; calling tools to process the images (such as preprocessing, detecting the location of water areas, determining the size of water areas, and the relative position of water areas on equipment); performing feature analysis and extraction on the images based on the automated analysis workflow; summarizing the workflow analysis; and using an inspection prompt word engineering library to integrate the images, image features, and inspection-related knowledge to generate judgment instructions, which are then input into the multimodal inspection model to obtain inspection results (such as whether the equipment is abnormal, the type of abnormality, the judgment basis, and improvement suggestions).

[0034] Furthermore, the equipment inspection method in this application embodiment may also include a multimodal large model training (fine-tuning) step: acquiring workshop data, which may specifically include, but is not limited to, at least one of the following: equipment images, equipment operation videos, and process text; performing full-scene annotation on the workshop data according to scenario, equipment, and knowledge; training the multimodal large model to be trained based on the fully annotated workshop data to obtain the trained first multimodal large model, second multimodal large model, and third multimodal large model.

[0035] Specifically, this application embodiment, combined with the industrial equipment inspection scenario of an alumina workshop, constructs a vertical-category base model for alumina workshop inspection. By annotating the equipment in the alumina workshop across the entire scene and feeding the full-scene data into a multimodal large-scale model for training and improvement, a multimodal large-scale model for alumina workshop inspection is constructed. This model possesses capabilities for "image feature extraction, text semantic understanding, and video temporal analysis" of the core equipment and scene in the alumina workshop, adapting to the complex environment of the workshop; it can accurately locate equipment and understand its status.

[0036] Data collection dimensions: Full collection of workshop data, including but not limited to equipment images (such as equipment images under different lighting conditions, workstations, and equipment operating states), equipment operation videos (such as the complete inspection process of equipment and the fault triggering process), and process texts (such as inspection specifications, equipment manuals, and anomaly judgment standards).

[0037] Labeling rules: Label according to the association of "scene-equipment-knowledge", for example, label the image as "scene: evaporation workshop evaporation station; equipment: pump, pipe, air duct machine, etc. (equipment name + coordinates); knowledge: equipment status, quantity, equipment oil leakage, material leakage, etc.", to build a multimodal aligned dataset and provide "visual-description-anomaly rule" associated samples for model training.

[0038] Among them, the annotation software can present the form as follows: Figure 3 As shown; the annotation methods (drawing frames and dividing annotations) can be as follows: Figure 4 As shown. The region description can be as follows: Figure 5 As shown.

[0039] This application's embodiments, focusing on the equipment inspection needs of an alumina workshop, construct three core task capabilities within a multimodal large model (corresponding to the first, second, and third multimodal large models mentioned above): 1) Scene Recognition: The model input is "equipment video frame + process text description", and the model output is "scene classification (e.g., ball mill position / rod mill position) + key features (e.g., presence of pumps, pipeline equipment, layout features)". Example of prompt: "Based on the following process requirements, identify the workshop scene corresponding to the video frame: [process text], video frame: [image link], output scene type and features".

[0040] 2) Process linkage: The model inputs "scene recognition result + equipment operation industrial description", and the model outputs "detection workflow trigger instruction (such as calling the leakage identification module)". The Prompt incorporates the logic of "arranging the detection process according to the raw material workshop dilution pump inspection specifications".

[0041] 3) Defect detection: The model input is "defect image + historical failure case", and the model output is "defect type (material leakage / air leakage) + risk impact (related process loss)". The Prompt strengthens "judging the severity of the defect by combining workshop process knowledge".

[0042] Furthermore, to reduce training costs, a model layering approach can be used to fine-tune the multimodal large model. Correspondingly, the above step "training the multimodal large model to be trained based on workshop data labeled with full scene annotations" can specifically include the following steps: freezing the parameters of the multimodal large model to be trained at a preset ratio (e.g., 90%) (preserving the ability to extract general features), training the frozen multimodal large model to be trained based on workshop data labeled with full scene annotations, and only fine-tuning the scene adaptation layer (adding "workshop scene classification head and defect detection head").

[0043] Furthermore, knowledge injection can be used to fine-tune the multimodal large model. Specifically, "manually formulated scenario rules" are transformed into instruction fine-tuning data (e.g., the rule "the welding station needs to identify whether the protective device is in place" is transformed into samples: input "image: welding station; instruction: check the status of the protective device"; output "in place / missing + compliance suggestion"). Low-Rank Adaptation (LoRA) technology is used to efficiently fine-tune the multimodal large model, enabling it to learn "scenario understanding under rule constraints".

[0044] Furthermore, the equipment inspection method in this application embodiment may also include verification and iteration steps of a multimodal large model: 1) Indicator monitoring: Use test sets to verify the scene recognition accuracy (target ≥ 95%) and defect detection recall (target ≥ 90%), and combine confusion matrix analysis to analyze misjudgment cases (such as "misidentifying the assembly station as the debugging station").

[0045] 2) Closed-loop optimization: Connect the above-mentioned "human participation in construction analysis (i.e. professional human verification)" step, feed the results of human correction (such as "misjudgment of scene → correct classification and labeling") back into the fine-tuning dataset, and retrain the adaptation layer after each iteration to continuously improve the accuracy of the model in the workshop scene.

[0046] Furthermore, the step S102 above, "performing scene analysis on the device in the image based on the first multimodal large model and the feature knowledge of different scenes," may specifically include the following steps: obtaining the image's metadata; obtaining the image's visual features and temporal features; converting the scene rule text associated with the image into semantic features; and inputting the metadata, visual features, temporal features, semantic features, and feature knowledge of different scenes into the first multimodal large model for scene analysis.

[0047] Specifically, in the scenario analysis steps: a. Input Processing and Feature Engineering 1) Intelligent video stream capture: Relying on FFmpeg to parse the device's video in real time, the image is captured according to the dynamic keyframe algorithm (such as the image capture triggered by the device status change frame or the scene switching frame), and the metadata of the image is extracted at the same time (such as location ID, timestamp, video encoding information, etc.) to provide "spatiotemporal-visual" related data for subsequent analysis.

[0048] 2) Multimodal feature fusion: The fine-tuned multimodal large model is invoked to extract visual features from the image (which can be converted into a 1024-dimensional vector using the clip visual encoder, including device outline, color, layout information, etc.); temporal features are extracted from the video context (20 frames before and after) using TimeSformer (capturing changes in device operating status, such as frame sequence patterns from still to running); the "manually defined scene rules" text is associated and converted into semantic feature vectors using the text encoder Sentence-BERT, constructing a three-dimensional feature space of "visual-temporal-semantic" to provide multi-dimensional basis for scene recognition.

[0049] b. Automatic scene judgment and closed loop 1) Model Inference and Output: Input the fused features into the fine-tuned multimodal large model, perform the "scene recognition & description" task, and output structured results.

[0050] 2) Scene classification: Based on feature space matching, output classifications such as "ball mill position / lime furnace position" with confidence scores (e.g., 0.92 indicates high confidence).

[0051] 3) Feature description: Extract key semantics of the scene (such as "XX type seed pump exists, and the on-site rate of protective device is 90%), associate them with "manually formulated scene rules", and verify whether they meet the preset features.

[0052] 4) Verification Closed-Loop Mechanism: When the output confidence level is <0.8, or when the feature description conflicts with the "manual rule" (e.g., the model identifies it as a "welding station" but the rule requires "no welding equipment in this area"), a professional manual verification process is triggered. The manually corrected results (e.g., re-labeling the scene, supplementing the rule description) are fed back into the model fine-tuning dataset to iteratively optimize the model, forming a closed loop of "data collection → model inference → manual verification → data iteration" to ensure long-term accuracy in scene judgment.

[0053] Furthermore, the aforementioned "inspection knowledge base" can be constructed through the following steps: acquiring all knowledge of workshop operations; breaking down all knowledge of workshop operations into knowledge fragments at the scenario layer, process layer, and standard layer; storing the knowledge fragments in the inspection knowledge base and establishing an index.

[0054] Specifically, a Retrieval-Augmented Generation (RAG) knowledge base is constructed.

[0055] a. Knowledge structuring: Collect all knowledge about workshop operations (such as inspection procedures, process manuals, and failure cases), and break it down into "scenario-process-standard" knowledge fragments using LangChain, such as: 1) Scenario layer: "Welding station → Related knowledge: High temperature protection specifications, welding slag cleaning process"; 2) Process layer: "Leakage detection process → Step 1: Image processing and noise reduction; Step 2: Contour segmentation and recognition"; Step 3: Defect contour size and contamination area; 3) Standard layer: "Oil leakage judgment criteria → 1. There is obvious dripping or a large amount of oil traces on the ground; 2. The leakage area ≤ 0.3MPa triggers the alarm (related to GB / T XXX)".

[0056] Use the Milvus vector database to store knowledge fragments and create an index by "scene ID + knowledge type" (e.g., "scene_welding - process") to support fast retrieval later.

[0057] b. Retrieval logic design: 1) Based on the "RAG of assignment knowledge → large language model" link, design a knowledge retrieval enhancement (RAG) process; 2) The large language model receives "scene judgment results (such as evaporator pump location) + detection requirements (such as leakage detection)", generates search keywords (such as "evaporator pump leakage detection specifications"); the vector database retrieves the top - 5 knowledge fragments by keywords and sends them back to the multimodal large model; 3) The large language model combines "retrieval knowledge + model-built-in semantics" to generate "work scenario specification requirements / inspection process", solving the problems of "outdated knowledge and insufficient understanding of industry rules" in the large model.

[0058] Furthermore, knowledge base-based process generation and verification can be achieved through the following steps: a. Multimodal large model workflow orchestration: After receiving the RAG retrieval results, the second multimodal large model arranges the workflow according to "scene adaptability," such as: 1) Evaporation station in the evaporation workshop: Prioritize the "oil stain and liquid detection of pumps → liquid position and area detection" process. Related knowledge: "When oil leakage occurs, there is usually a lot of oil stains and oily dirt on the ground. When material leakage occurs, there is usually material on the ground. When the area exceeds 1 / 2 of the pump area, it should be judged as serious. When water or material overflows or sprays out, it should be judged as serious."

[0059] 2) Welding station: The “Slag spatter detection → Protective device compliance check” process is triggered first, and the related knowledge is “Welding station needs to be protected against arc light damage, and protective equipment is verified first”.

[0060] The output process includes: step sequence, execution standards (e.g., leakage detection must reach 99% accuracy), and abnormal triggering conditions (e.g., air pressure value exceeds ±10% to trigger an alarm).

[0061] b. Closed-loop verification by professional personnel: The verification dimensions are as follows: 1) Process compliance: Whether it conforms to the workshop standard operating procedure (SOP), such as "whether the leak detection is performed after the parts are assembled".

[0062] 2) Knowledge accuracy: Whether it is related to the latest process standards (such as the newly revised clauses of GB / T 2023).

[0063] 3) Executability: Whether the detection process can be called by downstream modules (such as detection point workflow).

[0064] Based on the results of manual corrections, the RAG knowledge base is updated synchronously (such as adding "workshop protection specifications"), and the instructions (i.e. prompts) of the multimodal large model are fine-tuned to ensure that the work process is consistent with reality in the long term.

[0065] Furthermore, the aforementioned "automatic analysis workflow" can be constructed through the following steps: 1) Dynamic workflow orchestration engine: The second multimodal large model receives "scene judgment results + RAG retrieval knowledge (such as the list of pump inspection items)" and calls Apache Airflow (workflow orchestration tool) to automatically generate inspection workflow.

[0066] 2) Node decomposition: Decompose the detection task according to knowledge fragments, such as "image processing → segmentation detection and recognition → raw material feature detection → water stain feature detection". Each node is bound to a capability library model / agent (e.g., leakage detection calls the target detection model such as YOLOv8 to detect the presence and size of raw materials, and liquid leakage detection calls the visual language model to detect the liquid in the image and quantify the size and region accordingly).

[0067] 3) Dependency configuration: Define node execution dependencies, such as "leaking material recognition is triggered only after water stain segmentation detection and recognition is completed", to ensure the correctness of the process logic.

[0068] 4) Scene adaptation optimization: Combined with process Figure 2 The "Scene Detection Process and Judgment Criteria" section defines process parameters for each scene.

[0069] Furthermore, the aforementioned end-to-end adaptive detection of inspection scene images can obtain the "inspection results" through the collaborative execution of multiple modules: 1) Image processing module: Uses OpenCV to perform denoising and enhancement (e.g., for welding station images, uses the CLAHE algorithm to enhance welding slag details), and outputs standardized images for downstream models to use.

[0070] 2) Segmentation, detection and recognition module: Call the Mask R-CNN model to segment equipment parts in the processed image, identify abnormal areas (such as missing outlines of parts caused by material leakage), and output "defect coordinates + type probability".

[0071] 3) Material / Gas Leakage Detection Module: The material leakage module combines semantic segmentation results (to determine missing material areas), while the gas leakage module integrates sensor data (air pressure, flow rate) and image analysis (pipe deformation, interface gaps) to output "abnormality type + quantitative indicators (such as leakage area 20cm², gas leakage rate 0.5L / s)".

[0072] 4) Results fusion and output: The multimodal large model receives the outputs of each module and performs fusion analysis according to "scenario knowledge + process rules".

[0073] 5) Abnormal correlation: For example, the leakage area is correlated with the "mother liquor tank process in the evaporation workshop" to determine the impact on the downstream process.

[0074] 6) Risk rating: Combine the "defect loss matrix" in the process text (e.g., material leakage may lead to 1 hour of downtime for maintenance, corresponding to a high risk level), output "abnormality level (high / medium / low) + impact range (list of associated equipment IDs)" to provide a basis for subsequent decision-making.

[0075] The system constructed in this application embodiment forms a complete closed loop from "multimodal large model construction" to "end-to-end adaptive detection," closely aligning with the process. Figure 2 The logic here ensures both technological innovation (multimodal fusion, RAG knowledge enhancement) and addresses the strong demands of industrial scenarios (accuracy, compliance) through a "manual verification closed loop," highlighting the innovative aspects of "multimodal adaptation to industrial scenarios, automated process orchestration, and knowledge closed-loop iteration." The implementation diagram of the inspection scenario system in this application embodiment is shown below. Figure 6 As shown.

[0076] In summary, the equipment inspection method of this application employs a multimodal large model to perform scene analysis and feature extraction on equipment images, and combines the multimodal large model with inspection-related knowledge in the inspection knowledge base to generate equipment inspection results. For image detection scenarios of different workshops and different equipment, end-to-end automatic adaptation can be achieved without extensive manual debugging intervention. Constructing a multimodal large model once enables rapid adaptation to multiple scenarios, solving the problems of large modeling workload, increased false detection rate, and inability to quickly adapt to new scenarios in related technologies. By layering the model and freezing some parameters for model fine-tuning, the training cost of the model is reduced. Multimodal feature fusion provides multi-dimensional evidence for scene analysis, making the scene analysis more accurate. By decomposing all knowledge of workshop operations into knowledge fragments and establishing an index, subsequent rapid and accurate retrieval is facilitated.

[0077] This application also proposes an equipment inspection device. For example... Figure 7 As shown, the equipment inspection device 700 of this application embodiment may specifically include: a first acquisition module 701, an analysis module 702, a second acquisition module 703, a first input module 704, an extraction module 705, and a second input module 706. Wherein: The first acquisition module 701 is used to acquire images from the device.

[0078] Analysis module 702 is used to perform scene analysis on the device in the image based on the first multimodal large model and the feature knowledge of different scenes, and obtain device and scene information.

[0079] The second acquisition module 703 is used to acquire corresponding inspection-related knowledge from the inspection knowledge base based on equipment and scenario information.

[0080] The first input module 704 is used to input equipment and scene information and inspection-related knowledge into the second multimodal large model to obtain the automatic analysis workflow output by the second multimodal large model.

[0081] The extraction module 705 is used to extract features from an image based on an automatic analysis workflow to obtain the image features.

[0082] The second input module 706 is used to input the image, the image features and inspection-related knowledge into the third multimodal large model to obtain the inspection results of the device output by the third multimodal large model.

[0083] In this embodiment of the application, the specific process by which each module in the equipment inspection device implements its function can be found in the relevant descriptions in the above-described equipment inspection method embodiments, and will not be repeated here.

[0084] In summary, the equipment inspection device of this application employs a multimodal large model to perform scene analysis and feature extraction on equipment images, and uses the multimodal large model combined with inspection-related knowledge in the inspection knowledge base to generate equipment inspection results. For image detection scenarios of different workshops and different equipment, end-to-end automatic adaptation can be achieved without extensive manual debugging intervention. Building a multimodal large model once enables rapid adaptation to multiple scenarios, solving the problems of large modeling workload, increased false detection rate, and inability to quickly adapt to new scenarios in related technologies. By layering the model and freezing some parameters for model fine-tuning, the training cost of the model is reduced. Multimodal feature fusion provides multi-dimensional evidence for scene analysis, making the scene analysis more accurate. By decomposing all knowledge of workshop operations into knowledge fragments and establishing an index, subsequent rapid and accurate retrieval is facilitated.

[0085] This application also proposes a readable storage medium storing one or more computer programs, the one or more computer programs including instructions, which, when executed by a processor, enable the processor to perform various processes of any of the above-described device inspection method embodiments.

[0086] The readable storage medium of this application embodiment employs a multimodal large model to perform scene analysis and feature extraction on equipment images, and uses the multimodal large model combined with inspection-related knowledge from an inspection knowledge base to generate equipment inspection results. For image detection scenarios of different workshops and different equipment, end-to-end automatic adaptation can be achieved without extensive manual debugging intervention. Building a multimodal large model once enables rapid adaptation to multiple scenarios, solving the problems of large modeling workload, increased false detection rate, and inability to quickly adapt to new scenarios in related technologies. By layering the model and freezing some parameters for model fine-tuning, the training cost of the model is reduced. Through multimodal feature fusion, multi-dimensional evidence is provided for scene analysis, making the scene analysis more accurate. By decomposing all knowledge of workshop operations into knowledge fragments and establishing an index, subsequent fast and accurate retrieval is facilitated.

[0087] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0088] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device inspection device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device inspection device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0094] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0095] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0097] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0098] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0099] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for equipment inspection, characterized in that, include: Acquire images of the device; Based on the first multimodal large model and the feature knowledge of different scenes, the device in the image is analyzed to obtain device and scene information; Based on the equipment and scenario information, retrieve the corresponding inspection-related knowledge from the inspection knowledge base; The equipment and scene information and the inspection-related knowledge are input into the second multimodal large model to obtain the automatic analysis workflow output by the second multimodal large model; Based on the automated analysis workflow, feature extraction is performed on the image to obtain the image's features; The image, its features, and the inspection-related knowledge are input into the third multimodal large model to obtain the inspection results of the device output by the third multimodal large model.

2. The method according to claim 1, characterized in that, The image acquired by the acquisition device includes: The video of the device is acquired using a video capture device; The video is extracted by frame-by-frame extraction at fixed time intervals to obtain the image.

3. The method according to claim 1, characterized in that, The equipment and scenario information includes at least one of the following: the workshop where the equipment is located, the location of the equipment in the workshop, the name of the equipment, and the process to which the equipment belongs; And / or, The inspection-related knowledge includes at least one of the following: a description of the normal state of the equipment, relevant parameters of the equipment, inspection methods of the equipment, and judgment criteria for each abnormal state of the equipment; and / or, The inspection results include at least one of the following: the abnormality determination result of the equipment, the basis for the abnormality determination, and the abnormality resolution suggestions.

4. The method according to claim 1, characterized in that, Before extracting features from the image based on the automated analysis workflow to obtain the features of the image, the process further includes: The tool is invoked to perform at least one of the following processes on the image: preprocessing, region detection, and region size analysis.

5. The method according to claim 1, characterized in that, The step of inputting the image, the image features, and the inspection-related knowledge into a third multimodal large model to obtain the inspection results of the device output by the third multimodal large model includes: Based on the image, the image's features, and the inspection-related knowledge, a prompt word for determining the equipment anomaly is generated; The prompt word is input into the third multimodal large model to obtain the inspection result output by the third multimodal large model.

6. The method according to claim 1, characterized in that, Also includes: Acquire workshop data, which includes at least one of the following: equipment images, equipment operation videos, and process text; The workshop data is labeled with full-scene annotations according to scenario, equipment, and knowledge; The multimodal large model to be trained is trained based on the workshop data labeled in the whole scene, to obtain the first multimodal large model, the second multimodal large model and the third multimodal large model after training.

7. The method according to claim 6, characterized in that, The training of the multimodal large model to be trained based on the workshop data labeled in the entire scene includes: The parameters of the multimodal large model to be trained are frozen at a preset ratio, and the frozen multimodal large model to be trained is trained based on the workshop data labeled in the full scene.

8. The method according to claim 2, characterized in that, The step of performing scene analysis on the device in the image based on the first multimodal large model and feature knowledge of different scenes includes: Obtain the metadata of the image; Obtain the visual and temporal features of the image; The scene rule text associated with the image is converted into semantic features; The metadata, visual features, temporal features, semantic features, and feature knowledge of different scenarios are input into the first multimodal large model for scene analysis.

9. The method according to claim 1, characterized in that, Also includes: Acquire comprehensive knowledge of workshop operations; The complete knowledge of workshop operations is broken down into knowledge fragments at the scenario layer, process layer, and standard layer. The knowledge fragments are stored in the inspection knowledge base and an index is created.

10. An equipment inspection device, characterized in that, include: The first acquisition module is used to acquire images from the device; The analysis module is used to perform scene analysis on the device in the image based on the first multimodal large model and the feature knowledge of different scenes, and to obtain device and scene information; The second acquisition module is used to acquire corresponding inspection-related knowledge from the inspection knowledge base based on the equipment and scenario information. The first input module is used to input the equipment and scene information and the inspection-related knowledge into the second multimodal large model to obtain the automatic analysis workflow output by the second multimodal large model; The extraction module is used to extract features from the image based on the automatic analysis workflow to obtain the features of the image; The second input module is used to input the image, the features of the image, and the inspection-related knowledge into the third multimodal large model to obtain the inspection results of the device output by the third multimodal large model.

Images