Installation specification detection method and device, electronic equipment and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for installation compliance inspection, belonging to the field of artificial intelligence technology. The method includes: retrieving regulatory clauses related to the inspection instructions; using a language model to decompose the regulatory clauses into at least one visual inspection sub-task for inspecting the installation image to obtain a structured task plan; using a visual model to execute the visual inspection sub-task on the installation image to obtain a structured visual inspection result; and inputting the structured visual inspection result and the regulatory clauses into the language model to obtain a judgment result on whether the product installation is compliant. The cognitive framework constructed by this invention enables the language model to understand the regulatory clauses in the knowledge base in real time and dynamically transform them into an executable inspection plan for the visual model. This collaborative model of knowledge-driven planning and visual execution solves the problems of traditional solutions relying on hard-coded logic and having poor scalability. It can quickly respond to changes in business specifications at extremely low cost, combining high accuracy and high agility.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an installation standardization detection method, device, electronic device, and storage medium. Background Technology

[0002] In numerous fields such as smart homes, industrial manufacturing, and equipment maintenance, the quality of the installation process after product or equipment delivery directly affects the safety, stability, and user experience of subsequent use. Therefore, conducting standardized testing based on on-site installation photos to verify whether the installation operation meets the established installation requirements is a crucial quality control step and constitutes a common technical need in the industry today.

[0003] Existing technologies have proposed various automated detection schemes based on image recognition. Among them, the mainstream approach is to use dedicated, small-scale computer vision models to analyze installation images. For example, for the detection item of "non-compliant grounding wire color," a YOLO (You Only Look Once) target detection model can be trained to locate the grounding wire in the image, and then a separate image classification model can be used to determine whether its color is "yellow-green". This approach usually requires training one or more dedicated models for each specific detection component or specification. By combining and connecting these scattered models, a complete detection process is formed, and the judgment logic of this process is usually hard-coded into the program code of the business system.

[0004] The aforementioned existing technical solutions have significant drawbacks in practical applications: Firstly, because the detection logic is embedded in the program code and model structure, whenever a new detection item needs to be added, or an existing standard needs to be adjusted (for example, the refrigerant pipe wrapping standard for a certain model of air conditioner changes), it is often necessary to redevelop, train, and deploy the corresponding dedicated model and modify the business code. The entire process is time-consuming and costly, making it difficult to quickly respond to dynamic changes in business. Secondly, the system lacks robustness and generalization ability. Based on the small model combination scheme, each model can only handle the specific task it was trained on, lacking a comprehensive understanding of the complex and ever-changing installation site environment. When encountering non-ideal situations such as poor shooting angle, excessively strong or dim local lighting, or partial obstruction of the target object, it is easy for a single model to fail to recognize the target, thereby causing the interruption or misjudgment of the entire detection chain. Summary of the Invention

[0005] This invention provides an installation compliance testing method, apparatus, electronic device, and storage medium to solve the problems of poor scalability, difficulty in flexibly adapting to dynamic changes in specification clauses, and insufficient robustness in the face of complex and ever-changing actual installation scenarios in the prior art.

[0006] This invention provides a method for testing installation compliance, comprising the following steps:

[0007] Receive installation images and inspection instructions, wherein the installation images are taken in the product installation scenario and the inspection instructions are instructions to perform standardized inspections on the product installation;

[0008] Retrieve at least one specification clause related to the testing instruction from the preset installation specification knowledge base;

[0009] The specification clauses are decomposed into at least one visual detection subtask for detecting the installation image using a language model, and a structured task plan containing all the visual detection subtasks is generated.

[0010] The visual detection subtask is performed on the installation image using a visual model to obtain structured visual detection results, which include visual feature information corresponding to each visual detection subtask.

[0011] The structured visual inspection results and the standard clauses are input into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model.

[0012] According to the present invention, an installation compliance inspection method is provided, wherein the method utilizes a language model to decompose the compliance clauses into at least one visual inspection subtask for detecting the installation image, and generates a structured task plan containing all the visual inspection subtasks, comprising:

[0013] Construct a structured prompt message that includes the aforementioned specification clauses and the aforementioned testing instructions;

[0014] The structured prompt information is input into the language model to obtain the structured task plan output by the language model.

[0015] According to the installation compliance detection method provided by the present invention, the structured prompt information includes role definition information, background knowledge information, task instruction information, and thought chain guidance information. Constructing a structured prompt information containing the compliance clauses and the detection instructions includes:

[0016] The role definition information is determined based on the preset system expert identity configuration;

[0017] Based on the aforementioned normative clauses, the background knowledge information is determined;

[0018] Based on the detection instructions and the preset task decomposition requirements, the task instruction information is determined;

[0019] Based on a preset logical reasoning paradigm, the thought chain guidance information is determined, and the logical reasoning paradigm is used to demonstrate the reasoning process of converting the detection instruction into specific operation steps.

[0020] According to an installation standardization testing method provided by the present invention, the structured task planning includes a task list;

[0021] Each item in the task list records the task identifier and task description of one of the visual detection subtasks.

[0022] According to the present invention, an installation conformity inspection method is provided, wherein performing the visual inspection subtask on the installation image using a visual model to obtain structured visual inspection results includes:

[0023] Traverse the task list and, for each visual detection subtask, construct a visual query instruction containing the task description;

[0024] The installation image and the visual query command are input into the visual model to obtain the single detection result for each visual detection subtask output by the visual model.

[0025] All individual detection results are aggregated to generate the structured visual detection result.

[0026] According to the installation standardization detection method provided by the present invention, the single detection result includes the position coordinate information of the detection target located by the visual model in the installation image, and the attribute state information of the detection target identified;

[0027] The structured visual inspection result is a chain of evidence data formed by the aggregation of all the individual inspection results.

[0028] According to the installation conformity detection method provided by the present invention, the visual model is trained based on the following steps:

[0029] Obtain a general visual language model as the base model;

[0030] A sample dataset for the field of home appliance installation is constructed, which contains multiple sample images with annotations of installation components and descriptions of installation defects.

[0031] Keep the pre-trained parameters of the base model frozen, and introduce trainable adapter parameters into the base model;

[0032] The adapter parameters are trained and updated using the sample dataset, and the base model containing the trained adapter parameters is determined as the visual model.

[0033] According to the installation compliance inspection method provided by the present invention, the step of inputting the structured visual inspection results and the compliance clauses into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model, includes:

[0034] Determine whether there are any outliers in the structured visual detection results. The outliers include detection results with a confidence level lower than a preset threshold or a status marked as uncertain.

[0035] If no such anomaly exists, the language model obtains the logical matching results between the structured visual detection results and the specification clauses, and directly outputs a qualified or unqualified judgment result.

[0036] If the aforementioned anomaly exists, a review mechanism for the anomaly is executed.

[0037] According to an installation conformity detection method provided by the present invention, the execution of a re-judgment mechanism for the anomaly includes:

[0038] The language model is used to generate a reassessment instruction for the target region corresponding to the anomaly, and the reassessment instruction includes image enhancement requirements or detail inspection requirements for the target region.

[0039] The re-judgment instruction is input into the visual model again to perform secondary detection on the local image in the installation image corresponding to the target area, and obtain the re-judgment detection result;

[0040] The language model updates the structured visual detection result based on the re-judgment detection result and regenerates the judgment result.

[0041] According to the installation compliance testing method provided by the present invention, the installation compliance knowledge base is created based on the following steps:

[0042] Obtain the original document containing the product installation specifications;

[0043] The original document is subjected to text extraction to divide the obtained text content into multiple independent semantic knowledge blocks;

[0044] Each semantic knowledge block is vectorized using a text embedding model to obtain a semantic vector corresponding to each semantic knowledge block.

[0045] The text content of each semantic knowledge block and its corresponding semantic vector are associated and stored in a vector database to construct the installation specification knowledge base.

[0046] According to an installation compliance testing method provided by the present invention, retrieving at least one specification clause related to the testing instruction from a preset installation specification knowledge base includes:

[0047] The detection command is vectorized using the text embedding model to obtain a query vector;

[0048] Calculate the similarity between the query vector and each semantic vector in the installation specification knowledge base;

[0049] Based on the similarity scores from highest to lowest, at least one semantic knowledge block that is most semantically relevant to the detection instruction is selected from the vector database and used as the specification clause.

[0050] The present invention also provides an installation standardization testing device, comprising:

[0051] An information input unit is used to receive installation images and inspection instructions. The installation images are taken in the product installation scenario, and the inspection instructions are instructions to perform standardized inspections on the product installation.

[0052] The specification retrieval unit is used to retrieve at least one specification clause related to the detection instruction from a preset installation specification knowledge base;

[0053] The task planning unit is used to decompose the specification clauses into at least one visual detection subtask for detecting the installation image using a language model, and generate a structured task plan containing all the visual detection subtasks.

[0054] A visual inspection unit is used to perform the visual inspection sub-task on the installation image using a visual model to obtain a structured visual inspection result, wherein the structured visual inspection result contains visual feature information corresponding to each visual inspection sub-task.

[0055] The result determination unit is used to input the structured visual inspection results and the standard clauses into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model.

[0056] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the installation conformity detection method as described above.

[0057] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the installation conformity detection method as described above.

[0058] The installation standardization detection method, device, electronic equipment, and storage medium provided by this invention, through the constructed cognitive framework, enable the language model to understand the standard clauses in the knowledge base in real time and dynamically transform them into an executable detection plan for the visual model. This collaborative mode of knowledge-driven planning and visual execution solves the problems of traditional solutions relying on hard-coded logic and poor scalability. It can quickly respond to changes in business specifications at extremely low cost, and has both high accuracy and high agility. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is one of the flowcharts illustrating the installation compliance testing method provided by the present invention.

[0061] Figure 2 This is a schematic diagram of the process for obtaining structured visual inspection results provided by the present invention.

[0062] Figure 3 This is a schematic diagram of the visual model fine-tuning training process provided by the present invention.

[0063] Figure 4 This is a schematic diagram of the process for determining whether the product installation is compliant, provided by the present invention.

[0064] Figure 5 This is a flowchart illustrating the execution mechanism for reviewing abnormal items provided by the present invention.

[0065] Figure 6 This is the second flowchart of the installation standardization testing method provided by the present invention.

[0066] Figure 7 This is a schematic diagram illustrating the construction process of the installation specification knowledge base provided by this invention.

[0067] Figure 8 This is a flowchart illustrating the search specification clauses provided by the present invention.

[0068] Figure 9 This is a schematic diagram of the installation standardization testing device provided by the present invention.

[0069] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0071] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0072] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0073] During the after-sales installation of home appliances, to ensure installation compliance, engineers need to take on-site photos and upload them to a server for compliance review. Currently, the industry commonly uses technical solutions based on specific target detection models such as YOLO. However, this approach has the following inherent drawbacks:

[0074] 1) Model fragmentation: Each detection scenario (such as "fixed power cord", "grounding wire connection", "compliant installation location" etc.) requires training and deployment of an independent dedicated small model, resulting in a large number of models and forming "model islands" respectively. This leads to complex system integration and extremely high maintenance costs.

[0075] 2) Poor generalization and scalability: A single model can only handle preset scenarios and cannot achieve universal understanding and detection across scenarios. When new home appliance models or new detection items appear, data must be collected again, labeled, and a completely new model must be trained, with a development cycle of up to several weeks, making it impossible to quickly respond to business changes.

[0076] 3) Low technical ceiling: Small models have limited feature extraction and understanding capabilities, making it difficult to handle complex scenes (such as occlusion, poor lighting, and changing angles) and detection tasks that require the integration of multimodal information (such as combining images with installation specification text for understanding). This results in low detection accuracy, especially in complex scenes, which is difficult to further improve.

[0077] To overcome the aforementioned shortcomings, this invention proposes a universal installation compliance detection method, device, electronic device, and storage medium based on a large visual language model. Its core lies in abandoning the traditional approach of "one model per scenario" and creatively adopting an integrated detection paradigm of "universal visual language model + logical language model collaboration" to achieve end-to-end universal detection for all scenarios. This solution achieves a technological leap from "image recognition" to "logical cognition," enabling the detection system to understand, reason about, and autonomously judge installation specifications. The following section will combine... Figures 1-10 This invention describes the installation conformity testing method, apparatus, electronic device, and storage medium provided by the present invention.

[0078] It should be noted that the installation compliance testing method provided in this embodiment of the invention can be executed by various computing devices, such as, but not limited to, servers, personal computers (PCs), embedded systems, or dedicated controllers with processing capabilities. In the following embodiments, for ease of description, the processor will be consistently used as the execution entity. The processor may be a general-purpose central processing unit (CPU), a graphics processing unit (GPU), a dedicated artificial intelligence acceleration chip (such as a neural processing unit, NPU), or a combination of these processors.

[0079] Figure 1 This is one of the flowcharts illustrating the installation compliance testing method provided by the present invention, such as... Figure 1 As shown in the figure, an installation conformity testing method provided by an embodiment of the present invention may specifically include the following steps:

[0080] Step 11: Receive the installation image and detection instructions.

[0081] The installation image is a digital image file that carries information about the installation status of the product to be inspected; for example, it can be a file in JPEG, PNG, or other formats. The installation image is typically captured by on-site installers, quality inspectors, or users using electronic devices with shooting capabilities, such as smartphones, tablets, or dedicated cameras, during the product installation process.

[0082] The product installation scenario can be, for example, the installation site of an air conditioner indoor and outdoor unit, a gas water heater, or a smart toilet in a home environment, or the assembly site of equipment components on a production line in an industrial environment. The content of the installation images intuitively reflects the physical installation status of the various components of the product.

[0083] The testing command is the input signal that triggers the standardized testing task. The testing command can take various forms. For example, it can be a command generated after the user clicks the "Start Testing" button or a specific testing item (such as "Grounding Wire Testing") on the application interface; it can also be a text command converted from the user's voice input "Check if the installation is qualified" by the speech recognition module; or it can be a batch processing task command automatically issued by the backend management system according to a preset quality inspection plan. The core function of the testing command is to clarify the objective of this task, namely, to test the standardization of product installation.

[0084] In one alternative embodiment, the processor receives the installation image and detection instructions by means of receiving the installation image and detection instructions from a remote server or mobile terminal device through its network interface, such as a wired network port or a wireless network module; or by means of the processor directly reading the pre-stored installation image and detection instructions from its local memory.

[0085] Step 12: Retrieve at least one specification clause related to the testing instruction from the preset installation specification knowledge base.

[0086] An installation specification knowledge base can be a pre-built electronic knowledge collection that stores a large amount of information related to product installation, including standards, rules, and operation manuals. The organization of this knowledge base can be varied, such as a relational database, a document-based database, or other structured or semi-structured data storage systems that facilitate information retrieval. The content stored in the knowledge base consists of specific specification clauses, each describing a clear installation standard or requirement.

[0087] The process of retrieving relevant specification clauses from the installation specification knowledge base can be a process where the processor searches for the most relevant specification clauses in the installation specification knowledge base based on the detection command received in the previous step. Specific retrieval methods can include: keyword-based retrieval, which extracts core terms from the detection command (such as "grounding wire") and performs a full-text search in the installation specification knowledge base; or, more further, a semantic similarity-based retrieval method, which first understands and represents the semantic content of the detection command, and then searches for the semantically closest specification clauses in the installation specification knowledge base. This method can more accurately match the user's intent, even if the user's command is expressed in a colloquial manner.

[0088] Taking air conditioner installation as an example, if the received detection instruction is "check whether the grounding is safe", the processor will retrieve the relevant specification clauses from the installation specification knowledge base containing the "Air Conditioner Installation Safety Specification", such as: "Specification Clause 1: The grounding wire must use yellow-green double-color cable" and "Specification Clause 2: The grounding wire connection must be firmly crimped using terminal blocks and must not be tangled".

[0089] Step 13: Use a language model to decompose the specification clauses into at least one visual detection subtask for detecting the installation image, and generate a structured task plan that includes all the visual detection subtasks.

[0090] A language model can be an artificial intelligence model trained on massive amounts of text data, possessing the capabilities of natural language understanding, reasoning, and generation. This language model can understand rules described in human language, such as normative clauses, and translate them into specific steps that can be executed by a machine.

[0091] The core principle of this step is that the language model receives the relatively abstract specification clauses retrieved in the previous step, and, in conjunction with the task objective (i.e., the requirement to verify them in the installation image), decomposes the specification clauses into a series of specific, atomic visual detection subtasks that can be verified on a single image through its internal logical reasoning capabilities.

[0092] Subsequently, the language model decomposes all the obtained visual detection subtasks into a computer-readable, ordered, structured task plan. The structured task plan can be a predefined data format, such as JSON, XML, or a list. This structured task plan clearly defines all the work that the visual model needs to perform, forming a clear execution blueprint.

[0093] Continuing with the example above, after receiving "Standard Clause 1" and "Standard Clause 2", the language model will decompose them into the following three visual detection sub-tasks: (1) locate the "grounding wire" in the installation image; (2) determine the color attribute of the located "grounding wire"; and (3) determine the state of the connection point of the located "grounding wire". Subsequently, the language model organizes these three visual detection sub-tasks into a structured task plan.

[0094] Step 14: Perform the visual detection subtask on the installation image using the visual model to obtain structured visual detection results, which include visual feature information corresponding to each visual detection subtask.

[0095] A visual model can be an artificial intelligence model trained on a large amount of image data and specifically designed to process and analyze visual information. In particular, the visual model can perform a variety of visual tasks such as object detection, image segmentation, and attribute classification.

[0096] After receiving the structured task plan generated in the previous step, the visual model will use this plan as a guide to execute each visual detection subtask contained within the received installation image. For each visual detection subtask, the visual model will analyze the installation image and extract corresponding visual feature information. For example, for the "location" task, the target's coordinate information is extracted; for the "color determination" task, the target's color value or color category is extracted; and for the "state determination" task, descriptive information such as the target's shape or texture is extracted.

[0097] After completing all visual detection subtasks, the processor aggregates all extracted visual feature information to obtain a structured visual detection result. The structured visual detection result is a set of raw factual data obtained through objective analysis of the installation image, and it records the visual findings corresponding to each visual detection subtask in a structured form.

[0098] Continuing with the example above, the visual model will perform three visual detection subtasks in sequence:

[0099] First, the visual model locates the grounding wire in the installation image and outputs its position coordinates. Then, the visual model analyzes the pixels in the area containing those coordinates, outputting a yellow-green color. Finally, the visual model examines the connection point areas related to the position coordinates, outputting the connection method as "terminal crimping." These output visual feature information collectively constitute a structured visual inspection result.

[0100] Step 15: Input the structured visual inspection results and the standard clauses into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model.

[0101] Specifically, the processor inputs the structured visual detection results representing objective facts obtained in the above steps, along with the retrieved normative clauses representing evaluation criteria, into the language model.

[0102] The language model then leverages its logical reasoning capabilities by logically comparing and matching the visual feature information provided by the visual model with the requirements of the specification clauses one by one. Generally, if all visual feature information meets the requirements of all relevant specification clauses, it is deemed "qualified"; if any visual feature information contradicts the specification clauses, it is deemed "unqualified".

[0103] Ultimately, the language model will output a clear judgment result that directly indicates whether the product installation is compliant. The judgment result can be a textual conclusion such as "qualified" or "unqualified", or a structured report containing detailed reasons.

[0104] Continuing the example above, suppose the language model receives visual feature information such as "color: yellow-green" and "connection method: terminal crimping," and compares it with "Standard Clause 1: Grounding wire must use yellow-green cable" and "Standard Clause 2: Terminal blocks must be used." Since the visual feature information perfectly matches the requirements of the standard clauses, the language model ultimately outputs "qualified" as the judgment result for this test.

[0105] The installation compliance detection method provided in this invention constructs a collaborative cognitive framework where a language model handles task planning and decision-making, and a visual model handles visual perception and execution. This decouples the quality inspection logic from traditional hard-coded methods and transforms it into a system driven by a flexibly updatable installation compliance knowledge base. When business specifications change, only the knowledge base document needs to be updated, without rewriting code or retraining the model, greatly improving the system's agility and scalability. Simultaneously, the introduction of the language model enables the system to dynamically decompose abstract compliance clauses into specific detection tasks, achieving a leap from simple image recognition to deep logical cognition, thereby significantly improving detection accuracy and robustness in complex installation scenarios.

[0106] Based on the above embodiments, this embodiment of the invention further specifies the implementation method of using a language model to decompose the specification clauses into at least one visual detection subtask for detecting the installation image, and generating a structured task plan containing all the visual detection subtasks, which may include, but is not limited to, the following steps:

[0107] First, construct a structured prompt message that includes the aforementioned specification clauses and the aforementioned testing instructions.

[0108] In this context, structured prompts can be understood as a data structure or text template specifically designed to guide and constrain the output of a language model. Unlike posing an open-ended question to the language model, the structured prompts in this embodiment combine multiple relevant information elements in a preset format, providing the language model with clear context, necessary input information, and explicit instructions for performing specific reasoning tasks, thereby ensuring that it can generate output that conforms to the expected format and content.

[0109] The process of constructing structured prompt information in this embodiment includes: the processor integrating the specification clauses and detection instructions obtained in the above steps according to a predefined template or format to form a complete text or data package that can be directly input into the language model. For example, the processor can use the detection instructions as the task objective, the specification clauses as the background knowledge required to complete the task, and fill both into a text template.

[0110] Continuing with the example of grounding wire detection during air conditioner installation, the processor can construct structured prompt information in the following form:

[0111] "Task objective: [Check if the grounding is safe]"

[0112] "Background specifications: [Specification clause 1: The grounding wire must be a yellow-green bicolor cable; Specification clause 2: The grounding wire connection must be firmly crimped using terminal blocks and must not be twisted]"

[0113] "Instruction: Based on the above task objectives and background specifications, please generate a step-by-step task plan for image detection."

[0114] By constructing the structured prompt information in the above manner, the originally scattered information can be integrated into a logically coherent whole.

[0115] Furthermore, the structured prompt information is input into the language model to obtain the structured task plan output by the language model.

[0116] The processor can input the structured prompts built in the previous step into the language model for processing via Application Programming Interface (API) calls or other communication methods. After receiving the structured prompts, the language model will understand and reason about them as a whole context.

[0117] Because the structured prompts provide clear task objectives and mandatory rules, the language model's reasoning process is effectively guided and focused, enabling it to accurately execute the logical operations of task decomposition rather than making irrelevant associations or generating conversational responses. Ultimately, the language model outputs a structured task plan containing specific visual detection subtasks, following the implicit or explicit formatting requirements in the structured prompts.

[0118] The installation compliance detection method provided in this embodiment of the invention guides the language model to decompose the task by constructing structured prompt information. This transforms an open-ended reasoning task into a controlled generation task with clear inputs and constraints, ensuring that the language model can accurately understand the task intent and always perform logical decomposition around the relevant normative clauses. This significantly improves the accuracy, consistency, and usability of the generated structured task plan.

[0119] Based on the above embodiments, the embodiments of the present invention further specify the composition of structured prompt information and its construction method.

[0120] In a more specific implementation, the structured prompt information may include role definition information, background knowledge information, task instruction information, and thought process guidance information, etc. Accordingly, the step of constructing structured prompt information containing the specification clauses and the detection instructions may specifically include the following steps:

[0121] First, the role definition information is determined based on the preset system expert identity configuration.

[0122] The role definition information is used to set a specific identity or persona for the language model. By clearly informing the language model of its role at the beginning of the structured prompt information, such as "You are a senior home appliance installation quality inspection expert" or "You are a strict on-site safety standard auditor", the response style, focus and use of professional terminology of the language model can be effectively guided, making its subsequent reasoning and content generation closer to the thinking mode of experts in a specific field.

[0123] The system expert identity configuration is a pre-stored identity description text that can be switched according to different detection fields (such as home appliances, industry, construction, etc.). The processor can select and determine the corresponding role definition information based on the field attributes of the current task.

[0124] Secondly, based on the aforementioned normative clauses, the background knowledge information is determined.

[0125] Background knowledge information is one of the core components of structured prompts, and its direct source is the retrieved specification clauses. The processor extracts the content of the specification clauses and explicitly identifies them as background knowledge or reference specifications, thereby determining the background knowledge information. The purpose of this is to provide the language model with the necessary and inviolable factual basis or legal provisions for performing task decomposition, ensuring that all decomposed visual detection subtasks originate from real specification requirements.

[0126] Next, based on the detection instructions and the preset task decomposition requirements, the task instruction information is determined.

[0127] Task instruction information is a direct and explicit action instruction issued to the language model. The processor combines the received detection instructions with the system's preset task decomposition requirements, such as "Please output a list of tasks in JSON format" and "Each task must be an atomic operation that can be verified on a single image," and integrates them into a piece of instructional text. This task instruction information is used to constrain the language model "what to do" and "in what form to complete it."

[0128] Finally, based on a pre-defined logical reasoning paradigm, the guiding information of the thought chain is determined.

[0129] Thought chain guidance can be seen as an advanced prompting engineering technique. A logical reasoning paradigm is a pre-set exemplary text that demonstrates how to think through an abstract instruction step by step and transform it into concrete operational steps. For example, this paradigm could demonstrate the reasoning process of "'Check if the appearance is intact' can be broken down into: 1. Check if the casing has scratches; 2. Check if any parts are missing; 3. Check if the color is uniform."

[0130] The processor incorporates logical reasoning examples into structured prompts to determine the thought process guidance. It's important to emphasize that its role is not to allow the language model to directly copy the example content, but rather to inspire or activate the language model to adopt a similar, step-by-step, logical thinking pattern to handle the current task—that is, to first perform internal logical reasoning and then provide the final structured output.

[0131] Continuing with the examples in the above embodiments, the structured prompt information that the processor ultimately constructs and inputs to the language model may be as follows:

[0132] #Role definition information:

[0133] You are a seasoned home appliance installation quality inspection expert.

[0134] #Background Information:

[0135] Clause 1 of the specification stipulates that the grounding wire must be a yellow-green bicolor cable.

[0136] Clause 2 of the specification stipulates that the grounding wire must be securely connected using a dedicated terminal block and must not be tangled.

[0137] #User requirements:

[0138] Check if the grounding wire is connected correctly.

[0139] #Task instruction information:

[0140] Please think through the steps one by one and output a JSON-formatted mind chain guide, in which each mind task includes a task identifier and a task description.

[0141] #Mind Chain Guiding Information:

[0142] 1. The user wants to check the grounding wire. First, I need to find the "grounding wire" object in the diagram. This is thought task 1.

[0143] 2. The knowledge base states that the grounding wire color must be "yellow-green". Therefore, I need to check the color of the grounding wire I found. This is thinking task 2.

[0144] 3. The knowledge base also states that the connection must be "secure," "use terminals," and "not tangled." Therefore, I need to check the condition of the grounding wire connection point. This is thinking task 3.

[0145] In this embodiment of the invention, by finely dividing the prompt information into at least four parts, such as roles, knowledge, instructions, and thought chain guidance, deep control and guidance of the language model's behavior are achieved. This not only ensures the accuracy of task decomposition, but more importantly, by stimulating the language model's logical reasoning ability through thought chain guidance information, it enables the model to think and plan like a human expert. This makes the decomposed visual detection sub-tasks more reasonable, comprehensive, and executable, further improving the intelligence level and reliability of the entire detection scheme.

[0146] Based on the above embodiments, the present invention further specifies the specific data structure for structured task planning output by the language model.

[0147] In one alternative implementation, the structured task planning may specifically include a task list; and each item in the task list records a task identifier and task description for one of the visual detection subtasks.

[0148] In this embodiment, the task list is an ordered data collection, such as an array in JSON format, or a list or sequence structure in other programming languages. Organizing the structured task plan into the form of the task list aims to provide a clear, ordered execution sequence that can be directly traversed and iterated over by a computer program. The processor can execute the visual detection subtasks contained within the task list one by one, in the order specified in the task list.

[0149] Furthermore, each element (i.e. each item) in the task list corresponds to an independent visual detection subtask, and the item itself is also a structured data unit containing at least two key fields: task identifier and task description.

[0150] The task identifier is an identifier used to uniquely identify each visual detection subtask. It can be an auto-incrementing number (e.g., 1, 2, 3, ...) or a globally unique string, such as a universally unique identifier (UUID).

[0151] The purpose of setting the task identifier is twofold: firstly, to provide a clear index for task execution, making it easier to track and manage the execution status of tasks; secondly, to establish a foundation for dependencies between tasks, such as allowing subsequent tasks to reference the task identifier of previous tasks to obtain their execution results; and thirdly, to accurately associate the visual feature information output by the visual model with the initially planned visual detection subtasks when summarizing the detection results.

[0152] Optionally, the task description is a text instruction that describes a specific visual inspection operation. This text instruction is natural language text generated by the language model after decomposition according to the specification clauses, specifically used to guide the operation of the visual model. The task description is characterized by its explicitness; it describes a single visual action that can be performed on the mounted image.

[0153] Continuing with the example from the above embodiments, after the language model performs task decomposition, the final output of the structured task plan (in JSON format, for example) might look like this:

[0154] "task_list":[{"task_id":1,"description":"Locate 'grounding wire' in the image."},{"task_id":2,"description":"Determine if the color of 'grounding wire' is 'yellow-green'."},{"task_id":3,"description":"Determine if the connection method of 'grounding wire' is 'terminal connection' and the status is 'securely connected'"}].

[0155] As shown above, the entire structured task planning is an object containing a list of tasks named task_list. Each item in the task list clearly records the task identifier (task_id) and the task description (description), providing clear and unambiguous input for subsequent automated processing.

[0156] In this embodiment of the invention, by concretizing structured task planning into a task list containing task identifiers and task descriptions, the planning content output by the language model has excellent machine readability and programmatic processing capabilities. This not only makes the execution flow of visual detection subtasks clear and orderly, but also enables precise traceability of the entire detection process through task identifiers. This ensures that every step from task planning to result verification is clear, controllable, and traceable, thereby enhancing the automation level and system stability of the entire detection method.

[0157] Figure 2 This is a schematic diagram of the process for obtaining structured visual inspection results provided by the present invention, as shown below. Figure 2 As shown, the embodiments of the present invention further specify the implementation method of performing the visual detection subtask on the installation image using a visual model to obtain structured visual detection results, which mainly includes, but is not limited to:

[0158] Step 141: Traverse the task list and, for each visual detection subtask, construct a visual query instruction containing the task description.

[0159] After receiving the structured task plan, the processor will perform a programmatic traversal of the task list it contains, that is, starting from the first item in the task list and visiting each item one by one to the last item.

[0160] During each access, the processor reads the task description recorded in the current item. Since the task description itself is natural language text, in order for the visual model to accurately understand and execute it, the processor converts it into a visual query instruction that conforms to the visual model's input format. For example, the task description can be directly used as text input. Alternatively, when a visual detection subtask depends on the result of a previous visual detection subtask, the processor will integrate the results of the preceding task (such as the located bounding box information) when constructing the visual query instruction, forming a more contextual and focused visual query instruction.

[0161] For example, for a visual inspection subtask task_002, the constructed visual query instruction could be: "Please determine whether the color of the object in the area where the coordinates [x1,y1,x2,y2] determined by the visual inspection subtask task_001 are 'yellow-green bicolor'."

[0162] Step 142: Input the installation image and the visual query instruction into the visual model, and obtain the single detection result for each visual detection subtask output by the visual model.

[0163] For each visual detection subtask in the task list, the processor performs a call to the visual model once. In each call, the processor inputs two core pieces of information into the visual model: the first is the fixed installation image; the second is the visual query instruction constructed in the previous step for the current subtask.

[0164] Upon receiving the installation image and visual query command, the visual model executes corresponding visual analysis operations. For example, when the visual query command is a localization task, the visual model runs its internal object detection algorithm; when the command is an attribute determination task, it runs its classification or recognition algorithm. After execution, the visual model outputs a single-item detection result for the current visual query command. The single-item detection result is the visual model's feedback on a single visual detection subtask, including its specific findings on the installation image.

[0165] Step 143: Summarize all the individual detection results to generate the structured visual detection result.

[0166] The processor waits for all visual detection subtasks in the task list to complete and collects the individual detection results for each subtask. Then, it aggregates and integrates these scattered individual detection results to generate a complete and final structured visual detection result.

[0167] The aggregation method can be to create a dataset (such as a JSON object or dictionary) and associate each individual detection result with its corresponding task ID, thereby forming a comprehensive data report that can fully reflect the execution status of all visual detection subtasks.

[0168] This invention decomposes a macroscopic visual inspection process into multiple independent, cyclically executable execution flows. This not only makes the calling and management of the visual model more procedural and automated, but also ensures that the execution process and results of each visual inspection subtask are clearly recorded by generating individual inspection results. The resulting structured visual inspection results are complete in content and have a clear structure, providing comprehensive and reliable visual evidence for subsequent language models to make accurate logical judgments.

[0169] In a specific embodiment, the individual detection result specifically includes the position coordinate information of the detection target located by the visual model in the installation image, and the attribute state information of the detection target identified; and the structured visual detection result is evidence chain data formed by the aggregation of all the individual detection results.

[0170] Location coordinate information is data used to accurately describe the region where the visual model finds the detected target in the mounted image. Its data form can be diverse. For example, it can be a rectangular bounding box defined by the coordinates of its top left and bottom right corners, such as [x1, y1, x2, y2]. It can also be a polygonal outline composed of a series of points to describe the detected target with irregular shape. Or, in more refined scenes, it can be a pixel-level segmentation mask that accurately marks every pixel contained in the detected target.

[0171] Attribute state information is data used to describe the inherent characteristics or current state of the detected target, and it varies according to the task description of the corresponding visual detection subtask. For example, if the visual detection subtask is to check color, the attribute state information may include "color: yellow-green"; if the visual detection subtask is to determine the connection method, it may include "connection type: terminal connection"; if the visual detection subtask is to check the existence of a certain component, it may include "existence: yes". In some embodiments, the attribute state information may further include a confidence score to represent the degree of confidence the visual model has in its recognition results.

[0172] Furthermore, this embodiment explicitly defines the structured visual inspection results, formed by the aggregation of all individual inspection results, as evidence chain data. This is because the data structure is not a simple accumulation of results, but rather forms a logically traceable chain of evidence. Specifically, each individual inspection result is closely associated with a specific visual inspection subtask in the structured task plan through its corresponding task identifier. When all individual inspection results are aggregated, they constitute a complete set of evidence that can respond to the initial task plan item by item. Through this evidence chain data, it is possible to clearly trace from the final judgment result to the visual feature information on which it is based, to which visual inspection subtask generated that information, and finally back to the specification clause corresponding to that visual inspection subtask.

[0173] This invention, through standardized definition of individual detection results and explicit identification of the final summary results as evidence chain data, significantly enhances the transparency and traceability of the detection process. This allows the language model to obtain clear, specific, and multi-dimensional objective factual evidence when making its final judgment. The evidence chain data format ensures that each judgment conclusion can be supported by corresponding visual evidence, making the entire detection method not only automated but also reliable and auditable, significantly improving the system's reliability.

[0174] Figure 3 This is a schematic diagram of the visual model fine-tuning training process provided by the present invention, as follows: Figure 3 As shown, the visual model is trained based on the following steps:

[0175] Step 31: Obtain a general visual language model as the base model.

[0176] Optionally, the general visual language model used is a large-scale deep learning model pre-trained on massive amounts of general image-text pairs, such as the LLaVA and Qwen-VL series models in the open-source community. From the beginning of training, it has already learned a wide range of visual recognition and language understanding capabilities for various objects, scenes, and concepts worldwide, forming a strong knowledge base.

[0177] One way to obtain a general visual language model is to download its pre-trained model weight file from a public model repository (such as Hugging Face) to your local machine and load it into your computing environment, using it as the base model for subsequent domain-specific training. Choosing to use this general visual language model as the base is to fully utilize its existing generalization capabilities and avoid the enormous data and computational overhead of training a large visual model from scratch.

[0178] Step 32: Construct a sample dataset in the field of home appliance installation. The sample dataset contains multiple sample images with annotations of installation components and descriptions of installation defects.

[0179] To enable the base model to understand the specialized knowledge required for home appliance installation, this embodiment constructs a dedicated sample dataset. This dataset contains multiple sample images, all of which are photographs taken in real-world home appliance installation scenarios.

[0180] More importantly, each sample image is accompanied by detailed annotation information. For example, there are annotations for installation components, which mark the location of key components such as "grounding wire", "copper pipe joint", and "pressure valve" in the image by means of bounding boxes or masks; and there are descriptions of installation defects, which describe the compliance or non-compliance in the image in natural language text, such as "the yellow-green cable in the picture is the correctly installed grounding wire" or "defect: the refrigerant pipe joint is not wrapped with insulation cotton", etc.

[0181] Step 33: Keep the pre-trained parameters of the base model frozen and introduce trainable adapter parameters into the base model.

[0182] A Parameter Efficient Fine Tuning (PEFT) strategy can be employed to fine-tune the training of the visual model. Specifically, after loading the base model, the processor freezes a large number of its pre-trained parameters, meaning these parameters will remain unchanged throughout subsequent training. This preserves the base model's existing general visual understanding capabilities and effectively prevents catastrophic forgetting.

[0183] At the same time, the processor introduces additional small-scale, trainable adapter parameters at specific locations in the network structure of the base model, such as attention layers or feedforward network layers.

[0184] A preferred implementation is to use Low Rank Adaptation (LoRA), which involves inserting two low-rank, trainable adapter parameters in parallel next to the original weight matrix. During training, only these newly added, minimal adapter parameters are updated, while the large number of base model parameters remain frozen.

[0185] Step 34: Use the sample dataset to train and update the adapter parameters, and determine the base model containing the trained adapter parameters as the visual model.

[0186] During training, the processor randomly samples images from the dataset and inputs these images and their corresponding descriptive text into the base model, which incorporates adapter parameters. The model makes predictions based on the input and calculates the loss between the predictions and the true descriptions. Subsequently, the adapter parameters are trained and updated only through backpropagation. Because the number of parameters to be trained is extremely small (typically only a few percent or even less of the total number of parameters in the base model), the computational resource requirements of the entire training process are significantly reduced, resulting in a substantial improvement in training efficiency.

[0187] After training, the adapter parameters have learned and encoded professional knowledge in the field of home appliance installation. At this point, the assembly of the base model containing the trained adapter parameters is determined as the vision model with domain expertise ultimately used to perform the vision inspection task in this embodiment of the invention.

[0188] The embodiments of this invention employ a parameter-efficient fine-tuning method represented by LoRA technology, which achieves efficient injection of professional knowledge in the field of home appliance installation into the model with extremely low training and time costs while retaining the powerful generalization ability of the general visual language model. The resulting visual model can not only recognize objects and environments in general scenarios, but also accurately locate and judge professional components and their states in specific installation scenarios, thus possessing both breadth and depth.

[0189] Figure 4 This is a schematic diagram of the process for determining whether product installation is compliant, as provided by the present invention. Figure 4 As shown, in one embodiment, the structured visual inspection results and the specification clauses are input into the language model to obtain a judgment result on whether the product installation is compliant, output by the language model. This may include, but is not limited to:

[0190] Determine whether there are any outliers in the structured visual detection results. Outliers include detection results with a confidence level lower than a preset threshold or a status marked as uncertain.

[0191] Specifically, before directly using the structured visual inspection results for the final logical judgment, the processor will first perform a preliminary health check on them. For example, it will traverse each individual inspection result contained in the evidence chain data in the aforementioned embodiment of the structured visual inspection results.

[0192] Each individual test result will be checked for any abnormal items. These abnormal items mainly refer to two types of situations:

[0193] The first category is detection results with a confidence score lower than a preset threshold. As described in the previous embodiments, when the visual model outputs the attribute state information it has recognized, it may include a confidence score. The processor compares this confidence score with a preset threshold (e.g., 0.7). If the score is lower than the preset threshold, it means that the visual model lacks confidence in its own judgment, and the result is marked as an outlier.

[0194] The second category is detection results marked as uncertain. In some cases, such as when the target being detected is severely occluded, blurred, glare-prone, or poorly lit in the installation image, the visual model may not be able to make a clear judgment. In this case, it will directly output a "uncertain", "unknown", or similar status mark. Such detection results that clearly indicate that they cannot be judged are also considered outliers.

[0195] The processor uses the above checks to determine whether there is at least one outlier in the structured visual inspection results. If no outlier is found, the language model obtains the logical matching results between the structured visual inspection results and the specification clauses, and directly outputs a pass / fail result.

[0196] Specifically, if the processor confirms that all individual detection results are clear and have high confidence, that is, there are no outliers in the structured visual detection results, it indicates that the visual model has provided clear visual evidence for all visual detection subtasks.

[0197] In this scenario, the standard decision-making process begins. The processor inputs the structured visual inspection results and relevant specification clauses into the language model. The language model then performs its core logical judgment function, comparing the structured visual inspection results with the specification clauses one by one. If all visual evidence conforms to the specification, a satisfactory judgment is output directly; otherwise, if any visual evidence contradicts the specification, an unsatisfactory judgment is output directly.

[0198] If the aforementioned anomaly exists, a review mechanism for the anomaly is executed.

[0199] If the processor detects anomalies in the structured visual inspection results, it indicates that the current visual evidence is incomplete or unreliable. Making judgments directly based on this questionable evidence may lead to erroneous conclusions. To avoid this, instead of immediately determining "qualified" or "unqualified," a re-evaluation mechanism is executed for the anomalies.

[0200] The aforementioned re-judgment mechanism is a special processing procedure designed to resolve uncertainties and obtain more reliable visual evidence. Its specific implementation will be further described in subsequent embodiments.

[0201] Figure 5 This is a flowchart illustrating the execution of the exception review mechanism provided by the present invention, as shown below. Figure 5 As shown, in one specific implementation, the execution of the review mechanism for the anomaly item may specifically include the following steps:

[0202] Step 151: Use the language model to generate a re-judgment instruction for the target region corresponding to the anomaly, the re-judgment instruction including image enhancement requirements or detail inspection requirements for the target region.

[0203] When the system needs to execute the re-judgment mechanism, the processor will input the relevant information of the abnormal item that caused the re-judgment, such as which task's detection result is uncertain, the description of the reason for the uncertainty, and the position coordinate information of the target area corresponding to the task in the installation image, back into the language model.

[0204] At this point, the language model analyzes the causes of the uncertainty and, leveraging its powerful reasoning and generative capabilities, generates a new, more targeted re-judgment instruction. Unlike the initial, broader visual detection subtask, this re-judgment instruction has a highly focused objective: specifically designed to resolve this particular uncertainty problem.

[0205] Optionally, the review instruction typically includes two types of advanced requirements:

[0206] One type is image enhancement requirements. For example, if the cause of the anomaly is that the target area is too dark, the language model may generate a reassessment instruction like this: "Please increase the brightness and contrast of the region at coordinates [x1,y1,x2,y2] in the image by 50%, and then reassess the connectivity of objects in that region."

[0207] Another type is the requirement for detailed inspection. For example, if the cause of the anomaly is that the target is partially obscured, making it impossible to determine whether it is secure, the language model may generate: "Please carefully examine the edge contour of the region at coordinates [x1,y1,x2,y2] in the image to determine whether there are any signs of entanglement rather than terminal connection."

[0208] These review instructions demonstrate the cognitive capabilities of the language model; they no longer simply repeat previous tasks but propose specific strategies for solving problems.

[0209] Step 152: Input the re-judgment instruction into the visual model again to perform secondary detection on the local image in the installation image corresponding to the target area and obtain the re-judgment detection result.

[0210] The processor will input the re-judgment instruction generated in the previous step, along with the original installation image, back into the visual model. Upon receiving the re-judgment instruction, the visual model will perform a highly focused secondary detection.

[0211] The core difference between secondary detection and primary detection lies in their targeting. The visual model strictly follows the requirements of the re-judgment instruction. If the re-judgment instruction includes image enhancement requirements, the visual model or its front-end image processing module will first perform corresponding image processing operations (such as brightening and sharpening) on ​​the local image corresponding to the target region in the installation image, and then analyze the processed local image. Because the target region of secondary detection is extremely small and may have undergone image enhancement processing, details that were previously invisible due to image quality issues become clearly visible, enabling the visual model to make a more accurate judgment. After execution, the visual model outputs a completely new re-judgment detection result.

[0212] Step 153: The language model updates the structured visual detection result based on the re-judgment detection result and regenerates the judgment result.

[0213] The processor submits the acquired re-judgment detection results to the language model, which then uses these results to find the original outlier in the structured vision detection results and updates or replaces it with the new, more reliable re-judgment detection results.

[0214] After updating the structured visual inspection results, the uncertainty in the chain of evidence is eliminated. At this point, the processor triggers the language model to regenerate the judgment result, that is, to repeatedly perform a comprehensive logical comparison between the updated and complete structured visual inspection results and the relevant standard clauses, and finally output the final judgment result as "qualified", "unqualified" or, in very rare cases (such as when it is still impossible to determine after re-judgment) marked as "requires manual review".

[0215] This invention employs a closed-loop "diagnosis-review-decision" mechanism, where a language model guides the planning and a visual model executes the process. This mechanism endows the system with self-correction and in-depth analytical capabilities. When encountering uncertain detection situations, instead of simply abandoning or outputting incorrect results, the system intelligently analyzes the causes of the problem and proactively initiates a more targeted secondary detection. This mimicking of human expert review behavior effectively overcomes the detection bottlenecks caused by objective factors such as image quality and occlusion in a single detection, significantly improving the accuracy of the detection results and the overall robustness of the system.

[0216] Figure 6 This is the second flowchart of the installation standardization testing method provided by the present invention, as shown below. Figure 6As shown, to illustrate the process more clearly, let's take checking whether the "exhaust pipe connection of a gas water heater is secured with screws" as an example for a simple explanation:

[0217] First, the system receives installation images and testing instructions. For example, a picture of the completed water heater installation taken by the installer is uploaded, along with a testing instruction to "check the flue pipe installation".

[0218] The system retrieves the relevant clauses from the installation specification knowledge base based on the command: "At least one screw must be used to tighten each connection of the exhaust pipe to ensure a secure connection."

[0219] The language model generates a structured task plan, which decomposes the clause into visual detection sub-tasks, such as {Task 1: Locate the exhaust pipe connection; Task 2: Detect the presence of screws in the connection area; Task 3: Determine whether the screws are tightened}.

[0220] The system iterates through the task list, constructs visual query instructions, and assigns them to a visual model fine-tuned based on sample data for detection.

[0221] During Task 3, due to poor lighting during filming, the screw at the connection point was in a shadow, making it impossible for the visual model to confirm whether the screw was tightened. Therefore, when generating the structured visual inspection results, the result of Task 3 was marked as an outlier, for example, {"task_id": 3; "status": "uncertain"; "reason": "too dark"}.

[0222] When the system makes a judgment, it finds an anomaly and the process enters the "yes" branch.

[0223] The system initiates a review mechanism and generates a review instruction. At this point, the language model generates a new, targeted instruction based on the cause of the anomaly, "too dim lighting": "Please enhance the brightness and contrast of the target area in the image and recheck whether the screws are tight."

[0224] The system performs secondary detection on a local image. After receiving the re-judgment instruction, the visual model only performs image enhancement processing on the local area where the screw is located in the image, and then performs detection. In the enhanced local image, the visual model clearly identifies that the screw is fully tightened.

[0225] The results of the second inspection updated the original structured visual inspection results, and at this point, no outliers were found. The process then passed through the decision node again and entered the "No" branch.

[0226] Ultimately, the system outputs a qualified conclusion.

[0227] Figure 7This is a schematic diagram of the construction process of the installation specification knowledge base provided by this invention, such as... Figure 7 As shown, in one specific implementation, the installation specification knowledge base is created based on the following steps:

[0228] Step 111: Obtain the original documents containing product installation specifications. Original documents refer to source files that record various product installation standards, operating procedures, safety requirements, technical parameters, and other knowledge. These source files come from a wide range of sources, including official installation manuals published by product manufacturers, standard specification white papers formulated by industry associations, and internal quality inspection standard documents accumulated by enterprises. Their formats are also diverse, such as PDF, Word, Excel, plain text, or web pages. The processor obtains these original documents containing rich professional knowledge through file uploads, web crawling, or database reading.

[0229] Step 112: Extract text from the original document to divide the obtained text content into multiple independent semantic knowledge blocks.

[0230] Because the original documents vary in format and are usually large, continuous blocks of text, direct use for machine retrieval is inefficient and inaccurate. Therefore, this embodiment will preprocess the original documents, including:

[0231] The processor uses appropriate parsing libraries for different document formats, such as PyMuPDF for PDF files and Python docx for Word documents, to extract plain text content. During this process, preliminary data cleaning is typically performed, such as removing headers, footers, page numbers, image titles, and other irrelevant information.

[0232] To improve the accuracy of subsequent searches, the processor does not treat the entire document's text as a single unit. Instead, it employs a chunking strategy, dividing long text contents into multiple independent, smaller-granular semantic knowledge blocks.

[0233] There are several strategies for segmenting semantic knowledge blocks. For example, they can be segmented according to the natural paragraphs and headings of the document; a fixed length threshold can be set, such as segmenting into a knowledge block every 500 characters; a better strategy is to use semantic segmentation, which uses natural language processing technology to segment at the places where semantic transitions or shifts occur while maintaining the integrity of the sentence, so as to ensure that each semantic knowledge block revolves around a relatively independent and complete topic.

[0234] For example, given the following passage in the original document: "Chapter 5: Grounding Specifications. 5.1 Grounding wires must use yellow-green bicolor cables with a diameter of not less than 2.5 square millimeters. 5.2 Grounding wires must be securely connected using dedicated terminals and must not be tangled," the processor can divide it into two semantic knowledge blocks: "Semantic Knowledge Block A: Grounding wires must use yellow-green bicolor cables with a diameter of not less than 2.5 square millimeters," and "Semantic Knowledge Block B: Grounding wires must be securely connected using dedicated terminals and must not be tangled."

[0235] Step 113: Use a text embedding model to vectorize each semantic knowledge block to obtain the semantic vector corresponding to each semantic knowledge block.

[0236] Furthermore, to enable computers to understand and compare the semantic similarity between different text fragments, semantic knowledge blocks in text form can be converted into mathematical vectors. The processor utilizes a pre-trained text embedding model, such as the open-source BERT, M3E, and BGE models. By successively inputting each semantic knowledge block obtained in the previous step into the text embedding model, the model performs deep semantic analysis on the input text and outputs a high-dimensional, fixed-length floating-point semantic vector (e.g., 768-dimensional or 1024-dimensional). This semantic vector can be viewed as a coordinate point of the semantic knowledge block in a multi-dimensional semantic space; semantically similar text blocks have corresponding semantic vectors that are closer in space. Through vectorization, each semantic knowledge block is given a unique mathematical representation that represents its core semantics.

[0237] Step 114: Associate and store the text content of each semantic knowledge block and its corresponding semantic vector in a vector database to construct the installation specification knowledge base.

[0238] For each semantic knowledge block, the processor associates and stores its original text content and corresponding semantic vector as a data pair. It can also store metadata, such as the source of the semantic knowledge block (original document name, chapter number, etc.), to facilitate subsequent source tracing.

[0239] The storage medium for these data pairs is a vector database, such as FAISS (Facebook AI Similarity Search), Milvus, or Pinecone. Vector databases are specifically designed for the efficient storage and retrieval of high-dimensional vector data. They establish a special index structure for the stored semantic vectors, enabling millisecond-level approximate nearest neighbor search on billions of data points. Once all processed semantic knowledge blocks and their semantic vectors are stored in the vector database, an efficient, searchable, and natural language-understanding installation specification knowledge base is completed.

[0240] This invention transforms unstructured, human-readable raw documents into a structured, machine-retrievable installation specification knowledge base. This semantic vector-based knowledge base construction method enables subsequent retrieval processes to move beyond rigid keyword matching and achieve intelligent retrieval based on semantic similarity. This significantly improves the accuracy and comprehensiveness of knowledge retrieval, providing a solid data foundation for the entire detection method to accurately understand user intent and find the correct evaluation criteria.

[0241] Figure 8 This is a flowchart illustrating the search specification clauses provided by the present invention, such as... Figure 8 As shown, in one specific embodiment, retrieving at least one specification clause related to the testing instruction from a preset installation specification knowledge base includes, but is not limited to, the following steps:

[0242] Step 121: The detection command is vectorized using the text embedding model to obtain a query vector.

[0243] Upon receiving the detection command, the processor needs to synchronously convert it into a semantic vector in order to perform a search within the semantic space of the installation specification knowledge base. The processor uses a text embedding model to accomplish this. Crucially, the text embedding model used here is exactly the same as the one used to create the installation specification knowledge base in the preceding embodiments, ensuring that the generated vectors reside within the same semantic space.

[0244] The processor inputs the text content of the detection command into the text embedding model. After calculation, the text embedding model outputs a high-dimensional vector with the same dimensions as other semantic vectors in the installation specification knowledge base. This vector is the query vector. The query vector is a mathematical representation of the user intent implied in the detection command.

[0245] Step 122: Calculate the similarity between the query vector and each semantic vector in the installation specification knowledge base.

[0246] The processor uses the query vector obtained in the previous step as the basis for the query and performs a search operation in the installation specification knowledge base. The vector database efficiently calculates the similarity between the query vector and each semantic vector stored therein. The similarity calculation can employ various mathematical methods; a preferred method is cosine similarity. Cosine similarity measures the directional consistency of two vectors by calculating the cosine of the angle between them in multidimensional space. Its value ranges from -1 to 1; the closer the value is to 1, the more semantically similar the two vectors are.

[0247] Step 123: Based on the similarity from largest to smallest, select at least one semantic knowledge block from the vector database that is most semantically relevant to the detection instruction, as a specification clause.

[0248] After calculating the similarity scores between the query vector and all the semantic vectors, the processor sorts all the semantic knowledge blocks in descending order of similarity. Then, the processor selects the top-ranked semantic knowledge blocks from the sorted results (e.g., the top 3 in similarity or those with similarity scores above a preset threshold). These selected semantic knowledge blocks are considered to be the knowledge content most semantically relevant to the detection instruction.

[0249] Finally, the processor extracts the text content corresponding to these selected semantic knowledge blocks and identifies these text contents as the normative clauses that need to be followed in this detection task.

[0250] Continuing with the example of air conditioner installation, when the received detection instruction is "Check if the grounding wire is securely connected," firstly, the text embedding model is used to convert this instruction into a query vector. Then, the cosine similarity between this query vector and all stored semantic vectors is calculated in the installation specification knowledge base. The calculation results may show that the semantic vector corresponding to "The grounding wire must be securely connected using a dedicated terminal block and should not be tangled" has the highest similarity score (e.g., 0.92), followed by the semantic vector corresponding to "The grounding wire must use a yellow-green cable" (e.g., 0.85). Finally, based on the similarity ranking, the system selects these two semantically most relevant knowledge blocks and uses their text content as the specification clauses for subsequent task decomposition.

[0251] Figure 9 This is a structural schematic diagram of the installation standardization testing device provided by the present invention, as shown below. Figure 9 As shown, the present invention also provides an installation standardization testing device, which mainly includes, but is not limited to:

[0252] The information input unit 91 is used to receive installation images and inspection instructions. The installation images are taken in the product installation scenario, and the inspection instructions are instructions to perform standardized inspections on the product installation.

[0253] The specification retrieval unit 92 is used to retrieve at least one specification clause related to the detection instruction from a preset installation specification knowledge base;

[0254] Task planning unit 93 is used to decompose the specification clauses into at least one visual detection subtask for detecting the installation image using a language model, and generate a structured task plan containing all the visual detection subtasks.

[0255] The visual detection unit 94 is used to perform the visual detection sub-task on the installation image using a visual model to obtain a structured visual detection result, wherein the structured visual detection result contains visual feature information corresponding to each visual detection sub-task.

[0256] The result determination unit 95 is used to input the structured visual inspection results and the standard clauses into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model.

[0257] It should be noted that the installation standardization testing device provided by the present invention can execute or implement the installation standardization testing method provided in any of the above embodiments in actual operation, and will not be described in detail here.

[0258] The cognitive framework constructed by the installation standardization detection device provided by this invention enables the language model to understand the standard clauses in the knowledge base in real time and dynamically transform them into an executable detection plan for the visual model. This collaborative mode of knowledge-driven planning and visual execution solves the problems of traditional solutions relying on hard-coded logic and poor scalability. It can respond quickly to changes in business specifications at extremely low cost, and has both high accuracy and high agility.

[0259] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 10As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communications bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communications bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute an installation compliance inspection method. This method includes: receiving an installation image and an inspection instruction, wherein the installation image is taken in a product installation scenario, and the inspection instruction is an instruction to perform compliance inspection on the product installation; retrieving at least one specification clause related to the inspection instruction from a preset installation specification knowledge base; decomposing the specification clause into at least one visual inspection subtask for inspecting the installation image using a language model, generating a structured task plan containing all the visual inspection subtasks; executing the visual inspection subtasks on the installation image using a visual model to obtain a structured visual inspection result, wherein the structured visual inspection result contains visual feature information corresponding to each visual inspection subtask; and inputting the structured visual inspection result and the specification clause into the language model to obtain a judgment result output by the language model indicating whether the product installation is compliant.

[0260] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0261] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the installation compliance detection method provided in the above embodiments, the method including: receiving an installation image and a detection instruction, the installation image being taken in a product installation scenario, the detection instruction being an instruction to perform compliance detection on the product installation; retrieving at least one specification clause related to the detection instruction from a preset installation specification knowledge base; decomposing the specification clause into at least one visual detection subtask for detecting the installation image using a language model, generating a structured task plan containing all the visual detection subtasks; executing the visual detection subtasks on the installation image using a visual model, obtaining a structured visual detection result, the structured visual detection result containing visual feature information corresponding to each visual detection subtask; inputting the structured visual detection result and the specification clause into the language model, obtaining a judgment result on whether the product installation is compliant output by the language model.

[0262] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the installation compliance detection method provided in the above embodiments. The method includes: receiving an installation image and a detection instruction, wherein the installation image is taken in a product installation scenario and the detection instruction is an instruction to perform compliance detection on the product installation; retrieving at least one specification clause related to the detection instruction from a preset installation specification knowledge base; decomposing the specification clause into at least one visual detection subtask for detecting the installation image using a language model, and generating a structured task plan containing all the visual detection subtasks; performing the visual detection subtasks on the installation image using a visual model to obtain a structured visual detection result, wherein the structured visual detection result contains visual feature information corresponding to each visual detection subtask; and inputting the structured visual detection result and the specification clause into the language model to obtain a judgment result output by the language model indicating whether the product installation is compliant.

[0263] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0264] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0265] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for testing installation compliance, characterized in that, include: Receive installation images and inspection instructions, wherein the installation images are taken in the product installation scenario and the inspection instructions are instructions to perform standardized inspections on the product installation; At least one specification clause related to the inspection instruction is retrieved from a pre-built installation specification knowledge base; the installation specification knowledge base is a pre-built electronic knowledge set that stores standards, rules and operation manuals related to product installation. The specification clauses are decomposed into at least one visual detection subtask for detecting the installation image using a language model, and a structured task plan containing all the visual detection subtasks is generated. The visual detection subtask is performed on the installation image using a visual model to obtain structured visual detection results, which include visual feature information corresponding to each visual detection subtask. The structured visual inspection results and the standard clauses are input into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model. The step of using a language model to decompose the specification clauses into at least one visual detection subtask for detecting the installation image, and generating a structured task plan containing all the visual detection subtasks, includes: Construct a structured prompt message that includes the aforementioned specification clauses and the aforementioned testing instructions; The structured prompt information is input into the language model to obtain the structured task plan output by the language model; The step of inputting the structured visual inspection results and the specification clauses into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model, includes: Determine whether there are any outliers in the structured visual detection results. The outliers include detection results with a confidence level lower than a preset threshold or a status marked as uncertain. If no such anomaly exists, the language model obtains the logical matching results between the structured visual detection results and the specification clauses, and directly outputs a qualified or unqualified judgment result. If the aforementioned anomaly exists, a re-evaluation mechanism for the anomaly is executed; The execution of the review mechanism for the anomaly includes: The language model is used to generate a reassessment instruction for the target region corresponding to the anomaly, and the reassessment instruction includes image enhancement requirements or detail inspection requirements for the target region. The re-judgment instruction is input into the visual model again to perform secondary detection on the local image in the installation image corresponding to the target area, and obtain the re-judgment detection result; The language model updates the structured visual detection result based on the re-judgment detection result and regenerates the judgment result.

2. The installation compliance testing method according to claim 1, characterized in that, The structured prompt information includes role definition information, background knowledge information, task instruction information, and thought process guidance information. The construction of a structured prompt information system containing the specification clauses and the detection instructions includes: The role definition information is determined based on the preset system expert identity configuration; Based on the aforementioned normative clauses, the background knowledge information is determined; Based on the detection instructions and the preset task decomposition requirements, the task instruction information is determined; Based on a preset logical reasoning paradigm, the thought chain guidance information is determined, and the logical reasoning paradigm is used to demonstrate the reasoning process of converting the detection instruction into specific operation steps.

3. The installation compliance testing method according to claim 1, characterized in that, The structured task planning includes a task list; Each item in the task list records the task identifier and task description of one of the visual detection subtasks.

4. The installation compliance testing method according to claim 3, characterized in that, The step of performing the visual detection subtask on the installation image using a visual model to obtain structured visual detection results includes: Traverse the task list and, for each visual detection subtask, construct a visual query instruction containing the task description; The installation image and the visual query command are input into the visual model to obtain the single detection result for each visual detection subtask output by the visual model. All individual detection results are aggregated to generate the structured visual detection result.

5. The installation compliance testing method according to claim 4, characterized in that, The single detection result includes the position coordinate information of the detection target located by the visual model in the installation image, and the attribute status information of the detection target identified; The structured visual inspection result is a chain of evidence data formed by the aggregation of all the individual inspection results.

6. The installation compliance testing method according to claim 1, characterized in that, The visual model was trained based on the following steps: Obtain a general visual language model as the base model; A sample dataset for the field of home appliance installation is constructed, which contains multiple sample images with annotations of installation components and descriptions of installation defects. Keep the pre-trained parameters of the base model frozen, and introduce trainable adapter parameters into the base model; The adapter parameters are trained and updated using the sample dataset, and the base model containing the trained adapter parameters is determined as the visual model.

7. The installation compliance testing method according to claim 1, characterized in that, The installation specifications knowledge base was created based on the following steps: Obtain the original document containing the product installation specifications; The original document is subjected to text extraction to divide the obtained text content into multiple independent semantic knowledge blocks; Each semantic knowledge block is vectorized using a text embedding model to obtain a semantic vector corresponding to each semantic knowledge block. The text content of each semantic knowledge block and its corresponding semantic vector are associated and stored in a vector database to construct the installation specification knowledge base.

8. The installation compliance testing method according to claim 7, characterized in that, The step of retrieving at least one specification clause related to the testing instruction from a preset installation specification knowledge base includes: The detection command is vectorized using the text embedding model to obtain a query vector; Calculate the similarity between the query vector and each semantic vector in the installation specification knowledge base; Based on the similarity scores from highest to lowest, at least one semantic knowledge block that is most semantically relevant to the detection instruction is selected from the vector database and used as the specification clause.

9. A standard installation testing device, characterized in that, include: An information input unit is used to receive installation images and inspection instructions. The installation images are taken in the product installation scenario, and the inspection instructions are instructions to perform standardized inspections on the product installation. The specification retrieval unit is used to retrieve at least one specification clause related to the test instruction from a preset installation specification knowledge base; the installation specification knowledge base is a pre-built electronic knowledge set that stores standards, rules and operation manuals related to product installation. The task planning unit is used to decompose the specification clauses into at least one visual detection subtask for detecting the installation image using a language model, and generate a structured task plan containing all the visual detection subtasks. The step of using a language model to decompose the specification clauses into at least one visual detection subtask for detecting the installation image, and generating a structured task plan containing all the visual detection subtasks, includes: Construct a structured prompt message that includes the aforementioned specification clauses and the aforementioned testing instructions; The structured prompt information is input into the language model to obtain the structured task plan output by the language model; A visual inspection unit is used to perform the visual inspection sub-task on the installation image using a visual model to obtain a structured visual inspection result, wherein the structured visual inspection result contains visual feature information corresponding to each visual inspection sub-task. The result determination unit is used to input the structured visual inspection results and the standard clauses into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model. The step of inputting the structured visual inspection results and the specification clauses into the language model to obtain the judgment result of whether the product installation is compliant, output by the language model, includes: Determine whether there are any outliers in the structured visual detection results. The outliers include detection results with a confidence level lower than a preset threshold or a status marked as uncertain. If no such anomaly exists, the language model obtains the logical matching results between the structured visual detection results and the specification clauses, and directly outputs a qualified or unqualified judgment result. If the aforementioned anomaly exists, a re-evaluation mechanism for the anomaly is executed; The execution of the review mechanism for the anomaly includes: The language model is used to generate a reassessment instruction for the target region corresponding to the anomaly, and the reassessment instruction includes image enhancement requirements or detail inspection requirements for the target region. The re-judgment instruction is input into the visual model again to perform secondary detection on the local image in the installation image corresponding to the target area, and obtain the re-judgment detection result; The language model updates the structured visual detection result based on the re-judgment detection result and regenerates the judgment result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the installation conformity testing method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the installation conformity testing method as described in any one of claims 1 to 8.

Images