Project construction site intelligent patrol method based on multi-modal large model

Abstract

The invention provides a project construction site intelligent patrol method based on a multi-modal large model, and relates to the cross field of multi-modal large model technology and building information, and the method comprises the steps: carrying out the standardization processing of historical patrol event data, and constructing a site patrol special knowledge base for supporting intelligent analysis; a field patrol special knowledge base is used as a training and reasoning basis, a generative AI technology and a visual analysis algorithm are fused, and a multi-modal intelligent assistant with natural language understanding and voice interaction functions is constructed; the method comprises the following steps: receiving project key information input by patrol personnel in a natural language through a multi-mode intelligent assistant; and analyzing the key information of the project through a multi-modal intelligent assistant, and querying the field patrol special knowledge base to obtain a recommended patrol item list matched with the key information. Efficient assistance is provided for engineering construction site patrol personnel, and the engineering construction quality is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of multimodal large model technology and building information, and in particular to an intelligent inspection method for project construction sites based on multimodal large models. Background Technology

[0002] A construction site intelligent inspection assistant based on a multimodal large model is mainly focused on the on-site inspection work of engineering construction projects. It provides inspectors with an integrated inspection tool, which includes functions such as reminders of inspection items before inspection, intelligent analysis of events during inspection, and closed-loop management of inspection events.

[0003] This invention utilizes multimodal large model technology, using historical inspection event data as the foundation for model training, to construct a specialized knowledge base for on-site inspections. It integrates natural language processing technology and machine learning algorithms to create an input-output pattern encompassing voice and semantic modalities, achieving intelligent interaction. On-site inspectors only need to input key project information (such as project type and construction stage) in text or voice format to obtain the required inspection items. During the inspection process, image algorithms can be used to conduct intelligent analysis of the on-site inspection content, generating preliminary inspection conclusions for reference. For issues identified during the inspection, a smart inspection assistant can generate and issue rectification tasks.

[0004] On-site inspections of engineering projects involve a significant workload. Current digital solutions typically focus on closed-loop management of inspection events, specifically covering the submission of inspection findings, the issuance of rectification tasks, feedback on rectification progress, and review by inspection personnel. This approach is suitable for experienced on-site inspectors. However, in actual project construction, there are many inexperienced inspectors, leading to oversights in the inspection work and consequently negatively impacting the quality of project construction. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an intelligent inspection method for project construction sites based on a multimodal large model, which provides efficient assistance to on-site inspection personnel and ensures the quality of project construction.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] Firstly, a method for intelligent on-site inspection of project construction based on a multimodal large model, the method comprising:

[0008] Standardize the historical inspection event data and build a special knowledge base for on-site inspections to support intelligent analysis;

[0009] Using a specialized knowledge base for on-site inspections as the foundation for training and reasoning, we integrate generative AI technology and visual analysis algorithms to build a multimodal intelligent assistant with natural language understanding and voice interaction capabilities.

[0010] The system receives key project information input by inspectors in natural language through a multimodal intelligent assistant; it also analyzes the key project information and queries the on-site inspection knowledge base to obtain a list of recommended inspection items that match the key information.

[0011] Based on the recommended list of inspection items, when an inspector selects a specific inspection item, the system receives the on-site visual data uploaded by the inspector. The multimodal intelligent assistant calls the visual analysis algorithm and combines it with the on-site inspection-specific knowledge base to identify and judge the on-site visual data in order to obtain a preliminary judgment on whether there are any abnormalities in the inspection item.

[0012] Based on the preliminary assessment, for inspection items deemed abnormal, a multimodal intelligent assistant automatically generates rectification tasks, assigns them to the relevant responsible persons, and tracks the task status. After the responsible persons provide feedback on the rectification results, the inspection personnel review the results through the multimodal intelligent assistant. If the review is successful, the process ends; if the review fails, the rectification process is restarted until closed-loop management of inspection events is achieved.

[0013] Furthermore, historical patrol event data is standardized to construct a specialized on-site patrol knowledge base to support intelligent analysis, including:

[0014] It receives and aggregates archived inspection event data from multiple historical project construction sites, including text and image records.

[0015] The collected patrol event data is cleaned and standardized to obtain a structured set of historical patrol data.

[0016] Based on the historical inspection data set, inspection items, related scenarios, problem types and handling rules are extracted as key features to obtain an initial inspection knowledge graph;

[0017] The initial inspection knowledge graph was reinforced and logically verified to ultimately build a specialized on-site inspection knowledge base to support intelligent analysis.

[0018] Furthermore, using a specialized on-site inspection knowledge base as the foundation for training and inference, generative AI technology and visual analysis algorithms are integrated to construct a multimodal intelligent assistant with natural language understanding and voice interaction capabilities, including:

[0019] Load the completed on-site inspection knowledge base, and based on the on-site inspection knowledge base, perform domain-adaptive fine-tuning on the pre-trained generative large language model to obtain a specialized language model with semantic understanding and generation capabilities in the construction site domain.

[0020] Based on a specialized knowledge base for on-site inspections, a pre-trained visual analysis model is trained under supervision to build a specialized visual analysis model capable of identifying specific scenarios and problems at the construction site.

[0021] It integrates specialized language models, specialized visual analysis models, and speech recognition and synthesis modules, and encapsulates and deploys them into a multimodal intelligent assistant service with multimodal input and output and intelligent interaction capabilities.

[0022] Furthermore, through a multimodal intelligent assistant, key project information input by inspection personnel in natural language is received; the multimodal intelligent assistant parses the key project information and queries the on-site inspection-specific knowledge base to obtain a list of recommended inspection items matching the key information, including:

[0023] The system receives voice input streams containing project type and construction stage from the front-end devices of the inspection personnel through the interaction interface of the multimodal intelligent assistant.

[0024] The voice input stream is parsed in a multimodal manner. If it is speech, it is converted into text by the speech recognition module, and the structured semantic features of key project information are extracted by a specialized language model.

[0025] Using structured semantic features as query conditions, semantic retrieval and matching are performed in the on-site inspection knowledge base to recall relevant candidate inspection items;

[0026] For each candidate inspection item, the items are sorted and filtered according to the relevance of their structured semantic features, and a list of recommended inspection items is output to the front-end device of the inspection personnel.

[0027] Furthermore, based on the recommended inspection item list, when an inspector selects a specific inspection item, the system receives on-site visual data uploaded by the inspector. The multimodal intelligent assistant invokes visual analysis algorithms and combines them with a specialized on-site inspection knowledge base to identify and judge the on-site visual data, thereby obtaining a preliminary judgment on whether the inspection item is abnormal, including:

[0028] For specific inspection items selected by the patrol personnel in the list of recommended inspection items on the front-end device, receive and load the on-site visual data associated with the specific inspection items uploaded;

[0029] A specialized visual analysis model is invoked to extract features and identify targets from the visual data on site, in order to obtain visual analysis results that include scene elements and potential problems;

[0030] Based on the semantic description of the current specific inspection item, retrieve the corresponding inspection standards and historical problem cases from the on-site inspection special knowledge base;

[0031] By combining the visual analysis results with the sub-inspection standards and historical cases, and using a specialized language model for fusion reasoning, a preliminary judgment on whether the inspection item is abnormal is obtained and output.

[0032] Furthermore, based on the preliminary assessment, for inspection items deemed abnormal, a multimodal intelligent assistant automatically generates rectification tasks, assigns them to the relevant responsible persons, and tracks the task status, including:

[0033] Receive the preliminary judgment conclusion generated for a specific inspection item and determined to be abnormal, along with the associated on-site visual data and semantic description;

[0034] Based on the preliminary judgment and semantic description, a specialized language model is invoked to automatically generate a structured description of the rectification task, which includes the problem location, rectification requirements and normative basis.

[0035] Based on the professional fields and responsibilities involved in the structured rectification task description, a match is made in the pre-configured organizational personnel database, and a single responsible person is automatically assigned and a task work order is generated.

[0036] The task work order is pushed to the terminal device of the assigned person in charge, and the status of the task work order is continuously tracked until the rectification results are received from the person in charge.

[0037] Furthermore, after the responsible party reports the rectification results, the inspection personnel review the results through the aforementioned multimodal intelligent assistant. If the review is successful, the process ends; if it fails, the rectification process is restarted until closed-loop management of the inspection event is achieved, including:

[0038] Receive rectification result information from the responsible party through their terminal device. The rectification result information includes on-site visual data and text descriptions after rectification.

[0039] The rectification results and original problem records are pushed to the front-end devices of the inspectors, and a special language model is called to generate key points prompts to assist in the review.

[0040] Receive the audit and judgment results made by the inspection personnel based on key points and suggestions;

[0041] If the review result is approved, the task work order status is updated to closed loop, and the inspection event is marked as completed; if the review result is not approved, the review comments are automatically recorded to generate a new rectification task description, and the task work order status is reset to re-execute a new round of rectification process until the review is approved.

[0042] Secondly, an intelligent inspection system for project construction sites based on a multimodal large model includes:

[0043] The acquisition module is used to standardize historical patrol event data and build a special knowledge base for on-site patrols to support intelligent analysis.

[0044] The module is used to build a multimodal intelligent assistant with natural language understanding and voice interaction capabilities by using a field inspection knowledge base as the basis for training and inference, integrating generative AI technology and visual analysis algorithms.

[0045] The parsing module is used to receive key project information input by inspectors in natural language through a multimodal intelligent assistant; it parses the key project information through the multimodal intelligent assistant and queries the on-site inspection knowledge base to obtain a list of recommended inspection items that match the key information.

[0046] The judgment module is used to receive on-site visual data uploaded by the inspector when the inspector selects a specific inspection item from the recommended inspection item list. The multimodal intelligent assistant calls the visual analysis algorithm and combines it with the on-site inspection-specific knowledge base to identify and judge the on-site visual data in order to obtain a preliminary judgment on whether there is any abnormality in the inspection item.

[0047] The processing module is used to automatically generate rectification tasks for inspection items identified as abnormal based on the preliminary judgment conclusion, assign them to the corresponding responsible persons, and track the task status. After the responsible persons provide feedback on the rectification results, the inspectors review the results through the multimodal intelligent assistant. If the review is passed, the process of the matter ends. If the review is not passed, the rectification process is restarted until the closed-loop management of the inspection event is achieved.

[0048] Thirdly, a computing device includes:

[0049] One or more processors;

[0050] A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0051] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0052] The above-described solution of the present invention has at least the following beneficial effects:

[0053] This project leverages technologies such as multimodal large models, generative AI, and image analysis algorithms to build an intelligent assistant for on-site inspections. This assistant effectively empowers the entire process of on-site inspections before, during, and after construction projects, thereby significantly improving the efficiency, completeness, and accuracy of on-site inspections.

[0054] Before the inspection, the inspection item recommendation function can innovate the traditional human-to-human training model. Relying on a specialized knowledge base and advanced generative artificial intelligence technology, it can provide professional, systematic, and efficient business guidance for inspection personnel who lack field experience, promote the rapid transfer and sharing of knowledge, skills, and practical experience, and significantly improve the efficiency of personnel training and task execution. On the other hand, this intelligent recommendation mechanism can also effectively reduce the risk of missing key items due to human negligence or lack of experience, ensuring that the items covered in each inspection task are comprehensive and without blind spots, and guaranteeing the standardization and integrity of the inspection work.

[0055] During the inspection, the real-time inspection situation judgment function can perform instant intelligent analysis on uncertain situations or complex scenarios encountered on site, and quickly generate reliable analysis conclusions and handling suggestions to assist inspection personnel in making accurate judgments, thereby improving the processing efficiency and accuracy of inspection work and reducing various risks caused by human misjudgment.

[0056] After the inspection, for the identified problems that require rectification, the intelligent assistant can be used to realize the full-process digital management from task issuance and rectification progress tracking to result verification, forming a traceable, assessable, and optimizable closed-loop management mechanism to ensure that problems are rectified and responsibilities are assigned to specific individuals, ultimately effectively guaranteeing the quality and safety level of the project construction. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating an intelligent on-site inspection method for project construction based on a multimodal large model, provided by an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of an intelligent inspection system for project construction sites based on a multimodal large model, provided by an embodiment of the present invention. Detailed Implementation

[0059] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0060] like Figure 1As shown, an embodiment of the present invention proposes an intelligent inspection method for project construction sites based on a multimodal large model. The method includes the following steps:

[0061] Step 1: Standardize the historical patrol event data and build a special knowledge base for on-site patrols to support intelligent analysis;

[0062] Step 2: Using the on-site inspection knowledge base as the basis for training and reasoning, we integrate generative AI technology and visual analysis algorithms to build a multimodal intelligent assistant with natural language understanding and voice interaction capabilities.

[0063] Step 3: Receive key project information input by the inspection personnel in natural language through the multimodal intelligent assistant; parse the key project information through the multimodal intelligent assistant and query the on-site inspection knowledge base to obtain a list of recommended inspection items matching the key information;

[0064] Step 4: Based on the recommended inspection item list, when the inspector selects a specific inspection item, the system receives the on-site visual data uploaded by the inspector; the multimodal intelligent assistant calls the visual analysis algorithm and combines it with the on-site inspection-specific knowledge base to identify and judge the on-site visual data in order to obtain a preliminary judgment on whether there is any abnormality in the inspection item.

[0065] Step 5: Based on the preliminary judgment, for inspection items identified as having abnormalities, a rectification task is automatically generated through the multimodal intelligent assistant, assigned to the corresponding responsible person, and the task status is tracked. After the responsible person reports the rectification results, the inspection personnel review the results through the multimodal intelligent assistant. If the review is passed, the process of the matter ends. If the review is not passed, the rectification process is restarted until the closed-loop management of the inspection event is achieved.

[0066] In this embodiment of the invention, because the invention processes historical inspection event data to construct a specialized on-site inspection knowledge base, and integrates generative AI technology and visual analysis algorithms based on this knowledge base, it creates a multimodal intelligent assistant with natural language understanding, voice interaction, and visual data recognition capabilities. Through the entire process of knowledge base construction, intelligent assistant setup, item recommendation, anomaly detection, and closed-loop management, it effectively overcomes the technical problems of inspectors' lack of clarity regarding inspection items, difficulty in judging on-site conditions, difficulty in forming a closed loop for event handling, and insufficient support for inexperienced inspectors in existing digital solutions, which can easily lead to inspection oversights. As a result, it significantly improves the efficiency, completeness, and accuracy of on-site inspections, innovates traditional training models to accelerate the transfer of knowledge and experience, reduces the risk of human error and omissions, ensures that problems are rectified and responsibilities are assigned to individuals, and ultimately strongly guarantees the quality and safety level of engineering construction.

[0067] In a preferred embodiment of the present invention, step 1 above may include:

[0068] Step 1.1: Receive and collect archived inspection event data from multiple historical project construction sites. This data includes both text and image records. Specifically, this involves initiating the collection of historical inspection event data, clearly defining the collection scope to cover multiple completed or partially completed construction projects, encompassing various project types such as building construction, roads, tunnels, and landscaping. It also includes key construction phases such as foundation pits, civil engineering, electromechanical systems, and decoration, ensuring the collected data covers the entire scene of construction site inspections. The focus is on collecting all officially archived inspection-related data from the project's construction process, with text records fully encompassing the inspection events. The core information includes the specific name of the item, a detailed description of the problem found on site, the specific time the problem was found, the temporary measures taken at the time, the clear rectification requirements, the acceptance standards after rectification, and the final audit results; the image record needs to collect various high-definition photos and complete video data taken by the inspectors on site that can truly reflect the operation status of various facilities and equipment on the construction site, the quality of building materials, the overall scene of the specific problem, and the surrounding construction environment conditions, while ensuring that the collected data are all historical data that have undergone formal archiving process and have authenticity and validity, without omitting any key project or key stage inspection archiving information.

[0069] Step 1.2 involves cleaning and standardizing the collected patrol event data to obtain a structured historical patrol data set. This includes: conducting detailed cleaning and standardization work in stages for all collected historical patrol event data; during the data cleaning stage, professional personnel will verify each data item to remove completely duplicate and redundant data items, avoiding interference from duplicate data in subsequent analysis; carefully correcting various errors in the data, including typos, ambiguities, errors in project type, time, or numerical recording, to ensure the accuracy of textual information; and addressing missing key information items in the data, such as unmarked project construction stages, patrol item categories, and the person who discovered the problem. Relevant information was supplemented and improved by reviewing the original construction files of the corresponding projects and contacting relevant historical staff to ensure the completeness of key information in each data entry. During the data standardization phase, a unified text recording specification was established, unifying the names of inspection items, problem types, and descriptions of handling measures—which differed in expression across projects but shared the same substantive meaning—into industry-standard expressions. All data were re-labeled and categorized according to a pre-set unified classification standard, ensuring consistency in classification criteria. All image records were uniformly converted to a common image format and adjusted to standard resolution and size ratios to eliminate differences caused by different shooting equipment and storage formats. After this comprehensive cleaning and standardization process, a structured historical inspection data set was finally obtained, characterized by a unified structure, standardized content, complete information, and direct support for analysis.

[0070] Step 1.3: Based on the historical inspection data set, extract inspection items, associated scenarios, problem types, and handling rules as key features to obtain an initial inspection knowledge graph. Specifically, this includes: using the cleaned and standardized structured historical inspection data set as the core foundation, organizing a professional data analysis team to conduct key feature extraction and initial knowledge graph construction; deeply analyzing each data entry in the structured data set to accurately identify and extract all specific inspection items from the text records, covering all inspection items in various stages and professional fields of engineering construction, such as support structure integrity inspection, support system deformation inspection, edge protection setting inspection, electromechanical equipment installation specification inspection, and building material quality inspection; then, for each extracted inspection item, clarify its corresponding associated scenario, i.e., the specific project type and specific construction stage to which the inspection item applies, establishing a correspondence between inspection items and application scenarios; and systematically analyzing all problems that occurred in the historical data for each inspection item. The system comprehensively categorizes and classifies problems into different types, such as structural deformation issues, lack of protection, improper installation, substandard material quality, and safety hazards. It also defines the specific characteristics and judgment criteria for each problem type. Simultaneously, it comprehensively analyzes the standard handling rules for each problem type from the data, including the reporting process after problem discovery, division of responsibilities, rectification measures, rectification completion deadlines, and acceptance standards. After extracting all key features, the system is systematically organized based on the inherent logical connections between these features and actual engineering construction site inspections. For example, it establishes a mapping relationship between specific project types and specific construction stages and corresponding inspection items, a correspondence between each inspection item and common problem types under that item, and a matching relationship between each problem type and its corresponding handling rules. Through this multi-dimensional organizational structure, an initial inspection knowledge graph is constructed, encompassing four core elements: inspection items, related scenarios, problem types, and handling rules, as well as the complex relationships between these elements.

[0071] Step 1.4 involves strengthening the associations and verifying the logic of the initial inspection knowledge graph, ultimately constructing a specialized on-site inspection knowledge base to support intelligent analysis. This includes: to ensure the constructed knowledge base accurately supports subsequent intelligent analysis, a comprehensive strengthening and logical verification optimization of the initial inspection knowledge graph is conducted. In the association strengthening phase, potential relationships between core elements are deeply explored, supplementing reasonable associations not reflected in the initial knowledge graph. For example, based on existing associations between inspection items and problem types, the differentiated associations between the severity of different problems and handling rules are further refined, and the adjustment of handling rules for the same problem type under different associated scenarios is supplemented. This strengthens the connection between inspection items and enriches the attribute information of each element in the knowledge graph, such as adding attributes like inspection frequency and inspection method to inspection items, and adding attributes like common occurrence probability and risk level to problem types. This enhances the completeness, tightness, and richness of the associations between elements in the knowledge graph.

[0072] In the logic verification stage, the knowledge graph is comprehensively checked against national and industry-related engineering construction specifications, on-site inspection management standards, and the business logic of actual inspection work. This involves checking whether the definitions of each core element are accurate and consistent, avoiding multiple definitions for the same element; verifying whether the relationships between elements are reasonable and compliant, and whether there are any relationships that conflict with industry standards or business logic; and investigating whether there are any logical contradictions in the knowledge graph, such as conflicting judgment standards for the same inspection item in the same related scenario, or contradictory handling procedures for the same problem type. For issues such as missing relationships, logical contradictions, and inconsistent definitions found during the verification process, professional technicians and senior inspection personnel are promptly organized to discuss and make corrections and adjustments based on industry standards and practical work experience. After multiple rounds of relationship strengthening and logic verification, a site inspection-specific knowledge base with a complete structure, rigorous logic, accurate content, rich relationships, and strong support for the multimodal intelligent assistant's functions such as inspection item recommendation, on-site situation judgment, and closed-loop management is finally obtained.

[0073] In this embodiment of the invention, by employing the technical means of receiving and aggregating archived text and image records of inspection events from multiple historical project construction sites, cleaning and standardizing the aggregated data to form a structured historical inspection data set, extracting inspection items, associated scenarios, problem types, and handling rules as key features based on the data set to construct an initial inspection knowledge graph, and then strengthening and logically verifying the initial graph to finally construct a specialized on-site inspection knowledge base, this invention overcomes the technical problems of historical inspection data being scattered, disorganized, inconsistent in format, weak in knowledge association, and lacking systematic organization, which makes it difficult to support accurate item recommendations and on-site situation judgments in intelligent inspections. Furthermore, existing solutions, lacking a professional structured knowledge base, cannot effectively meet the needs of inexperienced inspection personnel. This invention achieves the technical effect of constructing a specialized knowledge base that is structurally complete, logically rigorous, accurate in content, and closely related, providing solid knowledge support for the full-process intelligent inspection function of the multimodal intelligent assistant, ensuring that inspection item recommendations are relevant to the actual project and that on-site situation judgments are accurate and reliable, while also providing professional knowledge empowerment for inexperienced inspection personnel and reducing inspection oversights.

[0074] In a preferred embodiment of the present invention, step 2 above may include:

[0075] Step 2.1: Load the completed on-site inspection knowledge base. Based on the on-site inspection knowledge base, perform domain-adaptive fine-tuning on the pre-trained generative large language model to obtain a specialized language model with semantic understanding and generation capabilities in the construction site domain. Specifically, this includes: starting the construction of the specialized language model; first, fully loading the previously completed on-site inspection knowledge base into the model training environment to ensure that the knowledge base can be fully called and learned by the pre-trained generative large language model. This specialized knowledge base contains rich and professional structured knowledge such as various inspection items, related scenarios, problem types, and handling rules in the field of engineering construction, providing a solid foundation for the domain adaptation of the model; and using all the structured data in the on-site inspection knowledge base as core training samples to formulate a targeted domain-adaptive fine-tuning plan.

[0076] During the fine-tuning process, the focus was on enhancing the pre-trained generative language model's ability to understand the professional semantics of the construction site domain. This enabled the model to accurately identify the meanings of professional terms such as project type, construction stage, inspection item names, and problem descriptions, and to accurately grasp the core semantics behind different expressions, avoiding misunderstandings caused by semantic ambiguity. Emphasis was placed on training the model's ability to generate professional content, enabling it to generate inspection item descriptions, problem analysis suggestions, and rectification requirements that conform to industry standards, are logically rigorous, and accurately expressed based on the actual needs of the construction site scenario. During the fine-tuning process, the training effect of the model was continuously monitored, and the training parameters were optimized and adjusted through multiple iterations to continuously improve the model's accuracy in understanding the semantics of the construction site domain and the quality of professional content generation, ultimately resulting in a highly adaptable construction site domain-specific language model.

[0077] Step 2.2: Based on the on-site inspection-specific knowledge base, supervised training is performed on the pre-trained visual analysis model to construct a specialized visual analysis model capable of recognizing specific scenarios and problems at the construction site. Specifically, this includes: initiating the construction of the specialized visual analysis model, using the completed on-site inspection-specific knowledge base as the core training basis, and extracting all key data related to visual recognition at the construction site, including standard on-site images corresponding to various inspection items, typical image samples of different problem types, and detailed image data of problem scenarios. The image data are accompanied by detailed text information such as category labels, problem descriptions, and judgment criteria, providing rich and accurate sample support for the supervised training of the model.

[0078] The extracted image samples are associated and integrated with the corresponding text annotation information to construct a structured supervised training dataset. This ensures that each image sample in the dataset corresponds to the professional knowledge in the specialized knowledge base. The supervised training dataset is then input into a pre-trained visual analysis model for targeted supervised training. During the training process, the model is guided to learn to recognize specific scenarios at construction sites, including foundation pit excavation, civil engineering, electromechanical installation, and decoration construction. This allows the model to accurately distinguish the core features of different scenarios and strengthens its ability to identify various specific problems at construction sites, such as damaged support structures, deformed support systems, missing edge protection, substandard materials, and non-standard equipment installation. The model can quickly locate potential problem areas from complex site images, extract problem features, and compare them with standard data in the knowledge base. During training, the training strategy is continuously adjusted based on the differences between the model's recognition results and the annotation information to optimize model parameters and continuously improve the model's accuracy and recall for specific scenarios and problems. Ultimately, a specialized visual analysis model capable of accurately identifying specific scenarios and various problems at construction sites is constructed.

[0079] Step 2.3 integrates the specialized language model, specialized visual analysis model, and speech recognition and synthesis module, encapsulating and deploying them into a multimodal intelligent assistant service with multimodal input / output and intelligent interaction capabilities. Specifically, this includes: initiating the integration and deployment of the multimodal intelligent assistant service, clarifying the integration logic and interaction flow of each core module, and ensuring that the specialized language model, specialized visual analysis model, and speech recognition and synthesis module can work efficiently together. During the module integration phase, the specialized language model serves as the semantic processing core, responsible for parsing the semantic information corresponding to the text or speech input by the patrol personnel, generating professional text responses and analysis conclusions; the constructed specialized visual analysis model serves as the image analysis core, responsible for processing the on-site image data uploaded by the patrol personnel, identifying scenes and problems, and outputting analysis results; the speech recognition and synthesis module serves as the voice interaction core, realizing the conversion of voice input to text and text to speech output, ensuring smooth voice interaction. By building a unified module interaction interface, a real-time data transmission channel is established between the modules to ensure the rapid transmission and efficient fusion of semantic processing results, image analysis results, and speech conversion data. When the patrol personnel... When inputting key project information via voice, the speech recognition module first converts the speech into text, then passes it to a specialized language model for semantic analysis. The language model, combined with a knowledge base, generates recommended inspection items, which are then fed back to the inspection personnel via the speech synthesis module. When inspection personnel upload images and initiate analysis requests, a specialized visual analysis model processes the images and passes the results to a specialized language model. The language model, combined with a knowledge base, generates preliminary judgments in natural language, which are then fed back to the user through an interactive interface. After module integration, the overall system is encapsulated, its architecture optimized, and its stability and operational efficiency improved. It is also adapted to mobile devices commonly used by inspection personnel, ensuring stable operation in complex network environments and usage scenarios at construction sites. Following standardized deployment procedures, the encapsulated system is deployed to a designated server or cloud platform, with the appropriate operating environment and resources configured. System initialization and testing are completed to ensure all system functions are implemented correctly. Ultimately, this forms a multimodal intelligent assistant service with multimodal input / output capabilities (text, voice, and images) and efficient intelligent interaction capabilities, providing inspection personnel with an integrated intelligent inspection tool.

[0080] In this embodiment of the invention, a pre-built on-site inspection knowledge base is loaded. Based on this knowledge base, a pre-trained generative large language model is fine-tuned for domain adaptation to obtain a specialized language model with semantic understanding and generation capabilities for the construction site domain. Simultaneously, a pre-trained visual analysis model is supervised and trained based on this knowledge base to construct a specialized visual analysis model capable of recognizing specific scenarios and problems at the construction site. Then, the specialized language model, specialized visual analysis model, and speech recognition and synthesis module are integrated, encapsulated, and deployed as a multimodal intelligent assistant service with multimodal input / output and intelligent interaction capabilities. Therefore, this overcomes the limitations of general generative large language models and visual analysis models, which lack domain-specificity for construction sites, and suffer from semantic understanding and scenario-based limitations. The existing solutions lack professionalism and accuracy in identification and problem judgment, and are not effectively integrated with the voice interaction module, failing to meet the multimodal and convenient interaction needs of inspection personnel. Furthermore, they struggle to provide precise semantic guidance and visual analysis support to inexperienced inspection personnel. This solution aims to enable the intelligent assistant to possess specialized semantic understanding and generation capabilities specific to the construction site, as well as accurate recognition of specific scenarios and problems. It achieves smooth intelligent interaction across multiple modalities (voice, text, and images), providing inspection personnel with more tailored professional support, improving the accuracy of inspection item recommendations, the reliability of on-site situation assessment, and the convenience of interactive operations. This effectively compensates for the shortcomings of inexperienced inspection personnel, further ensuring the quality and efficiency of inspection work.

[0081] In a preferred embodiment of the present invention, step 3 above may include:

[0082] Step 3.1: Through the interaction interface of the multimodal intelligent assistant, receive the voice input stream containing project type and construction stage sent by the front-end device of the inspection personnel. Specifically, the inspection personnel log in to the deployed multimodal intelligent assistant application through the front-end device, such as a smartphone or tablet. In the application interface, find the entry point for the inspection item recommendation function and click to enter. The system automatically jumps to the corresponding interaction interface. This interaction interface has a dedicated voice input channel. According to the system prompts, the inspection personnel clearly speak relevant information containing project type and construction stage into the voice acquisition device of the front-end device, such as the main structure construction stage of a building project or the roadbed paving stage of a road project. The interaction interface of the multimodal intelligent assistant is in real-time data receiving state, receiving the voice input stream transmitted by the front-end device through a stable network connection. During the receiving process, the data transmission status is monitored in real time to ensure that the voice input stream is complete and without loss. At the same time, the transmitted data is initially formatted to ensure that the parsing work can be carried out smoothly and the voice input data containing the core key information of the project can be successfully obtained.

[0083] Step 3.2 involves multimodal parsing of the voice input stream. If it is voice, it is converted into text by the speech recognition module, and the structured semantic features of key project information are extracted using a specialized language model. Specifically, after receiving the voice input stream, the multimodal intelligent assistant immediately initiates the multimodal parsing process. The system first identifies and determines the type of input data. After confirming that the received data is a voice input stream, it transmits it to the built-in speech recognition module. The speech recognition module uses a speech recognition algorithm adapted to the engineering construction field, which can accurately identify professional terms related to the construction site and effectively avoid recognition errors caused by industry-specific expressions. The continuous voice input stream is converted into corresponding text information segment by segment. After the text conversion is completed, the system automatically calls the previously constructed specialized language model and inputs the converted text information into the model. Based on the professional knowledge in the on-site inspection specialized knowledge base, the specialized language model performs deep semantic analysis on the text information, accurately extracts the two core key information categories of project type and construction stage, and organizes the information according to the preset structured format to obtain structured semantic features including project type identifiers, construction stage identifiers, and other dimensions. This ensures that the extracted features can accurately reflect the core needs of the inspection personnel and provide accurate data support for retrieval and matching.

[0084] Step 3.3 uses structured semantic features as query conditions to perform semantic retrieval and matching in the on-site inspection knowledge base, recalling relevant candidate inspection items. Specifically, after the multimodal intelligent assistant receives the voice input stream transmitted from the front-end device, it immediately starts the multimodal parsing process to identify and determine the type of input data. Through a preset interface, it transmits the voice input stream completely to the voice recognition module, triggering the module's preprocessing process. The module automatically performs noise reduction processing on the voice input stream and splits long voices according to semantic integrity. After the module completes the preprocessing, it starts the core recognition process: relying on the acoustic features and language models learned during training in the engineering field, it extracts features frame by frame from the voice segments, converting the voice signal into a feature vector that the model can recognize; then, through the decoding stage, combined with the engineering-specific dictionary and contextual semantics, the feature vector is mapped to the corresponding text information. Because the module enhances its ability to recognize professional terms and combined expressions during its construction, it can accurately avoid errors caused by industry-specific expressions. When inspectors mention the main structure construction stage of a building project, the module will not misidentify it because the main structure is an industry-specific expression, nor will it break down the combination logic of project type + construction stage. Instead, it will directly convert it into the standard text "main structure construction stage of building project". For inspection-related expressions such as whether the support structure is complete, it can also accurately restore the original meaning of the terms and avoid semantic deviations.

[0085] After the speech recognition module completes the text conversion, it automatically transmits the standardized text information to the system's core processing unit, triggering the invocation of the specialized language model. The text output by the module must meet two core requirements: first, standardized terminology; and second, semantic completeness. This lays the foundation for deep semantic analysis and accurate extraction of structured semantic features by the specialized language model. If the converted text contains terminology errors, it will directly lead to deviations in structured feature extraction, affecting the accuracy of the inspection item recommendations. However, this module, through its engineering scenario-oriented construction, precisely avoids this problem, ensuring the reliability of the text input.

[0086] Step 3.4: For the candidate inspection items, sort and filter them according to the relevance of their structured semantic features, obtain and output the recommended inspection item list to the front-end device of the inspection personnel. Specifically, this includes: after obtaining the set of candidate inspection items, starting the relevance analysis and sorting and filtering process, calculating the degree of relevance between each candidate inspection item and the current query conditions based on the structured semantic features extracted by the specialized language model. The relevance calculation is mainly based on a comprehensive evaluation of multiple dimensions such as the matching accuracy between the candidate item and the project type construction stage, and the importance weight in the same stage of similar projects. Sort all candidate inspection items in order of relevance from high to low, and put the inspection items that are most suitable for the current project scenario and have the most critical impact on the quality and safety of the project at the top.

[0087] After sorting, according to the preset filtering rules, duplicate inspection items, items that are not applicable to the actual construction scenario of the current project, and minor and unnecessary peripheral items are removed from the candidate set. The candidate item list is then optimized, and the recommended inspection item list after sorting and filtering is output to the front-end device of the inspection personnel in a clear and intuitive form. Each inspection item in the list includes a detailed name, core inspection content, and brief inspection basis prompts, which makes it convenient for the inspection personnel to quickly view and understand.

[0088] In this embodiment of the invention, the technical means of receiving a voice input stream containing project type and construction stage sent by the front-end device of the inspection personnel through the interaction interface of a multimodal intelligent assistant, performing multimodal parsing on the voice input stream, extracting structured semantic features of key project information with the help of a specialized language model, and then using the structured semantic features as query conditions to conduct semantic retrieval and matching in the on-site inspection-specific knowledge base to recall related candidate inspection items, and finally sorting and filtering the candidate inspection items according to the correlation between the candidate inspection items and the structured semantic features and outputting the recommended inspection item list to the front-end device of the inspection personnel, overcomes the technical problems of inspection personnel, especially those with little experience, not being clear about the inspection items corresponding to different project types and construction stages, and the lack of targeted inspection item recommendation support in existing solutions, which easily leads to the omission of key inspection items. Thus, it achieves the technical effect of enabling inspection personnel to quickly obtain accurate inspection item recommendations that fit the actual needs of the project through a convenient voice input method. The operation is efficient and convenient, effectively making up for the business knowledge shortcomings of inexperienced inspection personnel, ensuring comprehensive coverage of inspection items without blind spots, and improving the standardization and completeness of inspection work.

[0089] In a preferred embodiment of the present invention, step 4 above may include:

[0090] Step 4.1: For specific inspection items selected by the patrol personnel in the recommended inspection item list on the front-end device, the system receives and loads the on-site visual data associated with those specific inspection items. Specifically, the patrol personnel find the specific inspection item corresponding to the current on-site inspection work in the recommended inspection item list on the front-end device and select it. The system automatically redirects to the special reporting interface for that inspection item. This interface has a pre-set visual data upload entry, supporting both real-time upload from on-site photography and upload of locally stored visual data. The patrol personnel take photos or videos that clearly reflect the actual on-site situation of the inspection item based on the actual inspection conditions and submit them through the upload entry. The multimodal intelligent assistant's interactive interface receives the visual data transmitted from the front-end device in real time, while automatically verifying the data's format compatibility and clarity to ensure the data meets analysis requirements. After successful verification, the system uniquely associates and binds the visual data with the selected specific inspection item to avoid data confusion and loads the complete data into the model analysis module's cache.

[0091] Step 4.2: Invoke the specialized visual analysis model to perform feature extraction and target recognition on the on-site visual data to obtain visual analysis results containing scene elements and potential problems. Specifically, after the visual data is loaded, the multimodal intelligent assistant automatically starts the invocation process of the specialized visual analysis model. The on-site visual data, after being associated and bound, is completely input into the model through the preset module interface. The specialized visual analysis model is trained based on a large number of construction site visual samples in the on-site inspection specialized knowledge base, which can accurately adapt to the scene recognition needs in the field of engineering construction. The model preprocesses the visual data, including image enhancement, noise reduction, brightness and contrast adjustment, etc., to improve data quality. The model starts the feature extraction process to accurately extract key visual features related to the current inspection item from the visual data, such as the structural dimensions, appearance status, installation position and connection method of the inspection object. Simultaneously, the model performs target recognition, accurately locates the core inspection objects in the visual data, identifies whether there are suspected problems such as positional deviation, structural damage, missing parts, deformation, or excessive stains, clearly marks the specific areas and preliminary features of potential problems, integrates the feature extraction results and target recognition results to obtain visual analysis results containing the core elements of the current inspection item, the location of potential problems, and the preliminary judgment of potential problem types. This result is then synchronously transmitted to the fusion reasoning module of the specialized language model.

[0092] Step 4.3: Based on the semantic description of the current specific inspection item, retrieve the corresponding inspection standards and historical problem cases from the on-site inspection knowledge base. Specifically, while receiving the visual analysis results, the specialized language model automatically extracts the complete semantic description information of the current specific inspection item, including the name of the inspection item, core inspection requirements, scope of application, and technical indicators. Using this semantic description as the retrieval basis, the system initiates a high-speed connection with the on-site inspection knowledge base and conducts a precise search within the knowledge base using a semantic matching algorithm. During the search, priority is given to matching entries that are completely consistent with the semantic description of the inspection item, obtaining the corresponding official inspection standards, industry specifications, project-specific requirements, etc., clarifying the qualification judgment basis and specific indicator parameters of the inspection item. Historical problem cases related to the inspection item are retrieved from the knowledge base, including detailed information such as the types of problems, problem manifestations, and rectification basis that have occurred in similar projects at the same stage. The system filters and organizes the retrieved inspection standards and historical problem cases, removing duplicate or less relevant content to obtain a structured search result set, ensuring that the fusion reasoning obtains accurate and effective reference basis.

[0093] Step 4.4: Integrating the visual analysis results with the sub-inspection standards and historical cases, a specialized language model is used for fusion reasoning to obtain and output a preliminary judgment on whether the inspection item is abnormal. Specifically, after receiving the visual analysis results and retrieved historical problem cases from the inspection standards, the specialized language model initiates a multi-dimensional fusion reasoning process. It compares the potential problem features in the visual analysis results with the pass / fail indicators in the inspection standards one by one to determine whether the potential problem exceeds the allowable range or meets the non-compliance criteria. It also performs a similarity analysis between the scene elements and potential problems in the current visual data and historical problem cases, referencing the judgment logic and conclusions of similar problems to provide guidance for the current judgment. During the reasoning process, the model fully integrates... The model employs professional logic from the engineering construction field, comprehensively considering influencing factors such as objective conditions of the construction environment to avoid misjudgments caused by mechanical comparisons. For support structure inspection items, the model combines the structural deformation extracted by visual analysis with the allowable deformation value in the inspection standard. It also refers to historical cases of deformation problems with similar foundation pit depths and support types to determine whether the current deformation is abnormal. After reasoning, the specialized language model generates a standardized and rigorous preliminary judgment conclusion. The conclusion clearly states whether the inspection item is abnormal, the specific location of the abnormality, the inspection standard clauses on which the judgment is based, and key information from historical cases. This is output to the front-end equipment of the inspection personnel in a clear and intuitive text format, providing professional and accurate reference for the inspection personnel's final judgment.

[0094] In this embodiment of the invention, the technical means of receiving and loading the on-site visual data associated with the specific inspection items selected by the patrol personnel in the recommended inspection item list on the front-end device, calling a specialized visual analysis model to extract features and identify targets in the on-site visual data to obtain visual analysis results containing scene elements and potential problems, retrieving the corresponding inspection standards and historical problem cases in the on-site inspection specialized knowledge base based on the semantic description of the current specific inspection item, and then using a specialized language model to integrate the visual analysis results, inspection standards, and historical cases for fusion reasoning, and finally obtaining and outputting a preliminary judgment conclusion on whether the inspection item is abnormal, overcomes the technical problems of patrol personnel, especially those with little experience, having difficulty judging the situation of specific inspection items during on-site inspections, relying on personal experience and being prone to misjudgment or uncertainty, and the lack of professional intelligent analysis support in existing solutions leading to low judgment efficiency and insufficient accuracy. Thus, it achieves the technical effect of providing patrol personnel with objective and accurate preliminary judgment conclusions as a reference, improving the efficiency and accuracy of on-site inspection situation judgment, effectively reducing the risk caused by human misjudgment, and providing professional technical support for inexperienced patrol personnel to accurately complete on-site inspection work.

[0095] In a preferred embodiment of the present invention, step 5 above may include:

[0096] Step 5.1: Receive the preliminary judgment conclusion generated for a specific inspection item and determined to be abnormal, along with the associated on-site visual data and semantic description. Specifically, this includes: after the abnormality judgment of the specific inspection item is completed, an automatically triggered closed-loop management process is initiated, and the multimodal intelligent assistant immediately starts the data receiving mechanism. First, the system receives a preliminary judgment conclusion from a specialized language model, indicating the presence of an anomaly. This conclusion includes core information such as the specific type of the anomaly (e.g., structural deformation, substandard materials, lack of protection), and the severity (e.g., minor, moderate, severe). It also receives the on-site visual data associated with the inspection item, including high-resolution photos and videos taken by inspectors reflecting the anomaly, along with auxiliary information such as shooting angle and time. Finally, it receives a complete semantic description corresponding to the inspection item, covering key information such as the item name, inspection location, project type, and construction stage. During this process, the system automatically and uniquely identifies and associates the preliminary judgment conclusion with the on-site visual data and semantic description, ensuring a one-to-one correspondence and preventing confusion. The system then performs a complete data integrity check, verifying for missing key information, such as unclear anomaly locations or lack of corresponding visual data. If any missing information is found, a prompt is automatically sent to the inspector's device, requesting supplementation. If the data is complete, the verification passes, and the associated complete set of data is stored in the system's designated database, providing comprehensive data support for subsequent rectification tasks.

[0097] Step 5.2: Based on the preliminary judgment and semantic description, the specialized language model is invoked to automatically generate a structured rectification task description containing problem location, rectification requirements, and regulatory basis. Specifically, after data verification, the multimodal intelligent assistant automatically invokes the specialized language model trained on the site inspection specialized knowledge base. The specialized language model first performs deep analysis on the received preliminary judgment semantic description, accurately extracting the core features of the abnormal problem, including the specific location of the abnormality, such as the north side of the three-story scaffolding on the east side of the foundation pit support structure, the abnormal manifestations such as cracks reaching five centimeters in length, the support system tilting three centimeters, and the lack of edge protection, and the potential impact of the abnormality on the quality and safety of the project. This completes the accurate description of the problem location. Based on the site inspection specialized knowledge base, the model retrieves relevant industry specifications, construction standards, historical handling rules, etc., for this type of abnormal problem. Based on the severity of the anomaly, the specific measures required for rectification are clearly defined, such as grouting the cracked areas, recalibrating the support system, installing standard edge guardrails, etc. The timeframes for rectification completion are also specified, such as completion within 24 hours and feedback of rectification results within 72 hours. The acceptance standards after rectification are defined, such as crack closure, no water seepage, verticality deviation of the support system within allowable range, and guardrail height conforming to specifications. This results in clear and specific rectification requirements. The model extracts corresponding regulatory basis from the knowledge base, including relevant national and industry standard numbers, specific clauses, project-specific construction plan requirements, etc., ensuring that the rectification work is systematic and regulated. The specialized language model, following a pre-set structured format, integrates and organizes precise problem identification, clear rectification requirements, and authoritative regulatory basis to generate a logically rigorous, complete, and standardized structured rectification task description, ensuring that the responsible party clearly understands the core rectification needs and execution standards.

[0098] Step 5.3: Based on the professional fields and responsibilities involved in the structured rectification task description, the system matches the pre-configured personnel database, automatically assigns a single responsible person, and generates a task work order. Specifically, after the structured rectification task description is generated, the system automatically parses the professional fields involved in the task, such as civil engineering corresponding to support structure issues, electromechanical engineering corresponding to equipment installation issues, and decoration engineering corresponding to wall flatness issues, etc., and the responsibilities, such as construction team leader, technical supervisor, and supervising engineer, etc. The pre-configured personnel database stores detailed information on all participants in the project, including name, department, professional qualifications (e.g., civil engineer, electromechanical technician, etc.), job responsibilities (e.g., responsible for foundation pit construction rectification, responsible for electromechanical equipment debugging rectification, etc.), work permissions, contact information, terminal equipment identifiers, and other structured data. Personnel information is updated and maintained in real time according to the actual situation of the project. Based on the parsed professional fields and responsibilities, the system initiates a matching process with the organization's personnel database. Through semantic matching algorithms, it filters out personnel whose professional qualifications perfectly match the task's professional field and whose job responsibilities cover the task's scope. If multiple qualified personnel are selected, the system further considers their current workload, such as the number of uncompleted rectification tasks and the urgency of the tasks, as well as the quality of historical rectification completion, such as past rectification pass rates and approval rates, to comprehensively evaluate and determine a single optimal responsible person. This avoids situations where responsibilities are unclear, with multiple people responsible or no one responsible. Once the responsible person is determined, the system automatically generates a task work order containing complete information, including a unique work order number, structured rectification task description, problem location, rectification requirements, specifications, responsible person information, contact information, rectification deadline, associated on-site visual data, and preliminary judgment conclusions. The work order format is standardized and uniform, facilitating subsequent tracking and traceability.

[0099] Step 5.4: Push the task work order to the assigned responsible person's terminal device and continuously track the status of the task work order until the responsible person's feedback on the rectification results is received. Specifically, after the task work order is generated, the system starts the push mechanism and sends the work order notification to the assigned responsible person's terminal device through multiple channels simultaneously. Push methods include pop-up reminders in the multimodal smart assistant application, mobile system notifications, and SMS reminders to ensure that the responsible person can receive the task information in a timely manner and avoid omissions. At the same time, the system automatically records information such as the work order push time, push method, responsible person's terminal device identifier, etc., to form a push log. After the push is completed, the system enters the task work order status tracking stage, monitoring the entire process status flow of the work order in real time. The work order status includes pending receipt, received, in rectification, pending review, and closed loop. Each status change requires the responsible person to confirm the operation through the smart assistant application on the terminal device. The system automatically records key information such as the status change time and the operator. After the responsible person views the work order, the system status is updated to received; after the responsible person clicks to start rectification, the status is updated to in rectification; after the responsible person submits the rectification results, the status is updated to pending review. During the tracking process, if a work order remains in a pending status for an extended period (e.g., exceeding four hours) or is about to expire (e.g., less than twelve hours before the rectification deadline), the system will automatically trigger a secondary reminder, notifying the responsible party again via a more prominent in-app notification or SMS to ensure timely progress of the rectification work. The system continuously tracks the work order status until it receives rectification results information from the responsible party via their terminal device, including post-rectification visual data such as repaired photos and images, and a description of the rectification completion status. At this point, the status is updated to "Pending Review." Throughout the entire process, the system records and stores all data related to the work order status in real time, ensuring that the rectification process is traceable and monitorable.

[0100] In this embodiment of the invention, by employing the technical means of receiving the preliminary judgment conclusion of the anomaly of a specific inspection item and the associated on-site visual data and semantic description, calling a specialized language model to automatically generate a structured rectification task description containing problem location, rectification requirements, and standard basis, matching and automatically assigning a single responsible person from a pre-configured organizational personnel database according to the professional field and scope of responsibility involved in the task, generating a task work order, pushing the work order to the responsible person's terminal device, and continuously tracking the status flow of the task work order until the rectification result is received, the invention overcomes the technical problems in existing solutions, such as non-standard rectification task descriptions, unclear basis, lack of precise matching of responsible persons, and difficulty in real-time tracking of rectification progress, which lead to difficulties in forming an efficient closed loop in the handling of inspection events, unclear responsibility implementation, and inconsistent rectification results. This achieves the technical effect of making rectification tasks more targeted and operable, ensuring accurate and efficient assignment of responsible persons, and making the rectification progress fully controllable, ensuring that responsibility is assigned to individuals and that rectification is carried out in a standardized manner, improving the efficiency and standardization of closed-loop management of inspection events, ensuring timely and effective problem rectification, and further strengthening the quality and safety defense line of engineering construction.

[0101] In a preferred embodiment of the present invention, step 5 above may include:

[0102] Step 5.5: Receive the rectification result information from the responsible person through their terminal device. The rectification result information includes the visual data and text description of the rectified site. Specifically, after completing the on-site rectification work, the responsible person logs into the multimodal intelligent assistant application through their terminal device, finds the corresponding rectification task work order in the personal task center, and enters the work order feedback interface. The system provides a dedicated entry point for uploading rectification result information, supporting the simultaneous submission of visual data and text description of the rectified site. The responsible person needs to upload high-definition photos or complete video materials that clearly show the actual situation after rectification, ensuring that the shooting angle is consistent with the original problem visual data, so that the inspectors can intuitively compare the rectification effect. At the same time, they can supplement the video with images of key links in the rectification process as auxiliary evidence. The text description section needs to fill in the specific measures taken for rectification, such as what materials were used for repair, which standard was followed for rectification, the specific time nodes of rectification construction, whether the actual state after rectification meets the standard requirements, and whether the quality and safety hazards caused by the original abnormal problem have been eliminated. After the responsible person submits the information, the multimodal intelligent assistant's interactive interface receives feedback data in real time and automatically associates the rectification results information with the corresponding task work order to avoid data confusion. At the same time, the system verifies the completeness and validity of the received data, checking whether the visual data is clear and legible and whether the text description contains the core rectification information. If the data is incomplete or does not meet the requirements, the system will send a supplementary prompt to the responsible person through the terminal device, requiring them to make improvements and resubmit. If the data verification passes, it will be stored in the system database and archived along with the original problem record to provide a complete basis for subsequent review work.

[0103] Step 5.6: Push the rectification result information and original problem records to the front-end devices of the inspectors, and call the specialized language model to generate key points prompts to assist in the review. Specifically, after the rectification result information is verified, the system automatically starts the review preparation process, retrieves the original problem records corresponding to the task work order from the database, including complete information such as preliminary judgment conclusions, original on-site visual data, structured rectification task descriptions, and standard basis, and integrates and packages them with the rectification result information reported by the responsible person. Subsequently, through the message push mechanism of the multimodal intelligent assistant, the integrated rectification result information and original problem records are simultaneously pushed to the front-end devices of the inspectors corresponding to the inspection item. The push methods include in-application pop-up reminders, system notifications, etc., to ensure that the inspectors are aware of the tasks to be reviewed in a timely manner. The system automatically calls the specialized language model that was previously trained based on the on-site inspection specialized knowledge base, and starts the process of generating key points prompts to assist in the review. The specialized language model deeply analyzes the abnormal core features in the original problem records, such as abnormal parts, problem types, and violation clauses, and the rectification measures and post-rectification status in the rectification result information, combined with the corresponding inspection targets in the on-site inspection specialized knowledge base. Based on expertise in standards, industry norms, and historical audit cases, this tool generates highly targeted key audit prompts. These prompts include core verification points for original anomalies, such as whether cracks in the support structure are completely closed and whether edge protection reaches the required height; verification points for the compatibility of rectification measures with relevant standards, such as whether the repair materials used meet design requirements and whether the construction process meets industry standards; and verification directions for comparing post-rectification visual data with original data, such as whether the rectified area matches the anomaly area and whether the image clarity is sufficient to prove the rectification effect. This helps inspectors, especially those with limited experience, to quickly grasp the core of the audit and avoid missing crucial audit steps.

[0104] Step 5.7: Receive the audit judgment results made by the inspectors based on the key points prompts. Specifically, after receiving the notification of the task to be audited through the front-end device, the inspectors can click to enter the audit interface to view the complete rectification result information, the original problem record, and the auxiliary audit key points prompts generated by the special language model. The audit interface adopts a column display design, which makes it convenient for the inspectors to compare the original problems and rectification results at the same time. At the same time, the key points prompts are highlighted with prominent marks to guide the inspectors to check one by one according to the key points. Inspectors, drawing on their professional experience and key points, conduct a comprehensive review of the rectification results. They focus on verifying the scientific effectiveness of the rectification measures, whether the post-rectification status fully meets the required standards, and whether the rectification results are genuine and reliable. After review, inspectors use the operation buttons on the review interface to determine the outcome. If the rectification is deemed satisfactory, they select "pass"; if it is deemed unsatisfactory, they select "fail." They then fill in detailed review comments in the corresponding input boxes, clearly pointing out specific issues that were not properly rectified, such as minor cracks remaining in the repaired areas, non-standard spacing of guardrail installation, and key information such as necessary supplementary rectification measures. This provides clear guidance for the responsible party to conduct secondary rectification. After the inspectors submit their review results, the system receives and records the results and corresponding review comments in real time, while also marking the review time and reviewer information to ensure the review process is traceable.

[0105] Step 5.8: If the review result is "passed," the task order status is updated to "closed loop," and the inspection event is marked as "completed." If the review result is "failed," a new rectification task description is automatically generated based on the review comments, and the task order status is reset. A new round of rectification process is then executed until the review is passed. Specifically, after receiving the review result from the inspector, the system immediately initiates the task order status processing flow. If the review result is "passed," the system automatically updates the task order status to "closed loop," marks the corresponding inspection event as "completed" in the system database, and generates a closed-loop management report. The report includes full-process information of the inspection event, such as inspection time, problem discovery, rectification process, and review results, facilitating subsequent review and statistical analysis by project management personnel. If the review result is unsatisfactory, the system automatically triggers a secondary rectification process: First, it retrieves the original problem record of the task order, the rectification result information, and the review comments of the inspectors, and transmits these three documents to a specialized language model. Based on the information and combined with the professional knowledge in the on-site inspection knowledge base, the specialized language model automatically generates a new structured rectification task description. The new description clearly points out the problems that have not been rectified, the rectification measures that need to be optimized, the more detailed execution standards, and the time limit for the second rectification, ensuring that the problems pointed out by the review are addressed in a targeted manner. The system triggers a task order status reset, changing the status from pending review to pending rectification, and updates the task order content to the new structured rectification task description, while retaining the historical flow record of the task order. Finally, the reset task order is pushed back to the terminal device of the original responsible person through a multimodal intelligent assistant, restarting the rectification notification and progress tracking process. The entire process is repeated until the inspectors review and approve the task, ultimately achieving complete closed-loop management of the inspection event.

[0106] In this embodiment of the invention, because it employs the technical means of receiving rectification result information, including on-site visual data and text descriptions after rectification, from the responsible party, pushing the rectification result information and original problem records to the front-end device of the inspectors, and calling a specialized language model to generate key point prompts to assist in the review, the inspectors receive the review judgment result based on the prompts. If the review is approved, the task work order status is updated to closed loop and the inspection event is marked as completed. If the review is not approved, a new rectification task description is automatically generated based on the original problem records and review opinions, the work order status is reset, and the rectification process is re-initiated until the review is approved. Therefore, this overcomes the lack of professional and targeted guidance for inspectors when reviewing rectification results in existing solutions. Inexperienced personnel may overlook key audit points, leading to inaccurate audits. The rectification results may not accurately match the original problems, and the process of resolving unsuccessful rectification is inefficient and lacks a cyclical rectification mechanism. This results in inadequate problem rectification and unclear accountability. Therefore, this approach provides precise and professional audit support for inspectors, improves the accuracy and efficiency of audit judgments, ensures that rectification results accurately match the original problems, enables automatic closed-loop cyclical rectification after unsuccessful rectification, and ensures full traceability of the entire process. This guarantees thorough problem rectification and traceable accountability, further strengthening the rigor and completeness of closed-loop management of inspection events and continuously consolidating the technical effectiveness of engineering construction quality and safety assurance.

[0107] like Figure 2 As shown, embodiments of the present invention also provide an intelligent inspection system for project construction sites based on a multimodal large model, comprising:

[0108] The acquisition module is used to standardize historical patrol event data and build a special knowledge base for on-site patrols to support intelligent analysis.

[0109] The module is used to build a multimodal intelligent assistant with natural language understanding and voice interaction capabilities by using a field inspection knowledge base as the basis for training and inference, integrating generative AI technology and visual analysis algorithms.

[0110] The parsing module is used to receive key project information input by inspectors in natural language through a multimodal intelligent assistant; it parses the key project information through the multimodal intelligent assistant and queries the on-site inspection knowledge base to obtain a list of recommended inspection items that match the key information.

[0111] The judgment module is used to receive on-site visual data uploaded by the inspector when the inspector selects a specific inspection item from the recommended inspection item list. The multimodal intelligent assistant calls the visual analysis algorithm and combines it with the on-site inspection-specific knowledge base to identify and judge the on-site visual data in order to obtain a preliminary judgment on whether there is any abnormality in the inspection item.

[0112] The processing module is used to automatically generate rectification tasks for inspection items identified as abnormal based on the preliminary judgment conclusion, assign them to the corresponding responsible persons, and track the task status. After the responsible persons provide feedback on the rectification results, the inspectors review the results through the multimodal intelligent assistant. If the review is passed, the process of the matter ends. If the review is not passed, the rectification process is restarted until the closed-loop management of the inspection event is achieved.

[0113] In a preferred embodiment of the present invention, the present invention proposes an intelligent assistant for construction site inspection based on a multimodal large model. This system analyzes historical inspection data to construct a specialized knowledge system for on-site inspection. On this basis, it creates a multimodal intelligent auxiliary tool with dimensions such as speech recognition and semantic understanding, providing efficient assistance to on-site inspection personnel and ensuring the quality of construction projects.

[0114] The technical aspects of this invention mainly include a specialized knowledge base and an intelligent assistant, while the application aspects mainly include patrol item recommendation, patrol status judgment, and patrol event management.

[0115] Specialized Knowledge Base: Historical data is processed to form a specialized knowledge base. Intelligent Assistant: Based on the specialized knowledge base, combined with generative AI technology and image analysis algorithms, a multimodal intelligent assistant is created, enabling intelligent interaction through semantics and voice. Inspection Item Recommendation: On-site inspectors only need to input project type, construction stage, etc., in the interactive question-and-answer window via semantics or voice. The intelligent assistant can then match inspection items based on keywords and output content through the interactive window, providing guidance to on-site inspectors. Inspection Status Judgment: For specific inspection items, after uploading on-site inspection photos, the system can identify areas where inspectors do not... To determine if a problem exists, a one-click analysis can be performed. The intelligent assistant, based on its built-in specialized knowledge base and image analysis algorithms, will generate a preliminary judgment to provide a reference for on-site inspectors. For on-site inspection items requiring rectification, tasks can be assigned based on the organizational structure and the intelligent assistant. Relevant personnel will be notified through various means, and they will then carry out on-site rectification. After rectification, a response can be provided, which will be reviewed by the inspectors. If the review is successful, the inspection event is closed; otherwise, it is returned to the rectification personnel for further processing, thus achieving closed-loop management of inspection events.

[0116] In a preferred embodiment of the present invention, on-site inspectors log in to the intelligent assistant via a mobile device, click on the recommended inspection items, and enter the intelligent interactive interface. The system prompts them to enter keywords such as project type and construction stage. Inspectors can input information in the interactive window, choosing either text or voice input, such as entering "foundation pit project" or "foundation pit excavation stage." The intelligent assistant will output corresponding inspection items, such as whether the support structure is complete, whether the support system is deformed, and whether edge protection has been set. Simultaneously, the system will continue to provide prompts, such as the depth of the foundation pit and the type of support. Inspectors can obtain further information according to their needs. During the inspection, clicking on "On-site Inspection Report" enters the reporting interface. This interface includes information such as the current stage, inspection type, inspector, whether rectification is required, and description. Clicking the "Add" option allows for information editing. When filling in the description, image upload is supported, and a one-click analysis button is provided. Clicking this button will automatically analyze the data and provide preliminary judgment results to assist inspectors in making accurate judgments.

[0117] During inspections, it is necessary to determine whether the inspected items require rectification. If the determination is yes, the task can be assigned in the operation column after the item is filled in, and the corresponding responsible person can be selected. After the responsible person is selected, the rectification process begins. After the responsible person completes the rectification work, the corresponding information must be submitted to the inspector for review. If the review is approved, the process ends; if the review is not approved, the process enters a loop to achieve closed-loop management of on-site inspections and effectively ensure construction quality.

[0118] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent on-site inspection of project construction based on a multimodal large model, characterized in that, The method includes: Standardize the historical inspection event data and build a special knowledge base for on-site inspections to support intelligent analysis; Using a specialized knowledge base for on-site inspections as the foundation for training and reasoning, we integrate generative AI technology and visual analysis algorithms to build a multimodal intelligent assistant with natural language understanding and voice interaction capabilities. The system receives key project information input by inspectors in natural language through a multimodal intelligent assistant; it also analyzes the key project information and queries the on-site inspection knowledge base to obtain a list of recommended inspection items that match the key information. Based on the recommended list of inspection items, when an inspector selects a specific inspection item, the system receives the on-site visual data uploaded by the inspector. The multimodal intelligent assistant calls the visual analysis algorithm and combines it with the on-site inspection-specific knowledge base to identify and judge the on-site visual data in order to obtain a preliminary judgment on whether there are any abnormalities in the inspection item. Based on the preliminary assessment, for inspection items deemed abnormal, a multimodal intelligent assistant automatically generates rectification tasks, assigns them to the relevant responsible persons, and tracks the task status. After the responsible persons provide feedback on the rectification results, the inspection personnel review the results through the multimodal intelligent assistant. If the review is successful, the process ends; if the review fails, the rectification process is restarted until closed-loop management of inspection events is achieved.

2. The intelligent on-site inspection method for project construction based on a multimodal large model according to claim 1, characterized in that, Historical patrol event data is standardized to construct a specialized on-site patrol knowledge base to support intelligent analysis, including: It receives and aggregates archived inspection event data from multiple historical project construction sites, including text and image records. The collected patrol event data is cleaned and standardized to obtain a structured set of historical patrol data. Based on the historical inspection data set, inspection items, related scenarios, problem types and handling rules are extracted as key features to obtain an initial inspection knowledge graph; The initial inspection knowledge graph was reinforced and logically verified to ultimately build a specialized on-site inspection knowledge base to support intelligent analysis.

3. The intelligent on-site inspection method for project construction based on a multimodal large model according to claim 2, characterized in that, Using a specialized knowledge base for on-site inspections as the foundation for training and inference, generative AI technology and visual analysis algorithms are integrated to construct a multimodal intelligent assistant with natural language understanding and voice interaction capabilities, including: Load the completed on-site inspection knowledge base, and based on the on-site inspection knowledge base, perform domain-adaptive fine-tuning on the pre-trained generative large language model to obtain a specialized language model with semantic understanding and generation capabilities in the construction site domain. Based on a specialized knowledge base for on-site inspections, a pre-trained visual analysis model is trained under supervision to build a specialized visual analysis model capable of identifying specific scenarios and problems at the construction site. It integrates specialized language models, specialized visual analysis models, and speech recognition and synthesis modules, and encapsulates and deploys them into a multimodal intelligent assistant service with multimodal input and output and intelligent interaction capabilities.

4. The intelligent on-site inspection method for project construction based on a multimodal large model according to claim 3, characterized in that, The system receives key project information input by inspection personnel in natural language through a multimodal intelligent assistant. The multimodal intelligent assistant analyzes key project information and queries the on-site inspection knowledge base to obtain a list of recommended inspection items matching the key information, including: The system receives voice input streams containing project type and construction stage from the front-end devices of the inspection personnel through the interaction interface of the multimodal intelligent assistant. The voice input stream is parsed in a multimodal manner. If it is speech, it is converted into text by the speech recognition module, and the structured semantic features of key project information are extracted by a specialized language model. Using structured semantic features as query conditions, semantic retrieval and matching are performed in the on-site inspection knowledge base to recall relevant candidate inspection items; For each candidate inspection item, the items are sorted and filtered according to the relevance of their structured semantic features, and a list of recommended inspection items is output to the front-end device of the inspection personnel.

5. The intelligent inspection method for project construction sites based on a multimodal large model according to claim 4, characterized in that, Based on the recommended inspection item list, when an inspector selects a specific inspection item, the system receives the on-site visual data uploaded by the inspector. The multimodal intelligent assistant invokes visual analysis algorithms and combines them with a specialized on-site inspection knowledge base to identify and judge the on-site visual data, in order to obtain a preliminary judgment on whether the inspection item has any anomalies, including: For specific inspection items selected by the patrol personnel in the list of recommended inspection items on the front-end device, receive and load the on-site visual data associated with the specific inspection items uploaded; A specialized visual analysis model is invoked to extract features and identify targets from the visual data on site, in order to obtain visual analysis results that include scene elements and potential problems; Based on the semantic description of the current specific inspection item, retrieve the corresponding inspection standards and historical problem cases from the on-site inspection special knowledge base; By combining the visual analysis results with the sub-inspection standards and historical cases, and using a specialized language model for fusion reasoning, a preliminary judgment on whether the inspection item is abnormal is obtained and output.

6. The intelligent inspection method for project construction sites based on a multimodal large model according to claim 5, characterized in that, Based on the preliminary assessment, for inspection items deemed abnormal, a multimodal intelligent assistant automatically generates rectification tasks, assigns them to the relevant responsible persons, and tracks the task status, including: Receive the preliminary judgment conclusion generated for a specific inspection item and determined to be abnormal, along with the associated on-site visual data and semantic description; Based on the preliminary judgment and semantic description, a specialized language model is invoked to automatically generate a structured description of the rectification task, which includes the problem location, rectification requirements and normative basis. Based on the professional fields and responsibilities involved in the structured rectification task description, a match is made in the pre-configured organizational personnel database, and a single responsible person is automatically assigned and a task work order is generated. The task work order is pushed to the terminal device of the assigned person in charge, and the status of the task work order is continuously tracked until the rectification results are received from the person in charge.

7. The intelligent inspection method for project construction sites based on a multimodal large model according to claim 6, characterized in that, After the responsible party reports the rectification results, the inspection personnel review the results through the aforementioned multimodal intelligent assistant. If the review is successful, the process ends; if the review fails, the rectification process is restarted until closed-loop management of the inspection event is achieved, including: Receive rectification result information from the responsible party through their terminal device. The rectification result information includes on-site visual data and text descriptions after rectification. The rectification results and original problem records are pushed to the front-end devices of the inspectors, and a special language model is called to generate key points prompts to assist in the review. Receive the audit and judgment results made by the inspection personnel based on key points and suggestions; If the review result is approved, the task work order status is updated to closed loop, and the inspection event is marked as completed; if the review result is not approved, the review comments are automatically recorded to generate a new rectification task description, and the task work order status is reset to re-execute a new round of rectification process until the review is approved.

8. A project construction site intelligent inspection system based on a multimodal large model, wherein the system implements the method as described in any one of claims 1 to 7, characterized in that, include: The acquisition module is used to standardize historical patrol event data and build a special knowledge base for on-site patrols to support intelligent analysis. The module is used to build a multimodal intelligent assistant with natural language understanding and voice interaction capabilities by using a field inspection knowledge base as the basis for training and inference, integrating generative AI technology and visual analysis algorithms. The parsing module is used to receive key project information input by inspectors in natural language through a multimodal intelligent assistant; The multimodal intelligent assistant analyzes key project information and queries the on-site inspection knowledge base to obtain a list of recommended inspection items that match the key information. The judgment module is used to receive on-site visual data uploaded by the inspector when the inspector selects a specific inspection item from the recommended inspection item list. The multimodal intelligent assistant calls the visual analysis algorithm and combines it with the on-site inspection-specific knowledge base to identify and judge the on-site visual data in order to obtain a preliminary judgment on whether there is any abnormality in the inspection item. The processing module is used to automatically generate rectification tasks for inspection items identified as abnormal based on the preliminary judgment conclusion, assign them to the corresponding responsible persons, and track the task status. After the responsible persons provide feedback on the rectification results, the inspectors review the results through the multimodal intelligent assistant. If the review is passed, the process of the matter ends. If the review is not passed, the rectification process is restarted until the closed-loop management of the inspection event is achieved.

9. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images