A natural language instruction interaction system and method for BIM software

Abstract

The application belongs to the technical field of BIM and AI, and particularly relates to a natural language instruction interaction system and method for BIM software. The system comprises: a plug-in layer running in a BIM software process and calling an SDK interface; a desktop application layer providing interactive input in the form of a WebSocket client as an independent process; and an intelligent agent layer as a central relay hub and a WebSocket server. The desktop application layer and the plug-in layer establish bidirectional decoupling communication through the intelligent agent layer, and the intelligent agent layer performs cross-process message asynchronous relay forwarding according to the matching result of the source and target client identification carried in the message body. The intelligent agent layer uses a Supervisor supervisor module to split tasks, and controls the assignment of a group of sub-Agent modules to cooperate with a Skills skill module and a double-channel retrieval enhancement generation retrieval module to execute tool calling and double-channel concurrent retrieval. The application realizes zero-invasion safe decoupling and data localization sandbox privacy isolation of heavy heterogeneous engineering processes.

Description

Technical Field

[0001] This invention relates to the fields of Building Information Modeling (BIM) and Artificial Intelligence (AI), specifically to a natural language command interaction system and method for BIM software. Background Technology

[0002] Building Information Modeling (BIM) technology has been widely applied in architectural design, construction management, and operation and maintenance. Mainstream BIM software such as Autodesk Revit, ArchiCAD, and Bentley provide efficient design and management tools for the entire construction industry by creating 3D digital models. Meanwhile, artificial intelligence (AI) technologies, represented by large language models, have developed rapidly. Their capabilities in natural language understanding, code generation, and knowledge retrieval offer a new technological path for the intelligent transformation of the construction industry. Against this industry backdrop, how to deeply integrate the natural language processing capabilities of large language models with BIM software to enable control of BIM software via natural language commands has become a current research hotspot.

[0003] However, existing natural language command interaction integration solutions have revealed significant shortcomings in practical applications. Some solutions employ a tightly coupled design, directly calling the large language model interface within the building information model software plugin, mixing the AI's reasoning logic with the software's operational logic within the same code module. This tightly coupled architecture results in extremely poor maintainability of the entire system; switching the underlying large language model provider necessitates modifying the plugin code and recompiling and redeploying it. Furthermore, with the daily updates and upgrades of the building information model software, the compatibility maintenance costs of the plugins are extremely high, and it is impossible to independently upgrade and iterate core AI capabilities such as knowledge retrieval enhancement systems or skill systems without affecting the normal operation of the building information model software.

[0004] Existing integrated interaction solutions generally suffer from insufficient data security and privacy protection. Most cloud service solutions typically mandate the uploading of high-density Building Information Modeling (BIM) data to the cloud for processing. Since architectural design documents often involve core intellectual property, uploading them to third-party cloud platforms poses a serious risk of trade secret leakage. Furthermore, sensitive documents such as project-specific specifications and internal meeting minutes cannot be used for AI-assisted decision-making while ensuring security. Simultaneously, critical sensitive credentials such as application programming interface keys, if stored on remote servers, also face a high risk of credential leakage.

[0005] Furthermore, some solutions, in pursuit of deep control, opt for modifications or extensive customization of the Building Information Modeling (BIM) software itself. Such invasive modifications not only easily violate software license agreements but also severely disrupt software stability and significantly interfere with normal design workflows. Moreover, due to the underlying logic differences between various BIM software platforms, invasive solutions often require separate development for each platform, resulting in extremely poor system versatility. More critically, existing general-purpose AI assistants lack the specialized cognitive capabilities required for BIM scenarios. They cannot understand the complex semantic structures of building components such as walls, doors, windows, and floors within the building model, nor can they accurately execute software operation commands such as creating components, modifying parameters, or generating reports. Therefore, they cannot truly provide designers with an efficient and real-time natural language-assisted design experience.

[0006] In summary, the industry urgently needs a novel technical solution that can seamlessly integrate intelligent agent interaction capabilities for architectural scenarios into existing design workflows without modifying the building information modeling (BIM) software. This solution needs to achieve secure and efficient two-way data interaction between the intelligent agent and desktop BIM software, support complete local deployment to ensure all core architectural data remains on the local machine, and, while allowing large language model providers to switch flexibly, possess powerful local knowledge retrieval enhancements and scalable skill-driven capabilities. Summary of the Invention

[0007] The purpose of this application is to provide a natural language command interaction system and method for BIM software to solve the above-mentioned technical problems.

[0008] Firstly, this application proposes a natural language command interaction system for BIM software, the system comprising: The plug-in layer runs within the BIM software process and extends functionality non-intrusively through the publicly available SDK interface of the BIM software. It is used to call the SDK interface to execute BIM operation commands issued by the intelligent agent layer. The desktop application layer is an independent desktop application process that connects to the WebSocket server module of the smart agent layer via the Agent protocol in the form of a WebSocket client, and is used to provide interactive input of natural language commands. The intelligent agent layer is an independent backend service process. As the only WebSocket server for the natural language command interaction system for BIM software, it maintains long connections with both the plug-in layer and the desktop application layer to carry message routing and intelligent agent orchestration and scheduling of the large language model. The desktop application layer and the plug-in layer establish bidirectional decoupled communication based on the Agent protocol and the plug-in protocol through the intelligent agent layer, which acts as a relay hub. According to the matching result of the source client identifier and the target client identifier carried in the message body by the intelligent agent layer, the intelligent agent layer performs cross-process bidirectional asynchronous message relay forwarding. The target client identifier is the unique identity of the desktop application layer that initiates the interaction request, and the source client identifier specifically refers to the unique identity of the plug-in layer that receives the interaction request and performs the operation.

[0009] In the above technical solution, a single, independent intelligent agent process is used as the central star-shaped relay hub of the entire network topology. A message queue maintains a full-duplex connection at both ends, preventing project collapse and freezing caused by frequent direct penetration or intrusion of external commands into the BIM core process. By using asynchronous bidirectional mapping and matching between source client identifiers and target client identifiers, the technical challenges of asynchronous multi-process lifecycles and difficulty in matching heterogeneous data formats when large multimodal intelligent agents face heavy local BIM software interactions are solved.

[0010] Furthermore, the plugin layer includes a communication module, a command execution module, and a connection management module, among which: The communication module and the intelligent agent layer establish a long-term communication connection via a plug-in protocol; The command execution module is used to parse the received BIM operation commands and call the SDK interface to perform specific operations in the BIM software; The connection management module is responsible for connection registration, heartbeat keep-alive, and abnormal reconnection.

[0011] In the above technical solution, not only is the memory stability of the host BIM software itself protected, but it also ensures that even if the large model proxy layer crashes and restarts unexpectedly, the host BIM software plugin can still remain resident and silently reconnect according to the preset step size, thus comprehensively improving the robustness of industrial-grade plugins.

[0012] Furthermore, the desktop application layer includes: a user interface module, an agent communication module, a dialogue history storage module, and an agent discovery module, among which, User interface module, used to provide a natural language dialogue interaction interface; The Agent communication module is used to connect to the intelligent agent layer and encapsulate and transmit Agent protocol signaling. The dialogue history storage module is used to persistently store historical interaction records to a local embedded relational database; The Agent discovery module is used to detect agreed-upon port files, perform port scans, and implement automatic start and stop control of backend services.

[0013] The aforementioned technical solution integrates graphical user operation, underlying Agent long signaling encapsulation, local lightweight storage, and probe self-starting service into a single lightweight desktop client that eliminates the need for manual user intervention. By leveraging the Agent discovery module's inherent self-probing and self-starting / stopping closed loop, the cumbersome engineering operation of manually launching the backend algorithm service in a black window is completely avoided. Its local embedded storage mechanism ensures that the foreground and background interaction records have the non-volatile characteristics of power-off data, greatly reducing the risk of context loss during sudden power outages in edge computing.

[0014] Furthermore, the user dialogue requests sent from the desktop application layer to the intelligent agent layer are chat messages, and the chat messages are transmitted in JSON format for front-end driver configuration. Chat message fields include: message type field, chat history message list, large language model configuration object, skill list array, RAG document directory path field, permanent file directory path field, temporary file absolute path list array, and optional memory configuration object containing sliding window parameters and summary trigger threshold; The intelligent agent layer does not perform local persistent storage of configuration parameters in chat messages.

[0015] In the above technical solution, by adopting a unified, strictly aligned field-formatted pure in-memory JSON chat message, large model orchestration objects and massive heterogeneous document paths, as well as temporary and permanent directory tree pointers, are all merged into a single message session packet, achieving a typical purely dynamic operation mechanism. By enforcing a "non-persistent storage" rule at the proxy layer, it is ensured that when the dialogue task ends or the session garbage collection mechanism is triggered, all sensitive metadata related to building entities, user preferences, and paths is discarded as needed in the proxy layer's memory, completely eliminating the security risks such as internal leaks or ransomware attacks that may be caused by long-term information residence at the physical storage layer.

[0016] Furthermore, the internal fields of the large language model configuration object include: large language model API key, API service address, model name, generation temperature parameter, maximum number of generated tokens, and a boolean field used to identify whether the thinking reasoning mode is enabled; Each element in the skill list array has the following internal fields: skill name, skill description, a boolean field indicating whether an executable script is included, a boolean field indicating whether a reference file is included, and the absolute path of the corresponding skill directory on the local machine.

[0017] In the aforementioned technical solution, by explicitly defining the temperature, generation constraints, and thinking mode fields at the underlying level of the high-dimensional large model, the system possesses precise control over the LLM divergence capability when generating accurate BIM operation commands and generating broad divergent dialogues. Simultaneously, by clearly defining the four golden legal boundary fields—name, description, script, and absolute path—in the skill array, the system ensures that the underlying heterogeneous plug-in packages possess an industrial-grade, unambiguous semantic self-describing protocol foundation during dynamic loading.

[0018] Furthermore, the intelligent agent layer includes a WebSocket server module, a message routing and forwarding module, a Supervisor module, and a group of sub-Agent modules: The WebSocket server module is used to maintain long-lived connections with various heterogeneous clients, handle connection registration and security authentication, and heartbeat keep-alive. The message routing and forwarding module is used to match and perform cross-process bidirectional message relay forwarding based on the source client identifier and the target client identifier in the local memory routing table; The Supervisor module is used to perform intent recognition, task splitting, parallel and serial scheduling, multi-source result data aggregation and streaming output, and execution failure rollback control on natural language input from the desktop application layer by leveraging the semantic parsing capabilities of large language models. The sub-Agent module group is under the centralized command and control of the Supervisor module, and performs deep reasoning and tool invocation under a specific professional execution chain; The sub-Agent module group includes a Building Information Model Interaction Sub-Agent, a Dual-Channel Retrieval Enhanced Generation Retrieval Sub-Agent, a Skill Execution Sub-Agent, and a General Dialogue Sub-Agent.

[0019] In the aforementioned technical solution, a perfect layered distributed control architecture is established in the intelligent agent layer of the system. This architecture consists of "WebSocket server handling authentication and keep-alive, message forwarding acting as a multi-process overpass, Supervisor implementing overall scheduling, and multiple sub-agent groups dedicated to specific tasks." By centrally controlling the four highly heterogeneous sub-agents—construction, retrieval, execution, and dialogue—high-efficiency decoupling is achieved for extremely complex engineering scenarios. This greatly alleviates the performance problems of "forgetting at both ends" and inference crashes that plague single models when facing highly mixed and long contexts.

[0020] Furthermore, the intelligent agent layer also includes a dual-channel retrieval enhancement generation module and a Skills module; The dual-channel retrieval enhancement generation retrieval module is used to perform vectorized parsing and storage of public building code documents and combine them with user-attached local files in real time. The Skills module acts as a dynamic tool factory, dynamically reading and instantiating structured extended skill files directly from the local machine. The collaborative processing mechanism utilizing the Supervisor module and the sub-Agent module group includes: In the intent recognition step, the Supervisor module uses a large language model to classify the user's input natural language commands to identify the corresponding task type. The task types include: BIM operation tasks, knowledge retrieval tasks, skill execution tasks, general dialogue tasks, and comprehensive tasks involving the linkage of multiple modules. In the task splitting step, if the identified task type is a comprehensive task, the Supervisor module will dynamically split the comprehensive task into a sequence of sub-tasks with logical sequential or parallel relationships. In the parallel-serial scheduling steps, the Supervisor module performs differentiated scheduling based on the dependencies within the sub-task sequence, and adopts the following strategy when the scheduling sub-Agent module group executes tool calls: In response to knowledge retrieval subtasks in the subtask sequence that do not involve BIM software operations, the dual-channel retrieval enhancement generation agent calls the dual-channel retrieval enhancement generation retrieval module to perform dual-channel concurrent retrieval in the public knowledge base set of the Chromadb vector database and the temporary vector set in the running memory, and returns the retrieved clause specification text as the execution result of the knowledge retrieval subtask to the Supervisor module. In response to BIM operation subtasks involving BIM software operations in the subtask sequence, the Building Information Model Interaction Agent uses the Skills module to dynamically instantiate and execute the corresponding BIM inspection skill based on the absolute path field of each element in the skill list array, and returns the inspection operation result as the execution result of the BIM operation subtask to the Supervisor module. In the result aggregation step, after each sub-Agent has finished executing, the Supervisor module dynamically splices and aggregates the execution results of each sub-task in the sub-task sequence into unified context data according to the dependency relationship and sequence. It then calls the large language model to perform integration and polishing processing based on the context data and streams the final dialogue response back to the desktop application layer through block messages and completion status messages. For independent subtasks, the Python asyncio.gather asynchronous concurrency mechanism is used to execute the parallel dispatch of multiple sub-agents; for subtasks with causal dependencies, the await asynchronous waiting and suspension mechanism is used to execute the serial dispatch of multiple sub-agents in sequence.

[0021] The aforementioned technical solution creatively implements a hybrid dual-channel retrieval enhancement mechanism that combines "public, large-scale industry static specifications" with "temporary private documents generated by users' drag-and-drop operations." During scheduling, differentiated process control is implemented for "non-operational knowledge retrieval," which has extremely uneven time consumption, and "BIM inspections involving physical models." High- and low-efficiency parallel asynchronous concurrent processing technology is used to saturate independent subtasks with concurrent processing, while causal subtasks are suspended. Finally, the Supervisor acts as the endpoint, dynamically aggregating, refining, and concurrently segmenting multi-source results before sending them to the front end. This results in a zero-latency, high-accuracy composite interactive experience within an extremely complex industrial knowledge graph and BIM action chain.

[0022] Furthermore, the intelligent agent layer also includes a short-term memory module, which is used to perform context pruning and automatic compression and merging of the dialogue history message list at runtime based on the sliding window parameters and the summary trigger threshold. Short-term memory optimization algorithms that combine sliding window and digest calculations using short-term memory modules include: The short-term memory module receives the input list of dialogue history messages and the optional memory configuration object, and performs a threshold determination on the number of messages in the dialogue history message list or the current context length; If the value is less than the first preset threshold, the original message list is returned directly and the process ends; if the value is not less than the first preset threshold, a status determination is performed on the dialogue history message list to determine whether it contains tool call messages or tool call markers. If it is determined that the message contains the tool call message or tool call flag, it is identified as an uneditable intermediate state, the original message list is returned directly, and the process ends; if it is determined that the message does not contain the tool call message or tool call flag, the keyword overlap between the current question message and the historical messages is calculated. If the keyword overlap is determined to be less than the set threshold, the current question is determined to be a new topic, only the current question message input is retained and the process ends; if the keyword overlap is determined to be not less than the set threshold, a second threshold determination is performed on the context length or number of messages related to the current topic to determine whether it exceeds the second preset threshold. If the threshold is not exceeded, the original message list is returned and the process ends. If the threshold is exceeded, the sliding window pruning and summary triggering judgment are performed in combination with the parameters N and M in the optional memory configuration object: the most recent N messages are retained as the sliding window inference context. When the number of messages in the waiting-to-be-summarized area outside the sliding window reaches M, the large language model is called to perform dialogue key point summary compression on the old messages outside the sliding window, generating a summary message with a preset number of characters and specific attribute identifiers. The summary message and some active messages in the sliding window are finally combined and retained in the running memory, and the waiting-to-be-summarized area is cleared after generation to end the process.

[0023] In the aforementioned technical solution, strong protection is implemented when facing extremely sensitive "intermediate states" such as tool calls, ensuring that the large model will not suffer semantic deficiencies or logical breaks in complex tool execution loops. By directly "truncating and clearing" when a completely new topic is identified, and by implementing a collaborative mechanism of sliding window and old message accumulation summary in the original topic, the context length of the input large model throughout its entire lifecycle is guaranteed to always remain within a preset, extremely low range. This not only eliminates the technical weakness of frequent errors in large models from the source, but also brings about a precipitous drop in end-to-end computing power token overhead and large model inference latency.

[0024] Furthermore, the dual-channel retrieval enhancement module includes a public knowledge base channel, a local document channel, and a retrieval and merging unit: The public knowledge base channel includes a document importer, a local text embedding model, and a Chromadb vector database. The document importer is used to perform asynchronous parsing, MD5 deduplication, and text chunking for document indexing tasks. The local text embedding model converts the chunked text into semantic vector data, which is then stored in the Chromadb vector database in isolation according to the knowledge base identifier. The local document channel includes a document parser, a text chunker, and a temporary vector set. The document parser and text chunker are used to perform real-time parsing and chunking of the user's temporary file list, which is constructed by the user dragging and dropping local files from the desktop application layer. The temporary vector set is vectorized and stored in the temporary vector set through a local text embedding model. The temporary vector set is dynamically created in the runtime memory with the session ID of the current Agent session as the namespace to achieve sandbox-style security isolation from the common knowledge base set in the Chromadb vector database. The temporary vector set is deleted and shredded at the end of the current Agent session lifecycle. The retrieval and merging unit includes a retrieval unit and a context merger. The retrieval unit is called by a retrieval agent generated by dual-channel retrieval enhancement, which performs dual-channel concurrent retrieval on the common knowledge base set in the Chromadb vector database and the temporary vector set in runtime memory. The context merger is used to deduplicate and sort the retrieved multi-source text paragraphs by relevance, and returns the merged and optimized execution results to the Supervisor module to call the large language model to perform inference.

[0025] In the above technical solution, by isolating document indexing, deduplication, and chunking logic from the Chromadb public library, especially in the local document channel, using the unique ID of the current session as a dynamic namespace, an insurmountable "physical privacy and security sandbox" is created for each natural language interaction session. Data temporarily dragged by the user is only retrieved within its own dynamic memory vector set, achieving physical isolation from the public library sandbox. After the specific completion command given by the large dialogue model is sent, data cleanup without residue is performed, ensuring that no third-party user or backend system can peek into the user's private engineering drawings and model metadata from the historical garbage heap.

[0026] Furthermore, the process of dynamically registering and invoking tools within the Skills module includes: Step A: Parse multiple structured extended skill files from the skill list array of chat messages; Step B involves iterating through the parsed skill list array and dynamically creating and instantiating multiple corresponding structured tool instances in the runtime memory of the intelligent agent layer. Step C involves injecting structured tool instances into the sub-Agents corresponding to the functions of the sub-Agent module group according to the tool descriptions. This allows the sub-Agent module group to perceive all available skills and autonomously select and call them based on the orchestration instructions of the Supervisor module. Specifically, BIM operation-related tools are injected into the Building Information Model Interaction Sub-Agent, retrieval tools are injected into the Dual-Channel Retrieval Enhanced Generation Retrieval Sub-Agent, and a general toolset is injected into the General Dialogue Sub-Agent. Step D: If a local directory path is determined, the Skills module directly reads the corresponding structured extended skill file content from the absolute path field of the skill list array; if no local directory path is determined, the WebSocket server module asynchronously sends a skill request to the desktop application layer and receives the structured extended skill file content returned by the desktop application layer. Step E involves injecting the read structured extended skill file content into the corresponding structured tool instance, so that the Building Information Model Interaction Sub-Agent can be instantiated and execute the BIM inspection skill.

[0027] In the above technical solution, only the skill list array needs to be parsed in runtime memory. A dynamic tool factory iterates through the array and instantiates structured tool objects in memory. These objects are then automatically categorized based on their self-description and fed to four heterogeneous sub-agents, achieving adaptive layered infusion of BIM, retrieval, and general toolchains. This is complemented by a dual-track network / local direct-read relay based on the presence or absence of local files determined by the absolute path field. Ultimately, all skill content retrieved via any path is merged and injected back into the structured tool instances, driving the sub-agents to perform actions. This enables the system to achieve hot-swappable and hot-updating of the skill library without stopping any services.

[0028] Furthermore, the system is also configured with a full-link data and communication security and privacy protection mechanism, which includes: The key protection mechanism ensures that the user's API key is stored in encrypted persistent storage and is only dynamically read when sending dialogue messages to the large language model. After the call is completed, the corresponding memory reference is immediately released and writing to the runtime log is refused. The model data localization mechanism keeps the user's model files in the memory space of the running BIM software process. When the plug-in layer receives BIM operation commands, it calls the SDK interface to read the corresponding component physical properties and transmits them unidirectionally to the intelligent agent layer and desktop application layer through a local loopback WebSocket connection for use-and-discard processing. The connection verification mechanism involves the WebSocket server module performing a validity check on the authentication token carried in the connection request during the initial handshake when establishing a WebSocket connection between the desktop application layer, the smart proxy layer, and the plugin layer. If the verification fails, the connection path is forcibly closed.

[0029] In the above technical solution, key leakage is eradicated through asymmetric non-volatile encryption of the key, high-strength protection by reading memory only at the moment of dialogue call and then releasing memory references in a "use-and-discard" manner, combined with technical means to completely cut off log retention. Model data is transmitted unidirectionally through local loopback long connections, and the proxy layer does not persist or store data on disk, directly locking the heavy building design model in the local BIM main process. Combined with strong verification of random tokens during the initial long connection establishment and an exclusive IP address mechanism to reject public networks, a hard-core physical firewall is constructed from the memory, process data, and network long connection layers.

[0030] Furthermore, a dual-end streaming interaction mechanism is configured between the desktop application layer and the intelligent agent layer. This mechanism includes connection establishment steps, dialogue interaction steps, and heartbeat keep-alive steps. In the connection establishment process, the desktop application layer initiates a WebSocket connection request to the smart proxy layer, and after receiving a successful connection establishment response from the smart proxy layer, sends an authentication message containing an authentication token to the smart proxy layer. The WebSocket server module then performs a validity check on the authentication token. In the dialogue interaction step, the desktop application layer responds to the user's natural language input by sending a chat message to the intelligent agent layer; the following asynchronous communication branches are configured in the dialogue interaction step: The streaming output branch, the intelligent agent layer consumes the output events of the large language model in a streaming manner, and pushes block messages to the desktop application layer in real time through the message routing and forwarding module; In the skill loading branch, in response to the sub-Agent needing to schedule external extended skill tools and determining that there is no local directory path, the intelligent agent layer asynchronously sends a skill request message to the desktop application layer and receives a skill response message returned by the desktop application layer containing the content of the structured extended skill file. The permission confirmation branch responds to the confirmation mode triggered by the invoked extended skill tool. The smart agent layer asynchronously sends a permission confirmation request message containing the request identifier and tool name to the desktop application layer, and receives a permission confirmation response message returned by the desktop application layer containing a boolean field for approval or denial. When the interaction ends, the intelligent agent layer pushes a completion message to the desktop application layer, which includes a reply text. The heartbeat keep-alive step involves the desktop application layer periodically sending heartbeat request messages to the intelligent agent layer and receiving heartbeat response messages from the intelligent agent layer to achieve state awareness and keep-alive of the long-connection link between the two ends.

[0031] The aforementioned technical solution defines a fine-grained timing mechanism for long-connection authentication and activation, asynchronous concurrent processing of multiple communication branches, and full-cycle heartbeat maintenance. An asynchronous generator is used to display the output events of the large language model on a token-by-token (word-by-word) front-end, coupled with asynchronous skill delivery and loading when there is no path, a front-end pop-up permission boolean value feedback mechanism when the highest security permissions are required, and a tail-end inference Done message push. This significantly reduces user anxiety during long-duration inference in heavy industrial scenarios, resulting in a silky-smooth interface text display, while providing a secure audit timing environment that can be interrupted at any time and is controllable.

[0032] Furthermore, the message routing and forwarding module is configured with a plugin connection approval and full lifecycle message routing mechanism, which includes: During the connection verification phase, in response to the connection message initiated by the BIM plugin in the plugin layer, the message routing and forwarding module registers the source client identifier carried in the connection message into the local memory routing table and changes the state of the BIM plugin. When in the permission mode that requires user approval, the state of the BIM plugin is marked as pending approval, and the plugin event message is forwarded one-way to the desktop application layer through the WebSocket server module to trigger the interactive interface to pop up the pending approval prompt, suspend the connection thread and wait for the desktop application layer to send back the user approval decision. During the channel activation phase, in response to being in automatic activation mode or receiving an approval instruction from the desktop application layer, the message routing and forwarding module constructs a plugin event response containing an approval flag and sends it back to the BIM plugin to formally activate the full-duplex bidirectional communication channel; in response to receiving a rejection instruction from the desktop application layer, it constructs a connection response message containing the rejection reason and sends it back to the BIM plugin, and the WebSocket server module forcibly closes the connection path. During the heterogeneous message routing phase, while the bidirectional communication channel is active, in response to the BIM operation sub-tasks issued by the Supervisor module, the message routing and forwarding module forwards the BIM command forwarding message of the BIM operation command to the desktop application layer. After encapsulation into a standard plug-in protocol command message, it is precisely routed to the target BIM plug-in through the message routing and forwarding module to trigger the BIM software to perform inspection operations; and During the result recall phase, the message routing and forwarding module receives the command result message returned after the target BIM plugin has completed execution, and recalls and forwards it to the desktop application layer according to the source client identifier field. The desktop application layer then encapsulates it into a command result feedback message and sends it back to the Supervisor module.

[0033] In the above technical solution, by registering the initiated connection in a routing table and dynamically suspending the thread according to the permission mode, and waking up the desktop layer through event push for approval and interception, the threat of illegal operations from unknown plugins is directly blocked before the connection is established. During normal communication, by routing the subtasks split from the Supervisor to the desktop through the relay hub to complete the UI context and encapsulate it into a standard plugin format, then accurately locating and sending it to the target BIM plugin, and finally recalling it along the original path according to the source identifier, a closed-loop control is achieved, realizing real-time asynchronous bidirectional message precise relay for heavy-duty BIM clients in cross-language and cross-process scenarios.

[0034] Secondly, this application proposes a natural language command interaction method for BIM software, which applies the natural language command interaction system for BIM software described in the first aspect. The method includes: S1, the desktop application layer encapsulates the received natural language instructions into chat messages, and sends the chat messages to the intelligent agent layer through the Agent communication module of the desktop application layer; S2, the Supervisor module in the intelligent agent layer receives the chat message, performs intent recognition and task splitting to generate at least one sub-task and assign it to the corresponding sub-Agent, and the sub-agent generates the corresponding BIM operation command according to the intent; S3 sends BIM operation commands to the desktop application layer through the message routing and forwarding module of the intelligent agent layer. After being assembled into standard plug-in protocol command messages, the message routing and forwarding module matches them in the local memory routing table according to the source client identifier carried in the message body, and performs cross-process bidirectional asynchronous message relay forwarding to the plug-in layer. The plug-in layer parses and calls the SDK interface to perform specific operations. S4, the plugin layer encapsulates the specific operation execution result into a command result message and returns it. The message routing and forwarding module recalls and forwards it to the desktop application layer based on the source client identifier. The desktop application layer encapsulates it into a command result return message and returns it to the Supervisor module based on the target client identifier. The Supervisor module performs the result aggregation step to call the large language model for inference and generate a dialogue response, and pushes the dialogue response to the desktop application layer for front-end display.

[0035] In the above technical solution, natural language commands are parsed into chat messages and sent across layers (S1). At the agent layer, Supervisor splits the tasks into sub-tasks and distributes them to heterogeneous agents, driving the generation of official BIM operation commands (S2). Then, the intelligent agent layer acts as a relay hub, distributing the commands to the desktop to complete the context. Following the source client identifier representing the plugin layer, the results are routed to the plugin layer to extract the local SDK driver (S3). Finally, the results are retrieved along the original path and sent back to Supervisor based on the target client identifier to complete the aggregation and polishing of multi-source result data, and finally, the data is streamed (S4). This method establishes a standardized, closed-loop, cross-process software process with absolutely aligned identity identifiers throughout its entire lifecycle, providing extremely high coverage and effectiveness in infringement determination.

[0036] Furthermore, prior to step S1, the following steps are also included: The desktop application layer launches and triggers its Agent discovery module, which searches for or automatically launches the intelligent agent layer through a fallback strategy. The fallback strategy includes: Port file detection: Read the agreed port file to obtain the process identifier and communication port number of the smart agent layer, check the liveness status of the process corresponding to the process identifier, and send a health check request to the communication port number. If no live smart agent layer process is found in the port file, a health check request is sent to the default port. Port scanning: If no response is received by the default port scan, perform a port scan within the preset port range and send a health check request to each port. Automatic startup: If the port scan does not find a usable communication port, the independent backend service process of the smart proxy layer will be automatically started at the operating system level, and health check requests will be continuously sent until the smart proxy layer activates the WebSocket server module after passing the health check.

[0037] In the above technical solution, before processing natural language interaction, the desktop client automatically activates the Agent discovery module and uses the method of "reading the agreed port file to find the PID and port liveness". If a file becomes invalid, it will switch to the default port to perform a health check. If the default fails, perform a full port multi-level scan within the preset range. In the event of a final failure, an independent backend process is launched directly at the system kernel level and continuously monitored until the system is healthy, by activating WebSocket. This four-stage degradation and adaptive disaster recovery strategy, with its four progressive layers of self-recovery, greatly reduces the probability of software crashing due to unexpected environmental events or port conflicts, providing end users with seamless, industrial-grade, fully automated, and highly available operational assurance.

[0038] Compared with the prior art, the beneficial results of the present invention are as follows: (1) This application overturns the limitations of traditional BIM secondary development tools, which must be strongly dependent on and deeply embedded in the host software process. By separating the intelligent proxy layer running in an independent backend service process and using it as the only WebSocket server relay hub in the entire system, it maintains full-duplex long-term connections with the desktop application layer process and the plug-in layer process respectively, thus creating a cross-process bidirectional asynchronous message relay forwarding mechanism. The plug-in layer communicates non-intrusively only through the SDK interface exposed by the BIM software, which not only completely eliminates the engineering risks of the host BIM software main thread freezing or crashing due to the time consumption of long context reasoning of large models or network fluctuations, but also achieves physical-level decoupling of the three-layer heterogeneous architecture. Each layer can independently perform algorithm framework iteration, UI visual optimization or host version adaptation, which greatly reduces the maintenance cost and evolution complexity of the system.

[0039] (2) This application establishes a rigorous localized non-intrusive extraction and disposable control loop for heavy building engineering models and key assets. The user's model files are locked in the memory space of the local BIM software process, and the extraction of component attributes is entirely carried out locally through a loopback localhost long connection for unidirectional transmission; the core large model API key is encrypted and persistently stored and dynamically read only at the moment the conversation is sent, and the memory reference is immediately released after the call and is not written to the log. More importantly, in the local document channel, the system dynamically creates a temporary vector set in the running memory with the unique ID of the current session as the namespace, which physically realizes sandbox-style security isolation between temporary files and the Chromadb public knowledge base set, and performs a thorough deletion and shredding cleanup of the temporary vector set at the end of the session, ensuring that sensitive engineering drawings and corporate privacy are never permanently stored or leaked.

[0040] (3) This application introduces a sub-Agent module group scheduling architecture centrally commanded and controlled by the Supervisor module. By transforming the high-dimensional semantic parsing capability of the large language model into structured intent recognition and task decomposition, the Supervisor module can dispatch different professional sub-Agents using differentiated Python asynchronous concurrent or serial suspension mechanisms for complex natural language instructions. The system innovatively performs dual-channel concurrent retrieval and multi-source result data aggregation on the "public knowledge base channel storing common design specifications" and the "local temporary document channel isolated based on the current session ID sandbox," so that the final generated dialogue response has both the high real-time physical attributes of the BIM model and the support of rigorous industry clauses. Combined with the streaming feedback of block messages and completion status messages, the response speed and accuracy in comprehensive and complex engineering scenarios such as BIM compliance review are greatly improved.

[0041] (4) This application configures an Agent discovery module at the desktop application layer, which performs a multi-tiered adaptive scan of the port file, the preset default port, and the preset port range before the system starts. If no available service is found, an independent backend process is automatically started at the system level. This progressive degradation strategy provides the system with a strong, highly available, and self-healing capability that requires no manual maintenance. At the same time, the built-in Skills module acts as a dynamic tool factory, directly parsing the skill list array in the running memory and instantiating it into structured tools in seconds. These tools are then categorized and injected into the corresponding sub-Agents. Combined with the network relay / local direct read mechanism that automatically switches based on the absolute path field, the system can achieve dynamic hot-plugging, hot updating, and LLM autonomous awareness and invocation of external BIM extended skill tools without modifying any hard-coded code or restarting the service. This perfectly achieves a fully automatic, highly elastic, and highly scalable intelligent interactive closed loop. Attached Figure Description

[0042] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Many anticipated advantages of the embodiments and other embodiments of the invention will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0043] Figure 1 This is a framework diagram of a natural language command interaction system for BIM software according to an embodiment of this application; Figure 2 This is a flowchart of a short-term memory optimization algorithm combining a sliding window and a summary in a natural language command interaction system for BIM software according to an embodiment of this application. Figure 3 This is a knowledge retrieval and merging process topology diagram of a dual-channel retrieval enhancement generation retrieval module in a natural language command interaction system for BIM software according to an embodiment of this application; Figure 4 This is a lifecycle flowchart of the common knowledge base channel document indexing task management state machine of the dual-channel retrieval enhancement generation retrieval module according to an embodiment of this application; Figure 5 This is a complete business process diagram of the Skills module from dynamic registration to multi-agent collaborative invocation in a natural language command interaction system for BIM software according to an embodiment of this application; Figure 6 This is a timing diagram of the dual-end streaming interaction between the desktop application layer and the intelligent agent layer in a natural language command interaction system for BIM software according to an embodiment of this application. Figure 7 This is a business process diagram of the plug-in connection approval and message routing lifecycle in a natural language command interaction system for BIM software according to an embodiment of this application; Figure 8 This is a deployment extension diagram of a natural language command interaction system for BIM software according to an embodiment of this application; Figure 9 This is a business process diagram of a natural language command interaction method for BIM software according to an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0045] refer to Figure 1 , Figure 1 A framework diagram of a natural language command interaction system for BIM software according to an embodiment of this application is shown. As shown in the figure, the system includes: The first layer, the plug-in layer, runs within the Building Information Modeling (BIM) software process. It extends functionality through the publicly available Software Development Kit (SDK) interfaces of the BIM software without modifying the original BIM code. Specifically, this plug-in layer includes: a communication module that establishes a long-lived communication connection with the WebSocket server module in the third-layer intelligent agent layer via a plug-in protocol, acting as a WebSocket client; a command execution module that parses received BIM operation commands and calls the SDK application programming interface to execute specific operations within the BIM software; and a connection management module responsible for connection registration, heartbeat keep-alive, and abnormal reconnection. The bidirectional asynchronous data interaction between the plug-in layer and the intelligent agent layer adopts a standardized plug-in protocol, which defines three core message types: connection messages, command messages, and command result messages.

[0046] The second layer, the desktop application layer, is an independent desktop application process. It connects to the third layer's intelligent agent layer's WebSocket server module via the Agent protocol in the form of a WebSocket client. It runs only as a client and does not start a local WebSocket server. It indirectly establishes non-intrusive, bidirectional, decoupled communication with the plugin layer through the message routing and forwarding module in the intelligent agent layer. Specifically, the desktop application layer includes: a user interface module providing a natural language dialogue interface and system parameter settings; an Agent communication module connecting to the third layer's unique WebSocket server and encapsulating and transmitting Agent protocol signaling; a dialogue history storage module persistently storing historical interaction records in a local embedded relational database; and an Agent discovery module for detecting agreed-upon port files, performing port scans, and automatically controlling the start and stop of backend services.

[0047] The third layer, the intelligent agent layer, is an independent backend service process. As the sole WebSocket server within the entire command and interaction system, it provides unified message access, precise cross-process routing, and asynchronous message forwarding capabilities. It also supports centralized intelligent agent scheduling and command, as well as various localized AI support capabilities specific to the building industry. Specifically, the intelligent agent layer includes: a WebSocket server module for maintaining long-lived connections with various heterogeneous clients and handling connection registration and security authentication; a message routing and forwarding module for matching and executing precise cross-process forwarding based on source and target client identifiers in a local memory routing table; a Supervisor module for using the semantic parsing capabilities of a large language model to perform intent recognition, task decomposition, independent sub-agent scheduling, and multi-source result data aggregation and streaming output on the natural language input from the second layer; and a group of sub-Agent modules that follow the centralized command and control of the Supervisor module and perform deep reasoning and tool calls within a specific professional execution chain. The sub-Agent module group includes a Building Information Model Interaction Sub-Agent, a Dual-Channel Retrieval Enhancement Generation Retrieval Sub-Agent, a Skill Execution Sub-Agent, and a General Dialogue Sub-Agent. In addition, the intelligent agent layer includes a dual-channel retrieval enhancement module for vectorizing and storing public building code documents and combining them with user-attached local files in real time; a Chromadb vector database providing public knowledge base index support; a local text embedding model that runs local vectorized encoding extraction; a permission governance module that acts as a firewall and executes asynchronous user confirmation decisions based on risk level classification; a short-term memory module that performs context pruning and automatic compression and merging of the dialogue history message list at runtime based on sliding window parameters and summary trigger thresholds; and a Skills module that acts as a dynamic tool factory, directly reading and instantiating structured extended skill files from the local machine. The Supervisor module and the sub-Agent module group independently initiate network requests to the external large language model cloud service through application programming interfaces to obtain global task scheduling decision support and dynamic tool-aware invocation capabilities for specific execution chains, respectively.

[0048] In some specific embodiments, communication between the plug-in layer and the intelligent agent layer adopts a plug-in protocol. Its message encapsulation format uses a unified JSON format, including message type, message body, message sending timestamp, and source and target client identifiers appended by the message routing and forwarding module during relay forwarding (see Table 1 for details). The plug-in protocol defines three message types: connection message, command message, and command result message. The connection message is used for connection establishment and authentication between the plug-in layer and the intelligent agent layer; its internal data body structure is detailed in Table 2. The command message is used by the desktop application layer to issue BIM operation commands to the plug-in layer after routing / forwarding via the Agent server; its internal data body structure is detailed in Table 3. The command result message is used by the plug-in layer to return the operation execution result after routing and forwarding via the Agent server; its internal data body structure is detailed in Table 4.

[0049] Table 1. Field Definitions in Message Encapsulation Format

[0050] Table 2. Definition of the data field in connection messages

[0051] Table 3. Definition of the data field in command messages

[0052] Table 4. Definition of the data field in command result messages

[0053] In some specific embodiments, the user dialogue request sent from the desktop application layer to the intelligent agent layer is a chat message, the specific field definitions of which are detailed in Table 5. The chat message fields include a message type field with a fixed value of "chat", a list of dialogue history messages consisting of a role and content, a large language model configuration object `llm_config`, a skills list array `skills`, a RAG document directory path field used to override the default configuration, a permanent file directory path field, an array of absolute paths for temporary files, and a sliding window parameter. and summary trigger threshold Optional memory configuration object.

[0054] Table 5. Field Definitions in Chat Messages

[0055] Furthermore, the sub-objects and nested arrays carried in the chat message are defined as follows: The specific field definitions of the large language model configuration object llm_config are detailed in Table 6. Its fields include the large language model API key, API service address, model name, generation temperature parameter with a default value of 0.7, maximum number of generated tokens with a default value of 2048, and a boolean field used to identify whether the thinking reasoning mode is enabled. The specific field definitions of each element in the skills list array are detailed in Table 7. The fields include skill name, skill description, a boolean field to indicate whether an executable script is included, a boolean field to indicate whether a reference file is included, and the absolute path of the corresponding skill directory on the local machine.

[0056] Table 6 Field definitions of the llm_config object

[0057] Table 7 defines the fields for each element in the skills array.

[0058] In addition to the chat messages transmitted by the front-end during the conversation phase, other interactive signaling exists between the desktop application layer and the intelligent agent layer throughout their lifecycle. Other Agent protocol message types are detailed in Table 8. As shown in Table 8, Agent protocol message types, based on actual business scenarios, include connection authentication and keep-alive, AI model streaming response, heterogeneous terminal relay and event notification, and asynchronous capability extension and security defense. Among these, connection authentication and keep-alive includes authentication messages (auth) sent by the desktop application layer after connection establishment, carrying a pre-configured authentication token, and heartbeat messages (ping / pong) sent periodically by the desktop application layer and responded to by the intelligent agent layer. AI model streaming response includes streaming token messages (chunk) pushed by the intelligent agent layer token-by-token to AI-generated content, reasoning messages (reasoning) pushed when thinking mode is enabled, and completion messages (done) marking the completion of the dialogue response and carrying the complete text. The Heterogeneous End Relay and Event Notification class includes plugin event messages (plugin_event) pushed by the intelligent agent layer for plugin connection, disconnection, and approval result events; BIM command forwarding messages (bim_command) generated and sent to the target plugin by the BIM interaction sub-Agent (i.e., Building Information Modeling interaction sub-Agent); command result feedback messages (bim_command_result) that report back the results after the plugin has completed execution; and system notification messages (system) that notify tool calls or changes in the scheduling status of the Supervisor sub-Agent. The Asynchronous Capability Extension and Security Defense class includes skill content request and response messages (skill_request / skill_response) that perform asynchronous association matching based on the request identifier (requestId); permission confirmation request and response messages (permission_request / permission_response) used to obtain user confirmation before high-risk tools are executed; and error messages (error) returned by the intelligent agent layer with error codes and error information when exceptions occur.

[0059] Table 8 Other Agent Protocol Message Types

[0060] In the communication mechanism defined by the Agent protocol, in addition to the chat messages mentioned above, the desktop application layer and the intelligent agent layer also achieve bidirectional asynchronous interaction through multiple sets of specific message types. After the WebSocket connection is established, the desktop application layer first sends an authentication message (auth) carrying a pre-configured authentication token to the intelligent agent layer. If authentication fails, the intelligent agent layer performs a connection closure operation. In the subsequent dialogue phase, the intelligent agent layer pushes AI-generated content to the desktop application layer one by one through streaming token messages (chunks). When the reasoning mode is enabled, it synchronously pushes the AI's reasoning process through reasoning messages (reasoning). At the same time, the intelligent agent layer uses system notification messages (system) to notify the desktop application layer of the current tool call and Supervisor sub-Agent scheduling status changes, and sends a completion message (done) containing complete text and optional reasoning content when the dialogue response is complete. For heterogeneous collaboration scenarios, the intelligent agent layer pushes plugin event messages (plugin_event) containing event types and client metadata to the desktop application layer. After the BIM interaction sub-agent generates BIM operation commands, it sends BIM command forwarding messages (bim_command) to the desktop application layer. The corresponding plugin execution results are encapsulated by the plugin layer, relayed by the intelligent agent layer, and finally returned to the supervisor module of the intelligent agent layer via command result feedback messages (bim_command_result) by the desktop application layer for further inference. In addition, the two parties asynchronously associate and interact with skill content request and response messages (skill_request / skill_response) and permission confirmation request and response messages (permission_request / permission_response) before high-risk tool execution based on the request identifier (requestId). The intelligent agent layer returns error messages (error) containing error codes and error information to the desktop application layer. At the same time, the system also maintains a keep-alive mechanism of heartbeat messages (ping / pong) sent periodically by the desktop application layer and responded to by the intelligent agent layer.

[0061] In some specific embodiments, the Agent discovery module in the desktop application layer performs automatic discovery of the intelligent agent layer through a four-level degradation strategy to establish cross-process connections. The four-level degradation strategy specifically includes: Port file detection: The Agent discovery module reads the agreed port file to obtain the process identifier and communication port number of the intelligent agent layer, checks the liveness status of the process corresponding to the process identifier, and sends a health check request to the communication port number; Default port detection: If the port file detection does not find a live intelligent agent layer process, the Agent discovery module sends a health check request to the default port 8000; Port scanning: If the default port detection does not receive a response, the Agent discovery module performs a port scan in the range of ports 8000 to 8010 and sends a health check request to each port; Automatic startup: If the port scan does not find an available communication port, the Agent discovery module automatically starts the independent backend service process of the intelligent agent layer at the operating system level and continuously sends health check requests until the intelligent agent layer passes the health check and formally establishes a WebSocket long connection.

[0062] In some specific embodiments, reference is made to Figure 2 , Figure 2 This diagram illustrates a flow topology of a short-term memory optimization algorithm combining a sliding window and a summary in a natural language command interaction system for BIM software, according to an embodiment of this application. Figure 2 As shown, the execution flow of the short-term memory optimization algorithm includes: The message optimization function of the short-term memory module receives an input list of dialogue history messages, including an optional memory configuration object (memory_config) with parameters N and M. At runtime, it performs a threshold check on the number of messages in the dialogue history message list or the current context length using preset criteria, determining if it is less than a first preset threshold. If the number of messages or the current context length is less than the first preset threshold, a first direct return branch is activated, directly returning the original message list and ending the short-term memory optimization algorithm. If the number of messages or the current context length is not less than the first preset threshold, a state check is further performed on the dialogue history message list to determine if it contains tool call messages. If the dialogue history message list contains tool call messages or tool call markers with the role attribute "tool", it is identified as an unpruned large language model inference intermediate state, a second direct return branch is activated, directly returning the original message list, preserving the tool call messages as they are, and ending the process. If the system determines that no tool call messages are included, it proceeds to the topic relevance quantification stage. Using well-known algorithms such as TF-IDF, Jaccard similarity, or text cosine similarity, it calculates the keyword overlap between the current question and historical messages to verify topic relevance. If the overlap is less than 15%, failing to meet the topic relevance condition, the current question is considered a new topic, the topic-irrelevant branch is activated, only the current question message is retained, and the process ends. If the overlap is not less than 15%, the current question is considered relevant to the historical dialogue topic. At this point, a second threshold judgment is performed on the context length or number of messages related to the current topic to determine if it exceeds a second preset threshold. If it does not exceed the second preset threshold, a third direct return branch is activated, directly returning the original message list and ending the process. If it exceeds the second preset threshold, the system combines parameters N and M to perform sliding window pruning and summary trigger judgment. Before pruning, the first message with the role attribute "user" in the dialogue history message list is extracted and marked as a message to be retained to maintain the initial semantics of the task. Then, the most recent N messages are retained by default as the sliding window inference context, where the default value of N is 6. When the number of messages exceeding the summarization area of ​​the sliding window reaches M, the large language model summarization generation mechanism is triggered, where the default value of M is 8. The large language model summarization generation mechanism compresses the dialogue key points of the old messages exceeding the sliding window by calling the large language model, generating a system summary message of no more than 150 characters with the attribute defined as role="system". Finally, the system summary message and the four most recent messages in the sliding window are combined and retained in runtime memory, and the summarization area is cleared after generation. This completes the dynamic pruning and summary loop optimization for short-term memory. Both parameters N and M can be dynamically configured through the optional memory configuration object (memory_config) in the message body.

[0063] In some specific embodiments, the Supervisor module and the sub-Agent module group achieve intelligent task scheduling and result aggregation through a collaborative processing mechanism; the collaborative processing mechanism includes: In the intent recognition step, the Supervisor module uses a large language model to perform deep intent classification on the natural language commands input by the user in order to accurately identify the corresponding task type. The task types include: BIM operation tasks, knowledge retrieval tasks, skill execution tasks, general dialogue tasks, and comprehensive tasks involving the linkage of multiple modules.

[0064] In the task decomposition step, when the identified task type is a comprehensive task, the Supervisor module dynamically decomposes the complex comprehensive task into a sequence of sub-tasks with logical sequence or parallel relationship; for example, the "BIM compliance inspection" task is decomposed into: a knowledge retrieval sub-task executed by the dual-channel retrieval enhancement generation retrieval sub-agent, a BIM operation sub-task executed by the building information model interaction sub-agent, and a comparative analysis sub-task executed by the large language model.

[0065] In the parallel-serial scheduling steps, the Supervisor module performs differentiated scheduling based on the dependencies within the sub-task sequence, and adopts the following strategy when the scheduling sub-Agent module group executes tool calls: In response to knowledge retrieval subtasks in the subtask sequence that do not involve BIM software operations, the dual-channel retrieval enhancement generation agent calls the dual-channel retrieval enhancement generation retrieval module to perform dual-channel concurrent retrieval in the public knowledge base set of the Chromadb vector database and the temporary vector set in the running memory, and returns the retrieved clause specification text as the execution result of the knowledge retrieval subtask to the Supervisor module. In response to BIM operation subtasks involving BIM software operations in the subtask sequence, the Building Information Model Interaction Agent uses the Skills module to dynamically instantiate and execute the corresponding BIM inspection skill based on the absolute path field of each element in the skill list array, and returns the inspection operation result as the execution result of the BIM operation subtask to the Supervisor module. For independent subtasks, the Python asyncio.gather asynchronous concurrency mechanism is used to execute the parallel dispatch of multiple sub-agents; for subtasks with causal dependencies, the await asynchronous waiting and suspension mechanism is used to execute the serial dispatch of multiple sub-agents in sequence.

[0066] In the result aggregation step, after each sub-Agent has completed its execution, the Supervisor module dynamically concatenates and aggregates the execution results of each sub-task in the sub-task sequence into unified context data according to dependencies and sequence. Then, it calls the large language model again to perform final integration, comparative analysis, and polishing based on the unified context data. Finally, it streams the final dialogue response back to the desktop application layer via chunk messages and done status messages. For failure rollback and fault tolerance, during scheduling, if the target sub-Agent returns a clear error message, the error result is directly fed back to the front-end user. If the target sub-Agent experiences a timeout and no response, an exception log is automatically recorded and a retry mechanism is triggered, with a maximum of two retries. If the process still fails after multiple retries, the Supervisor module activates a partial degradation generation strategy, calling the large language model to generate a "partial answer" based on the currently successfully acquired context data, and clearly marking the incomplete parts that were not completed due to the exception in the generated dialogue response.

[0067] In some specific embodiments, reference is made to Figure 3 , Figure 3 This document illustrates a knowledge retrieval and merging process topology diagram for a dual-channel retrieval enhancement generation retrieval module in a natural language command interaction system for BIM software, according to an embodiment of this application. (For example...) Figure 3As shown, the dual-channel retrieval enhancement generation retrieval module processes data through a shared knowledge base channel and a local document channel that operate in isolation and in parallel. In the shared knowledge base channel, the administrator submits a document indexing task containing enterprise public documents, building codes, or design standards to the system. The document importer performs asynchronous parsing, MD5 deduplication, and text segmentation. The BGE local text embedding model is then used to extract vectorized encoded text from the segmented text, and the generated semantic vector data is stored in the Chromadb vector database, isolated according to the knowledge base identifier. During runtime, the dual-channel retrieval enhancement generation retrieval agent calls the corresponding retrieval tool to retrieve relevant specification text fragments. The local document channel allows users to drag and drop local files to build a temporary file list via the desktop application layer. The dual-channel retrieval enhancement generation module in the intelligent agent layer receives these files and performs real-time parsing on PDF, Word, Excel, or Markdown files using a document parser. It then uses a text segmenter to perform chunking. The segmented text is also vectorized and extracted using the BGE local text embedding model. A temporary vector set is dynamically created in memory for the current Agent session, automatically cleaned up at the end of the conversation, and stored in real-time until the conversation ends. The retrieval and merging unit includes a retrieval unit and a context merger. When a user initiates a conversation through the user interface module, the dual-channel retrieval enhancement generation retrieval sub-agent receives the user's question and performs a dual-channel concurrent retrieval using the built-in retrieval unit on both the public knowledge base set in the Chromadb vector database and the temporary vector set in runtime memory. The retrieved multi-source text paragraphs are deduplicated and sorted by relevance using the context merger, and the final merged and optimized execution result is returned to the Supervisor module to invoke the large language model for inference.

[0068] For details, please refer to Figure 4 , Figure 4 A lifecycle flowchart of the common knowledge base channel document indexing task management state machine of the dual-channel retrieval enhancement generation retrieval module according to an embodiment of this application is shown. Figure 4As shown, the lifecycle of document indexing task management in the public knowledge base channel includes: After the administrator executes the user-submitted document operation through the document management interface, the document importer dynamically creates an indexing task record in the background database. At this time, the initial state of the indexing task record is marked as pending. The data dictionary fields of the indexing task record specifically include task ID, document path, MD5 hash, knowledge base identifier, creation time, current status, and error information. Subsequently, the system executes an asynchronous task start operation. The asynchronous task queue extracts the target task from the tasks in the pending state and updates the current status field of the target task in real time to the indexing state. During the indexing process, the corresponding document parser is dynamically matched based on the file extension of the file to be processed to extract the plain text content. The MD5 hash value of the plain text content is then calculated for incremental comparison and deduplication. If the same hash value already exists in the specified public knowledge base set corresponding to the Chromadb vector database, it is skipped. Otherwise, the text segmenter is invoked to perform recursive character segmentation. When segmenting text, Chinese punctuation marks ".", "!", "?", and ";" are prioritized as cutting boundaries, limiting the size of a single text block to no more than 1000 characters, and setting the overlap area between adjacent text blocks to 200 characters. After segmentation, the BGE local text embedding model is invoked to perform vectorized encoding extraction to generate 768-dimensional semantic vector data, which is then persistently stored in the specified set of the Chromadb vector database. If the above vectorization storage is successful, the indexing is considered successful, and the current status field of the indexing task record is updated to the completed state. If an exception occurs during any stage of parsing, chunking, deduplication, or vectorized storage, the indexing is deemed a failure. The current status field of the indexing task record is set to "failed," and the captured exception stack trace is written to the error information field. This allows the user to manually trigger a re-indexing operation on the front end, changing the task status back from "failed" to "indexing." Furthermore, regardless of whether the indexing task record is currently in a pending, indexing, completed, or failed state, the user can perform a deletion operation on the front end to directly clear the corresponding indexing task record and terminate its entire lifecycle.

[0069] In some specific embodiments, reference is made to Figure 5 , Figure 5 This document illustrates a complete business process diagram of the Skills module in a natural language command interaction system for BIM software, according to an embodiment of this application, from dynamic registration to multi-agent collaborative invocation. As shown in the figure, the business process diagram includes: The first phase involves skill configuration and element assembly. In the system's skill directory structure, each independent extended skill corresponds to a separate skill directory on the local disk, and each skill directory contains a structured extended skill file that serves as the core driver. When the desktop application starts, the background asynchronously scans and reads the corresponding local skill directory, assembling and encapsulating it into the skill list array of the chat message according to the Agent protocol. The specific fields within each element of the skill list array are limited to: skill name, skill description, a boolean field indicating whether an executable script is included, a boolean field indicating whether a reference file is included, and the absolute path of the corresponding skill directory on the local machine.

[0070] The second stage involves the instantiation and injection of dynamic tool factories. Upon receiving a chat message, the intelligent agent layer's Skills module, acting as a dynamic tool factory, parses multiple structured extended skill files from the skill list array in the chat message. It then iterates through the skill list array, dynamically creating and instantiating multiple corresponding structured tool instances based on the metadata of each skill element. These are represented as Structured Tool A, Structured Tool B, and Structured Tool C, respectively. After instantiation, these structured tool instances are categorized and injected into the corresponding sub-Agents within the sub-Agent module group. This allows the sub-Agent module group to perceive all available skills and autonomously select and invoke them based on the orchestration instructions from the Supervisor module.

[0071] The third stage involves autonomous selection and path branching. During execution scheduling, structured tool instances are categorized and injected into the corresponding sub-Agents within the sub-Agent module group. Specifically, the Building Information Modeling (BIM) interaction sub-Agent is injected with specific model inspection skills tools related to BIM operations; the dual-channel retrieval enhanced generation retrieval sub-Agent is injected with retrieval tools related to specification merging; and the skill execution sub-Agent is injected with a set of general extended skill tools containing executable scripts. Based on the sub-task intents derived from the Supervisor module, the corresponding sub-Agents within the sub-Agent module group autonomously select and invoke a specific skill tool. At runtime, the Skills module performs conditional attribute checks on the metadata of this skill tool, specifically determining whether the skill tool has a valid local directory path.

[0072] The fourth stage involves a dual-track reading mechanism and result aggregation output. If a local directory path is determined, the local direct read branch is used. The Skills module directly reads the corresponding structured extended skill file content from the local skill directory (i.e., the absolute path field of each element in the skill list array). If no local directory path is determined (e.g., in cloud or remote lightweight deployment scenarios), the network relay branch is used. The WebSocket server module asynchronously sends the corresponding skill request to the desktop application layer and receives the structured extended skill file content returned by the desktop application layer. Ultimately, regardless of which branch is used, the read structured extended skill file content is injected into the corresponding structured tool instance for the Building Information Modeling (BIM) interaction agent to dynamically instantiate and execute the BIM inspection skill. The inspection operation results are then returned to the Supervisor module for result aggregation. The corresponding sub-agents in the sub-agent module group perform deep inference based on the skill workflow instructions and generate professional sub-task results. Finally, the Supervisor module aggregates the outputs of multiple sub-agents and streams them to the desktop application layer for display, thus completing the complete lifecycle loop of the Skills module from registration to invocation.

[0073] In some specific embodiments, the permission governance module acts as a firewall, performing asynchronous user confirmation and permission decisions based on risk level classification. Specifically, the permission governance module is configured with three levels of tool risk levels and three permission modes; the three tool risk levels are: low risk level, corresponding to read-only operation tools; medium risk level, corresponding to external network request tools; and high risk level, corresponding to file write operation tools. The three permission modes are: automatic mode, configured to allow all tools to execute directly; denial mode, configured to deny all tool calls; and confirmation mode, configured to determine whether user confirmation is required based on the tool's risk level. The permission decision-making process of the permission governance module is as follows: If the system is in automatic mode, all tool calls are allowed directly; if the system is in denial mode, all tool calls are denied directly and the denial result is fed back to the large language model; if the system is in confirmation mode, the risk level of the called tool is queried. If it is determined to be low-risk, it is allowed directly; if it is determined to be medium-risk or high-risk, a permission request message is triggered and sent to the desktop application layer through the WebSocket server module, so that the front end can pop up a confirmation dialog box and receive the request identifier and whether to allow boolean field returned by the desktop application layer through the permission response message after the user selects; if no response is received within the preset time limit of 60 seconds, the permission governance module will automatically deny the request.

[0074] In some specific embodiments, the natural language command interaction system for BIM software is also configured with a full-link data and communication security and privacy protection mechanism, which specifically includes: API key encryption storage and a memory-based disposable mechanism. User API keys are persistently stored in an encrypted field of a local relational database using an asymmetric encryption algorithm, and are protected by the underlying hardware-level security provided by the current operating system's credential management interface. API keys are dynamically read only when the intelligent agent layer sends dialogue messages to the large language model, and the corresponding memory reference is immediately released after the intelligent agent layer completes its call to the large model interface, refusing to write API keys to any runtime logs or perform persistent storage. Furthermore, the Supervisor module forcibly destroys all sub-Agent instances and their corresponding configuration parameters within the current session's lifecycle after each dialogue task is completed, cutting off key leakage paths at the physical memory level.

[0075] A localized, non-intrusive extraction mechanism for Building Information Modeling (BIM) data. The user's model files are always locked within the memory space of the running BIM software process, refusing any form of network transmission. The plug-in layer running within the BIM software process only passively calls the SDK interface to read the corresponding component physical attributes when it receives BIM operation commands, and then transmits them unidirectionally to the intelligent proxy layer and desktop application layer via a local loopback WebSocket long connection (localhost). The intelligent proxy layer, acting as a relay hub, performs on-demand processing of component physical attributes within the process, refusing to persistently store any BIM data on the backend.

[0076] Local loopback exclusive listening and connection token verification mechanism. The three-layer WebSocket connection built between the desktop application layer, the smart proxy layer, and the plugin layer is strictly limited to listening on the network IP address 127.0.0.1, rejecting any connection requests from external networks. Upon initial startup, a globally unique authentication token is automatically and randomly generated in the local configuration file. During the initial handshake of subsequent three-layer WebSocket long connections, the WebSocket server module in the smart proxy layer performs a validity check on the authentication token carried in the connection request. If the check fails, the connection path is immediately and forcibly closed.

[0077] Local document privacy sandbox isolation and timed cleanup mechanism. The local document channel of the dual-channel retrieval enhancement generation module in the intelligent agent layer is passively activated only in response to a user-initiated drag-and-drop attachment of a local file, refusing to automatically scan the user's operating system file system. When parsing local documents, a temporary vector set is dynamically created in runtime memory using the session ID of the current Agent session as the namespace, achieving physical sandbox-like security isolation between it and the public knowledge base set in the vector database. When the large language model determines that the dialogue has ended and sends a completion message, the temporary vector set and all temporary vector data stored within it are immediately and automatically deleted and shredded, ensuring that user privacy is not persistent. Specifically, the temporary vector set is released by calling the deletion interface of the Chromadb vector database, ensuring that temporary data is not persistently stored, thus protecting user privacy from persistent storage.

[0078] Continue to refer to Figure 6 , Figure 6 This diagram illustrates a two-way streaming interaction sequence between a desktop application layer and an intelligent agent layer in a natural language command interaction system for BIM software according to an embodiment of this application. As shown, the two-way streaming interaction includes three stages: During the connection establishment phase, the desktop application layer first initiates a WebSocket connection request to the smart proxy layer. After receiving a successful connection establishment response from the smart proxy layer, the desktop application layer sends an authentication message (auth authentication message) containing an authentication token to the smart proxy layer. The WebSocket server module in the smart proxy layer performs a validity check on the authentication token and sends an authentication pass message back to the desktop application layer after the check passes. At this point, the long connection path between the two ends is officially activated.

[0079] During the dialogue interaction phase, the desktop application layer responds to the user's natural language input by sending a chat message (chat message) to the intelligent agent layer containing a list of dialogue history messages, a large language model configuration object, and a skill list array. Subsequently, the system enters three parallel or cyclical asynchronous communication branches: a streaming output loop (loop), where the intelligent agent layer uses an asynchronous generator to stream the output events of the large language model and pushes chunk messages to the desktop application layer in real-time, token-by-token, for front-end text stream display; and an optional branch for tool invocation and skill loading (opt): when a sub-agent needs to schedule an external extended skill tool and determines that there is no local directory path, the intelligent agent layer first sends a system notification message (system notification) indicating that a tool is being invoked to the desktop application layer, and asynchronously sends a skill request message (skill_request) containing a request identifier, skill name, and skill description; the desktop application layer then reads the corresponding structured extended skill file locally. Following the content, the same request identifier and structured extended skill file content are sent back to the intelligent agent layer via a skill response message (skill_response). The optional branch for tool invocation and permission confirmation (opt) occurs when the invoked extended skill tool triggers the confirmation mode in the permission mode. The intelligent agent layer, acting as a firewall, asynchronously sends a permission confirmation request message (permission_request) containing the request identifier, tool name, and invocation parameters to the desktop application layer. The desktop application layer displays a confirmation dialog box on the front end and waits for user action. After the user selects, a permission confirmation response message (permission_response) is sent back to the intelligent agent layer with the same request identifier and a boolean value field indicating approval or rejection. At the end of this stage of the streaming interaction, the intelligent agent layer pushes a completion message (done) to the desktop application layer. The message body of the completion message explicitly carries the complete response text and optional inference content fields.

[0080] During the heartbeat keep-alive phase, a periodic heartbeat detection loop is entered during the long connection maintenance. The desktop application layer periodically sends heartbeat request messages (ping messages) to the intelligent proxy layer and receives heartbeat response messages (pong messages) returned in real time by the intelligent proxy layer, so as to realize the state awareness and keep-alive of the long connection link between the two ends.

[0081] In some specific embodiments, reference is made to Figure 7 , Figure 7 This paper illustrates a business process diagram of the plug-in connection approval and message routing lifecycle in a natural language command interaction system for BIM software according to an embodiment of this application. As shown in the figure, the plug-in connection approval and message routing implementation includes: Step 701: The BIM plugin in the plugin layer is started.

[0082] Step 702: Initiate a connection request to the WebSocket server in the intelligent proxy layer. The BIM plugin, as a WebSocket client, establishes an underlying physical connection with the WebSocket server module in the intelligent proxy layer.

[0083] Step 703: Send a connection message. Specifically, the BIM plugin assembles registration information containing plugin metadata according to the plugin protocol and sends it unidirectionally to the intelligent agent layer as a connection message. The message format of the connection message is limited by specific key-value pairs. Its type field is configured as connection, the action field is configured as connect, the client_id field used to carry the source client identifier is configured as UUID v4, and it carries the metadata field containing its own plugin information.

[0084] Step 704: Does user approval require? If the result is "yes", proceed to step 705; if the result is "no" (automatic mode), proceed to step 707. After receiving the connection message, the smart proxy layer registers the client's unique identifier field, client_id, in the local memory routing table through the message routing and forwarding module, and performs a conditional attribute determination based on the current system's permission mode to determine whether user approval is required.

[0085] In step 705, the user interface displays the connection information pending approval. If it is determined in step 704 that the current permission mode requires user approval, the message routing and forwarding module marks the corresponding plugin status in the routing table as a plugin pending approval status, and asynchronously sends a plugin event message (plugin_event message) containing the event type event_type configured as connected to the desktop application layer through the WebSocket server module, so that the front-end user interface of the desktop application layer can pop up a prompt window for the connection information pending approval.

[0086] In step 706, the user decides that, after the thread is in a pending approval state and a prompt window is displayed, the thread should be suspended and the user's final approval decision should be sent back from the desktop application layer. If the decision is "Approve", then proceed to step 707; if the decision is "Reject", then proceed to step 711.

[0087] Step 707: Send a connection response (approval flag: true). If it was determined in step 704 that the system is in automatic mode and user approval is not required, or if it was determined in step 706 that the user performed an approval operation, then the message routing and forwarding module constructs a connection response message containing the approval flag as true and sends it to the BIM plugin via the WebSocket server module to execute step 708. Specifically, if the user performs approval on the front end, the front-end initiating component of the desktop application layer calls the `approve_client` command. The message routing and forwarding module constructs a plugin event response with the event type `event_type` configured as `approved` and the status code field `status` configured as 200, updates the routing table status, and then sends it back to the BIM plugin.

[0088] Step 708: Establish a two-way communication channel. After sending a connection response message with the approval flag set to true to the BIM plugin, a full-duplex two-way communication channel is officially established.

[0089] Step 709: During the normal communication phase, the desktop client sends a command message and the plugin returns a command result message. During the normal operation of the bidirectional communication channel, the message routing and forwarding module performs BIM command routing and forwarding interaction across multiple process components, executing step 710. Specifically, when the Building Information Model Interaction Sub-Agent generates a BIM operation command containing the command name field `command_name` and the parameter field `params` according to the intent, the Supervisor module sends a BIM command forwarding message (bim_command message) containing the request identifier field `requestId` and the target client identifier field `target_client_id` to the desktop application layer. This message is encapsulated as a standard plugin protocol command message and then routed and forwarded by the message routing and forwarding module. The sending module precisely routes and forwards data to the target BIM plugin based on the target client identifier field `target_client_id`, allowing the target BIM plugin to call the local BIM software's SDK to perform corresponding model attribute extraction or inspection operations. After execution, the target BIM plugin encapsulates the execution result into a command result message and returns it. The message routing and forwarding module then precisely routes and forwards this message back to the desktop application layer according to the source client identifier field `source_client_id`. Finally, the desktop application layer sends the command result back to the Supervisor module via a command result return message (`bim_command_result` message), allowing the Supervisor module to dynamically stitch together and aggregate the corresponding subtask execution results.

[0090] Step 710: Determine if the heartbeat is normal? During the normal communication phase, a heartbeat keep-alive check is periodically performed on all connections across the entire link. Both the desktop application layer and the BIM plugin are configured to periodically send heartbeat request messages to the WebSocket server module and receive heartbeat response messages (pong messages). Simultaneously, the intelligent proxy layer and the desktop application layer are configured to check all active connections in the routing table every 60 seconds. If the last active time of the current target connection exceeds a preset 90-second time limit, meaning three consecutive heartbeat packets have failed to arrive, it is determined to be in an abnormal state of timeout or disconnection. An asynchronous event notification message is triggered to notify the desktop application layer frontend of the connection disconnection event via the plugin_event message format, and then step 712 is executed to close the connection. If the connection is normal during the normal communication phase in step 709, the heartbeat check in step 710 is continuously executed in a loop. If the BIM plugin layer detects an abnormal connection closure locally, it will automatically initiate a preset exponential backoff reconnection mechanism locally, re-initiating the connection request in increments of 1 second, 2 seconds, 4 seconds, and 8 seconds, and limiting the maximum waiting time for a single reconnection to 30 seconds.

[0091] Step 711: Send a connection response (approval flag: false). If the user's final decision is determined to be a rejection in step 706, the desktop application layer calls the reject_client interface to send a corresponding rejection instruction to the intelligent agent layer. The message routing and forwarding module constructs a connection response message containing the approval flag set to false and sends it back to the corresponding BIM plugin, proceeding to step 712. In this connection response message, the approved boolean field is configured to false, and the specific reason for rejection is carried through the message field of the error message body.

[0092] Step 712: Close the connection. After sending a connection response message with a false approval flag back to the BIM plugin, or when a heartbeat abnormality is detected subsequently, the WebSocket server module forcibly cuts off the long connection path, closes the current connection, and terminates this lifecycle.

[0093] refer to Figure 8 , Figure 8 The diagram illustrates a deployment extension of a natural language command interaction system for BIM software according to an embodiment of this application. As shown, the system is deployed on a user's computer and includes three layers, each running in an independent process: The first layer is the plugin layer, which runs as a BIM add-in within the BIM modeling software process. Taking Autodesk Revit as an example, the plugin layer extends functionality non-intrusively through the publicly available SDK interfaces of the BIM modeling software. The plugin layer acts as a WebSocket client, actively connecting to the intelligent agent backend service in the third layer, the intelligent agent layer. After establishing the connection, it encapsulates metadata including the plugin name, BIM software version, project file name, and plugin version number into a connection message and registers it with the intelligent agent backend service via the plugin protocol. Upon receiving a BIM operation command, the plugin layer parses the command name and operation parameters in the message body to call the corresponding BIM SDK interface to execute the specific operation, and encapsulates the execution result into a command result message containing an HTTP-style status code and result data, returning it to the intelligent agent backend service.

[0094] The second layer is the desktop application layer, which runs in an independent desktop application process. It includes a user interface module that enables bidirectional interaction via inter-process communication and a desktop application backend. The user interface module is implemented through a dialog view, dialog input, settings interface, and model configuration form. The desktop application backend includes a WebSocket client module and a WebSocket command module for Agent communication, an intelligent agent management module for Agent discovery, an embedded relational database module for dialogue history storage, and a configuration module for parameter control. The desktop application layer connects to the intelligent agent layer's WebSocket server module via the desktop application backend as a WebSocket client and through the Agent protocol. The desktop application layer itself does not run a local WebSocket server. The connection status between the desktop application layer and the plugin layer is uniformly managed by the message routing and forwarding module in the intelligent agent layer to achieve non-intrusive cross-process bidirectional asynchronous message relay forwarding.

[0095] The third layer is the intelligent agent layer. This layer runs as an independent backend service process, developed in Python and built on the FastAPI framework. As the sole WebSocket server within the entire command interaction system, it listens on port 8000 by default, providing unified message access, precise cross-process routing, and asynchronous message forwarding capabilities. The intelligent agent layer includes: a WebSocket server module, a message routing and forwarding module, a Supervisor module, a group of sub-Agent modules, a dual-channel retrieval enhancement generation module, a Chromadb vector database, a local text embedding model, a permission management module, a short-term memory module, and a Skills module. During runtime, the intelligent agent layer establishes a long-lived connection channel after detecting network connection requests initiated by the desktop application layer via the Agent protocol through the WebSocket server module. During system startup, the Agent discovery module in the desktop application layer performs liveness checks, health checks, port scanning, and automatic startup control based on agreed-upon port files. The Supervisor module and the sub-Agent module group independently initiate network requests to the external large language model cloud service through application programming interfaces (APIs) to obtain global task scheduling decision support, dynamic tool awareness and invocation capabilities for specific execution chains, and scheduling and streaming response support for artificial intelligence models. Furthermore, the intelligent agent layer uses a message routing and forwarding module to match the source client identifier and target client identifier carried in the message body in a local memory routing table, enabling cross-process bidirectional message relay and asynchronous message forwarding between the desktop application layer and the plugin layer.

[0096] This application also proposes a natural language command interaction method for BIM software. The method includes: S1, the desktop application layer encapsulates the received natural language instructions into chat messages, and sends the chat messages to the intelligent agent layer through the Agent communication module of the desktop application layer; S2, the Supervisor module in the intelligent agent layer receives the chat message, performs intent recognition and task splitting to generate at least one sub-task and assign it to the corresponding sub-Agent, and the sub-agent generates the corresponding BIM operation command according to the intent; S3 sends BIM operation commands to the desktop application layer through the message routing and forwarding module of the intelligent agent layer. After being assembled into standard plug-in protocol command messages, the message routing and forwarding module matches them in the local memory routing table according to the source client identifier carried in the message body, and performs cross-process bidirectional asynchronous message relay forwarding to the plug-in layer. The plug-in layer parses and calls the SDK interface to perform specific operations. S4, the plugin layer encapsulates the specific operation execution result into a command result message and returns it. The message routing and forwarding module recalls and forwards it to the desktop application layer based on the source client identifier. The desktop application layer encapsulates it into a command result return message and returns it to the Supervisor module based on the target client identifier. The Supervisor module performs the result aggregation step to call the large language model for inference and generate a dialogue response, and pushes the dialogue response to the desktop application layer for front-end display.

[0097] In some specific embodiments, before the desktop application layer sends chat messages to the intelligent agent layer, the method further includes a system initialization phase and a plugin connection establishment phase. During the system initialization phase, the desktop application layer is first launched and started. The intelligent agent layer is located or automatically started through the Agent discovery module in the desktop application layer. After startup, the intelligent agent layer activates and runs a single WebSocket server module, and simultaneously initializes the local memory routing table of the message routing and forwarding module and the Supervisor module in local memory. Subsequently, the desktop application layer initiates a network connection to the intelligent agent layer as a WebSocket client through the Agent communication module, and sends an authentication message carrying a pre-configured authentication token through a long connection channel to complete authentication. Simultaneously, the BIM plugin in the plugin layer also connects to the WebSocket server module of the intelligent agent layer as a WebSocket client through the communication module.

[0098] During the plugin connection establishment phase, the BIM plugin in the plugin layer initiates an explicit connection request to the intelligent agent layer. Upon receiving this request, the WebSocket server module in the intelligent agent layer encapsulates it as a plugin event message and asynchronously routes it to the desktop application layer via the message routing and forwarding module. After the user interface module in the desktop application layer receives the user's approval instruction, the approval signal is relayed back to the BIM plugin via the intelligent agent layer, which acts as a relay hub. Simultaneously, the intelligent agent layer registers and saves the source client identifier mapping relationship, carried by the client's unique identifier field, in its local memory routing table, thus formally establishing a non-intrusive, full-duplex, bidirectional communication channel.

[0099] Upon entering the user-initiated dialogue phase, when the user inputs specific natural language commands in the user interface module of the desktop application layer, the dialogue history storage module within the desktop application layer automatically reads the associated historical interaction records from the local embedded relational database. Next, strictly adhering to the Agent protocol, the desktop application layer structurally assembles the read historical records and current interaction parameters, dynamically encapsulating them into a unified JSON-formatted chat message. This message includes a message type field, a list of dialogue history messages, a large language model configuration object, a skill list array, a RAG document directory path field, a permanent file directory path field, an array of temporary file absolute paths, and an optional memory configuration object containing sliding window parameters and a summary trigger threshold. Finally, this message is directed and transmitted to the intelligent agent layer for subsequent multi-agent orchestration and scheduling.

[0100] In some specific embodiments, after receiving a BIM operation command issued across process boundaries, the command execution module in the plug-in layer immediately performs technical parsing of the BIM operation command. Subsequently, the BIM plug-in in the plug-in layer directly calls the publicly available SDK interface of the BIM software to traverse model components, dynamically execute specific BIM operations, and obtain the corresponding physical parameters or execution status. After execution, the BIM plug-in in the plug-in layer encapsulates the successfully obtained data and status codes into a command result message containing execution result data, and returns it to the WebSocket server module of the intelligent proxy layer in the form of a WebSocket client through the communication module.

[0101] After receiving the command result message, the intelligent agent layer uses its built-in message routing and forwarding module as a relay hub. Based on the source client identifier carried in the message body, it matches the message in its local memory routing table and performs cross-process bidirectional asynchronous message relay forwarding, sending the command result message back to the desktop application layer. Upon receiving this message, the desktop application layer encapsulates the result data with the same request identifier into a command result return message via the Agent communication module, and then, based on the target client identifier, sends it back across processes to the Supervisor module within the intelligent agent layer.

[0102] After receiving the command result feedback message, the Supervisor module initiates the execution result aggregation step. According to the dependencies and sequence of subtasks, the Supervisor module dynamically concatenates and aggregates the execution results of each subtask into unified context data. It then calls the large language model to perform integration and polishing processing based on the context data, automatically generates a comprehensive inspection report, and uses this report to generate the final dialogue response, officially entering the streaming response output stage.

[0103] During the streaming response output phase, the intelligent proxy layer, through the WebSocket server module, uses block messages and completion status messages to stream the final dialogue response back to the user interface module of the desktop application layer for front-end streaming rendering and dynamic interactive display. During this period, if the reasoning mode in the large language model configuration object is enabled, the intelligent proxy layer will simultaneously push reasoning process prompts reflecting the large language model's reasoning process while pushing block messages. When the dialogue state or tool call state changes, the Supervisor module centrally directs and controls the dispatch of these changes, and synchronously dispatches the state change to the desktop application layer via system notification messages.

[0104] When the large language model determines that the dialogue response is completely complete, the intelligent agent layer sends a completion message marking the end of the response to the desktop application layer. After receiving the completion message, the desktop application layer uses the dialogue history storage module to package and persistently store the complete dialogue record of this interaction and the front-end driver configuration parameters in a local embedded relational database. This completes the closed loop of the entire business chain involving BIM operations and result output.

[0105] refer to Figure 9 , Figure 9 A business process diagram of a natural language command interaction method for BIM software according to an embodiment of this application is shown. Combined with... Figure 9 The method is illustrated using the specific interaction scenario of a user inputting "Help me check if the height of all walls in the current Revit model is up to standard." The method includes: Step 901: The user enters a natural language command in the desktop application layer, such as "Help me check if the height of all walls in the current Revit model is up to standard".

[0106] Step 902: The desktop application layer reads the conversation history and assembles the chat messages according to the Agent protocol. In this scenario, the desktop application layer reads historical interaction records from the local conversation history storage module and assembles the chat messages, including sliding window parameters, according to the Agent protocol. and summary trigger threshold The optional memory configuration object is encapsulated in a unified JSON format chat message.

[0107] Step 903: The desktop application layer sends the chat message to the smart agent layer.

[0108] Step 904: The intelligent agent layer performs short-term memory optimization. After receiving chat messages, the short-term memory module of the intelligent agent layer performs context pruning and automatic compression and merging of the dialogue history message list at runtime based on the sliding window parameters and summary trigger threshold, thus optimizing the input data for subsequent model inference.

[0109] Step 905: Enter the large language model inference stage. The Supervisor module in the intelligent agent layer, as the global scheduling core, uses the semantic parsing capability of the large language model to perform large language model inference on the received chat messages, performs intent recognition and task decomposition, and accurately classifies the natural language instructions into the "BIM compliance check" task.

[0110] Step 906: Does the large language model decide to call the tool? If the determination is "no", proceed to step 907; if the determination is "yes", proceed to step 908. The multi-agent orchestration and scheduling logic makes conditional judgments based on the inference results. If it is determined that the large language model can respond directly without relying on external capabilities, then the determination is "no", and step 907 is executed; in this specific scenario, it is necessary to obtain external specifications and model data, therefore the determination is "yes", and step 908 is executed.

[0111] Step 907: The large language model directly generates a text response and streams it to the desktop application layer.

[0112] Step 908: Does the tool involve BIM operations? The Supervisor module performs routing judgment based on the attributes of the split multi-source sub-tasks. If the sub-task does not require inter-process interaction with the BIM software itself, it is judged as "No," guiding the data flow to the retrieval branch that does not involve BIM operations, and proceeding to step 912; if the sub-task must rely on the underlying data extraction or model modification of the BIM software, it is judged as "Yes," guiding the data flow to the execution branch that involves BIM operations, dispatching the Building Information Model Interaction Sub-Agent, and proceeding to step 909. In this scenario, the Supervisor module adopts a parallel scheduling strategy, activating and running the above two branches simultaneously.

[0113] Step 909: Encapsulate command messages according to the plugin protocol. The Building Information Modeling (BIM) interaction agent dynamically reads, instantiates, and executes the corresponding BIM inspection skill from the local machine by calling the Skills module. The Supervisor module generates the corresponding BIM operation command based on the skill execution request. Its internal message type carries the corresponding operation command name `commandName="get_all_walls_height"` and sets a unique request identifier `requestId` within the message body. The generated BIM operation command message is sent to the desktop application layer through the message routing and forwarding module built into the intelligent agent layer. After receiving the BIM command message, it is assembled into a command message format that the plugin can parse according to the plugin protocol and a relay request is sent to the intelligent agent layer. The desktop application layer then uses this as the sole security trust domain to filter and supplement context parameters, generating the final message instruction JSON object sent to the BIM plugin, thus achieving process security and control decoupling and isolation between the intelligent agent layer and the local heavy-duty BIM plugin.

[0114] Step 910: Send the command to the BIM plugin for execution. After receiving the command message, the intelligent agent layer parses the message body using its built-in message routing and forwarding module. Based on the target client identifier carried in the message body, it matches the message in the local memory routing table and accurately sends it across process boundaries to the plugin layer in the BIM software process for execution, completing the non-intrusive command delivery.

[0115] Step 911: The plugin returns the execution result, proceeding to step 913. After receiving the relayed command message, the BIM plugin running in the BIM software process parses the command name and operation parameters in the message body, and calls the publicly available SDK interface of the BIM software to traverse the wall components to dynamically obtain the actual wall height parameters. The BIM plugin in the plugin layer encapsulates the successfully obtained height data into a command result message containing a Hypertext Transfer Protocol style status code and execution result data, and returns it to the intelligent agent layer.

[0116] Step 912: The intelligent agent layer directly executes the tool. The multi-agent orchestration and scheduling logic dispatches the dual-channel retrieval enhancement generation retrieval sub-agent to execute the standard retrieval tool call that does not involve BIM operations. The dual-channel retrieval enhancement generation retrieval sub-agent directly calls the dual-channel retrieval enhancement generation retrieval module to search for the standard text of the clauses on "wall height specifications" in parallel in the public knowledge base of the vector database and local documents.

[0117] Step 913: The result is fed back to the LLM, and step 905 continues. The branch execution result involving BIM operations returned in step 911 is finally accurately fed back to the Supervisor module. At the same time, the clause specification text retrieved in step 912 is directly fed back to the Supervisor module as tool result data within the process. At this point, the data flow converges again and triggers step 905. The Supervisor module aggregates the dual-channel retrieval enhancement during the large language model inference stage to generate the "wall height specification clause" fed back by the retrieval sub-Agent and the "actual wall height data" fed back by the building information model interaction sub-Agent. The large language model performs deep inference and comparative analysis to automatically generate a comprehensive inspection report. Finally, the Supervisor module performs the result aggregation step and streams the final dialogue response back to the desktop application layer for front-end display through block messages and completion status messages, completing the closed loop of the entire business process.

[0118] In some specific embodiments, the Supervisor module utilizes an asynchronous generator to achieve streaming interactive communication with the sub-Agent module group and various external tools; specifically, the streaming interactive communication and message processing flow is as follows: When performing task scheduling and aggregation, the Supervisor module calls the asynchronous streaming event acquisition interface astream_events to obtain an asynchronous event iterator and dynamically generates various streaming event types according to the timestamp order. These streaming event types include: on_llm_stream large language model streaming output event corresponding to token events, on_agent_action sub-Agent dispatch event corresponding to sub-Agent dispatch, on_tool_start / end tool call events corresponding to the start and end of tool calls, and on_supervisor_step supervisor aggregation event corresponding to supervisor aggregation. The message routing and forwarding module in the intelligent agent layer performs streaming consumption and encapsulation classification of events in the asynchronous event iterator through the asynchronous loop asyncfor mechanism: if the streaming event is an on_llm_stream large language model streaming output event, it is encapsulated into a chunk message and pushed to the desktop application layer in real time for text stream display; if the streaming event is a reasoning event containing the solution ideas of the large model, it is encapsulated into a reasoning inference message and sent to the desktop application layer; if the streaming event is an on_agent_action sub-Agent dispatch event, it is encapsulated into a system notification message and sent to the desktop application layer; at the end of the streaming interaction of the current session, the message routing and forwarding module encapsulates the final output result into a done status message. The message body of the completed status message explicitly includes the message field as the final reply text and the reasoning_content field as the complete reasoning content, and pushes it to the desktop application layer, which then persists the complete dialogue record to the local relational database.

[0119] Although the principles of the present invention have been described in detail above with reference to preferred embodiments, those skilled in the art should understand that the above embodiments are merely illustrative explanations of the implementation of the present invention and are not intended to limit the scope of the present invention. The details in the embodiments do not constitute a limitation on the scope of the present invention. Any obvious changes, such as equivalent transformations or simple substitutions, based on the technical solutions of the present invention without departing from the spirit and scope of the present invention fall within the protection scope of the present invention.

Claims

1. A natural language command interaction system for BIM software, characterized in that, The system includes: The plug-in layer runs in the BIM software process and performs non-intrusive functional extensions through the SDK interface exposed by the BIM software. It is used to call the SDK interface to execute BIM operation commands issued by the intelligent agent layer. The desktop application layer is an independent desktop application process that connects to the WebSocket server module of the intelligent agent layer via the Agent protocol in the form of a WebSocket client, and is used to provide interactive input of natural language commands; The intelligent proxy layer is an independent backend service process. As the only WebSocket server of the natural language command interaction system for BIM software, it maintains long connections with the plug-in layer and the desktop application layer respectively, and is used to carry message routing and intelligent proxy orchestration and scheduling of large language models. Specifically, the desktop application layer and the plugin layer establish bidirectional decoupled communication based on the Agent protocol and the plugin protocol through the intelligent agent layer, which acts as a relay hub. According to the matching result of the source client identifier and the target client identifier carried in the message body by the intelligent agent layer, the intelligent agent layer performs cross-process bidirectional asynchronous message relay forwarding. The target client identifier is the unique identity of the desktop application layer that initiates the interaction request, and the source client identifier specifically refers to the unique identity of the plugin layer that receives the interaction request and performs the operation.

2. The natural language command interaction system for BIM software according to claim 1, characterized in that, The plug-in layer includes a communication module, a command execution module, and a connection management module, wherein: The communication module and the intelligent agent layer establish a long-term communication connection through the plug-in protocol; The command execution module is used to parse the received BIM operation command and call the SDK interface to perform specific operations in the BIM software; The connection management module is responsible for connection registration, heartbeat keep-alive, and abnormal reconnection.

3. The natural language command interaction system for BIM software according to claim 1, characterized in that, The desktop application layer includes: a user interface module, an agent communication module, a dialogue history storage module, and an agent discovery module, wherein... The user interface module is used to provide a natural language dialogue interaction interface; The Agent communication module is used to connect to the intelligent agent layer and encapsulate and transmit Agent protocol signaling; The dialogue history storage module is used to persistently store historical interaction records in a local embedded relational database; The Agent discovery module is used to detect agreed port files, perform port scans, and implement automatic start and stop control of backend services.

4. The natural language command interaction system for BIM software according to claim 1, characterized in that, The user dialogue request sent from the desktop application layer to the intelligent agent layer is a chat message, and the chat message uses JSON format to transmit the front-end driver configuration. The chat message fields include: message type field, dialogue history message list, large language model configuration object, skill list array, RAG document directory path field, permanent file directory path field, temporary file absolute path list array, and optional memory configuration object containing sliding window parameters and summary trigger threshold; The intelligent agent layer does not perform local persistent storage of the configuration parameters in the chat messages.

5. The natural language command interaction system for BIM software according to claim 4, characterized in that, The internal fields of the large language model configuration object include: large language model API key, API service address, model name, generation temperature parameter, maximum number of generated tokens, and a boolean field used to identify whether the thinking reasoning mode is enabled; The internal fields of each element in the skill list array include: skill name, skill description, a boolean field to identify whether an executable script is included, a boolean field to identify whether a reference file is included, and the absolute path field of the corresponding skill directory on the local machine.

6. The natural language command interaction system for BIM software according to claim 1, characterized in that, The intelligent agent layer includes the WebSocket server module, message routing and forwarding module, Supervisor module, and sub-Agent module group: The WebSocket server module is used to maintain long-lived connections with various heterogeneous terminals, handle connection registration and security authentication, and heartbeat keep-alive. The message routing and forwarding module is used to match the source client identifier and the target client identifier in the local memory routing table and perform relay forwarding of the cross-process bidirectional message; The Supervisor module is used to perform intent recognition, task splitting, parallel and serial scheduling, multi-source result data aggregation and streaming output, and execution failure rollback control on the natural language input from the desktop application layer by utilizing the semantic parsing capabilities of the large language model. The sub-Agent module group is under the centralized command and control of the Supervisor module, and performs deep reasoning and tool invocation under a specific professional execution chain; The sub-Agent module group includes a building information model interaction sub-Agent, a dual-channel retrieval enhancement generation retrieval sub-Agent, a skill execution sub-Agent, and a general dialogue sub-Agent.

7. The natural language command interaction system for BIM software according to claim 6, characterized in that, The intelligent agent layer also includes a dual-channel retrieval enhancement generation retrieval module and a Skills module; The dual-channel retrieval enhancement generation retrieval module is used to perform vectorized parsing and storage of public building code documents and combine them with user-attached local files in real time. The Skills module, acting as a dynamic tool factory, directly and dynamically reads and instantiates structured extended skill files from the local machine. The collaborative processing mechanism utilizing the Supervisor module and the sub-Agent module group includes: In the intent recognition step, the Supervisor module uses the large language model to classify the natural language commands input by the user to identify the corresponding task type; wherein, the task type includes: BIM operation task, knowledge retrieval task, skill execution task, general dialogue task, and comprehensive task involving the linkage of multiple modules; In the task splitting step, if the identified task type is the comprehensive task, the Supervisor module dynamically splits the comprehensive task into a sequence of sub-tasks with logical sequential or parallel relationships. In the parallel-serial scheduling step, the Supervisor module performs differentiated scheduling based on the dependencies within the sub-task sequence, and adopts the following strategy when scheduling the sub-Agent module group to execute tool calls: In response to a knowledge retrieval subtask in the subtask sequence that does not involve BIM software operation, the dual-channel retrieval enhancement generation retrieval agent is called to perform a dual-channel concurrent retrieval in the common knowledge base set in the vector database and the temporary vector set in the running memory. The retrieved clause specification text is then returned to the Supervisor module as the execution result of the knowledge retrieval subtask. In response to a BIM operation subtask in the subtask sequence that involves BIM software operation, the Building Information Model Interaction Agent uses the Skills module to dynamically instantiate and execute the corresponding BIM inspection skill based on the absolute path field of each element in the skill list array, and returns the inspection operation result as the execution result of the BIM operation subtask to the Supervisor module. In the result aggregation step, after each sub-Agent has finished executing, the Supervisor module dynamically splices and aggregates the execution results of each sub-task in the sub-task sequence into unified context data according to the dependency relationship and sequence. It then calls the large language model to perform integration and polishing processing based on the context data and streams the final dialogue response back to the desktop application layer through block messages and completion status messages. For independent subtasks, the Python asyncio.gather asynchronous concurrency mechanism is used to execute the parallel dispatch of multiple sub-agents; for subtasks with causal dependencies, the await asynchronous waiting and suspension mechanism is used to execute the serial dispatch of multiple sub-agents in sequence.

8. The natural language command interaction system for BIM software according to claim 6, characterized in that, The intelligent agent layer also includes a short-term memory module, which is used to perform context pruning and automatic compression and merging of the dialogue history message list at runtime based on sliding window parameters and summary trigger threshold. The short-term memory optimization algorithm combining sliding window and digest is executed using the aforementioned short-term memory module, including: The short-term memory module receives the input dialogue history message list and optional memory configuration object, and performs a threshold determination on the number of messages in the dialogue history message list or the current context length; If the value is determined to be less than the first preset threshold, the original message list is returned directly and the process ends; if the value is determined to be not less than the first preset threshold, a status determination is performed on the dialogue history message list to determine whether it contains a tool call message or a tool call marker. If it is determined that the message contains the tool call message or the tool call flag, it is identified as an uneditable intermediate state, the original message list is returned directly, and the process ends; if it is determined that the message does not contain the tool call message or the tool call flag, the keyword overlap between the current question message and the historical messages is calculated. If the keyword overlap is determined to be less than the set threshold, the current question is determined to be a new topic, only the current question message is retained and the process ends; if the keyword overlap is determined to be not less than the set threshold, a second threshold determination is performed on the context length or number of messages related to the current topic to determine whether it exceeds the second preset threshold. If the threshold is not exceeded, the original message list is returned and the process ends. If the threshold is exceeded, sliding window pruning and summary triggering are performed in combination with parameters N and M in the optional memory configuration object: the most recent N messages are retained as the sliding window inference context. When the number of messages in the waiting-to-be-summarized area outside the sliding window reaches M, the large language model is called to perform dialogue key point summary compression on the old messages outside the sliding window, generating a summary message with a preset number of characters and specific attribute identifiers. The summary message and some active messages in the sliding window are finally combined and retained in the running memory, and the waiting-to-be-summarized area is cleared after generation to end the process.

9. The natural language command interaction system for BIM software according to claim 7, characterized in that, The dual-channel retrieval enhancement generation retrieval module includes a common knowledge base channel, a local document channel, and a retrieval and merging unit: The public knowledge base channel includes a document importer, a local text embedding model, and a Chromadb vector database. The document importer is used to perform asynchronous parsing, MD5 deduplication, and text segmentation on the document indexing task, and converts the segmented text into semantic vector data through the local text embedding model, which is then stored in the Chromadb vector database in isolation according to the knowledge base identifier. The local document channel includes a document parser, a text chunker, and the temporary vector set. The document parser and the text chunker are used to perform real-time parsing and chunking on the user's temporary file list constructed by the user dragging and dropping local files through the desktop application layer, and vectorize and store it in the temporary vector set through the local text embedding model. The temporary vector set is dynamically created in the runtime memory with the session ID of the current Agent session as the namespace to achieve sandbox-style security isolation from the public knowledge base set in the Chromadb vector database. The temporary vector set is deleted and shredded at the end of the current Agent session lifecycle. The retrieval and merging unit includes a retrieval unit and a context merger. The retrieval unit is invoked by the dual-channel retrieval enhancement generation retrieval agent and is used to perform dual-channel concurrent retrieval on the common knowledge base set in the Chromadb vector database and the temporary vector set in the runtime memory. The context merger is used to deduplicate and sort the retrieved multi-source text paragraphs by relevance, and return the merging and optimization execution result to the Supervisor module to invoke the large language model to perform inference.

10. The natural language command interaction system for BIM software according to claim 7, characterized in that, The process of dynamically registering and invoking tools in the Skills module includes: Step A: Parse multiple structured extended skill files from the skill list array in the chat messages; Step B involves iterating through the parsed skill list array and dynamically creating and instantiating multiple corresponding structured tool instances in the runtime memory of the intelligent agent layer. Step C involves injecting the structured tool instances into the sub-Agents corresponding to the functions of the sub-Agent module group according to their tool descriptions. This allows the sub-Agent module group to perceive all available skills and autonomously select and call them based on the orchestration instructions of the Supervisor module. Specifically, BIM operation-related tools are injected into the Building Information Model Interaction sub-Agent, retrieval tools are injected into the dual-channel retrieval enhancement generation retrieval sub-Agent, and a general toolset is injected into the general dialogue sub-Agent. Step D: If a local directory path is determined to exist, the Skills module directly reads the corresponding structured extended skill file content from the absolute path field of the skill list array; if no local directory path is determined to exist, the WebSocket server module asynchronously sends a skill request to the desktop application layer and receives the structured extended skill file content returned by the desktop application layer. Step E involves injecting the read content of the structured extended skill file into the corresponding structured tool instance, so that the Building Information Model Interaction Sub-Agent can be instantiated and execute the BIM inspection skill.

11. The natural language command interaction system for BIM software according to claim 1, characterized in that, The system is also configured with a full-link data and communication security and privacy protection mechanism, which includes: The key protection mechanism ensures that the user's API key is stored in encrypted persistent storage and is only dynamically read when sending dialogue messages to the large language model. After the call is completed, the corresponding memory reference is immediately released and writing to the runtime log is refused. The model data localization mechanism keeps the user's model file in the memory space of the running BIM software process. When the plug-in layer receives the BIM operation command, it calls the SDK interface to read the corresponding component physical properties and transmits them unidirectionally to the intelligent agent layer and the desktop application layer through the local loopback WebSocket connection for use-and-discard processing. The connection verification mechanism involves the WebSocket server module performing a validity check on the authentication token carried in the connection request during the initial handshake when the WebSocket connection between the desktop application layer, the smart proxy layer, and the plugin layer is established. If the verification fails, the connection path is forcibly closed.

12. The natural language command interaction system for BIM software according to claim 1, characterized in that, A dual-end streaming interaction mechanism is configured between the desktop application layer and the intelligent agent layer. The dual-end streaming interaction mechanism includes connection establishment steps, dialogue interaction steps, and heartbeat keep-alive steps. In the connection establishment step, the desktop application layer initiates a WebSocket connection request to the smart proxy layer, and after receiving a successful connection establishment response from the smart proxy layer, sends an authentication message containing an authentication token to the smart proxy layer, and the WebSocket server module performs a validity check on the authentication token. In the dialogue interaction step, the desktop application layer responds to the user's natural language input by sending a chat message to the intelligent agent layer; wherein, the following asynchronous communication branches are configured in the dialogue interaction step: In the streaming output branch, the intelligent agent layer consumes the output events of the large language model in a streaming manner and pushes block messages to the desktop application layer in real time through the message routing and forwarding module. In the skill loading branch, in response to the sub-Agent needing to schedule external extended skill tools and determining that there is no local directory path, the intelligent agent layer asynchronously sends a skill request message to the desktop application layer and receives a skill response message containing the structured extended skill file content returned by the desktop application layer. In the permission confirmation branch, in response to the confirmation mode being triggered by the invoked extended skill tool, the smart agent layer asynchronously sends a permission confirmation request message containing the request identifier and tool name to the desktop application layer, and receives a permission confirmation response message returned by the desktop application layer containing an approval or denial boolean value field. When the interaction ends, the intelligent agent layer pushes a completion message to the desktop application layer, which carries a reply text. The heartbeat keep-alive step involves the desktop application layer periodically sending heartbeat request messages to the intelligent proxy layer and receiving heartbeat response messages from the intelligent proxy layer to achieve state awareness and keep-alive of the dual-end long connection link.

13. The natural language command interaction system for BIM software according to claim 6, characterized in that, The message routing and forwarding module is configured with a plugin connection approval and full lifecycle message routing mechanism, which includes: During the connection verification phase, in response to the connection message initiated by the BIM plugin in the plugin layer, the message routing and forwarding module registers the source client identifier carried in the connection message into the local memory routing table and changes the state of the BIM plugin. When in a permission mode that requires user approval, the state of the BIM plugin is marked as pending approval, and the plugin event message is forwarded one-way to the desktop application layer through the WebSocket server module to trigger the interactive interface to pop up a pending approval prompt, suspend the connection thread and wait for the desktop application layer to send back the user approval decision. During the channel activation phase, in response to being in automatic activation mode or receiving an approval instruction from the desktop application layer, the message routing and forwarding module constructs a plugin event response containing an approval flag and sends it back to the BIM plugin to formally activate the full-duplex bidirectional communication channel; in response to receiving a rejection instruction from the desktop application layer, it constructs a connection response message containing a rejection reason and sends it back to the BIM plugin, and the WebSocket server module forcibly closes the connection path. During the heterogeneous message routing phase, while the bidirectional communication channel is active, in response to the BIM operation sub-task issued by the Supervisor module, the message routing and forwarding module forwards the BIM operation command to the desktop application layer. After encapsulating it into a standard plug-in protocol command message, the message routing and forwarding module precisely routes it to the target BIM plug-in to trigger the BIM software to perform an inspection operation; and During the result recall phase, the message routing and forwarding module receives the command result message returned after the target BIM plugin has completed execution, and recalls and forwards it to the desktop application layer according to the source client identifier field. The desktop application layer then encapsulates the message into a command result feedback message and sends it back to the Supervisor module.

14. A natural language command interaction method for BIM software, characterized in that, The method of using a natural language command interaction system for BIM software as described in any one of claims 1 to 13 includes: S1, the desktop application layer encapsulates the received natural language instructions into a chat message, and sends the chat message to the intelligent agent layer through the Agent communication module of the desktop application layer; S2, the Supervisor module in the intelligent agent layer receives the chat message, performs intent recognition and task splitting to generate at least one sub-task and assign it to the corresponding sub-Agent, and the sub-agent generates the corresponding BIM operation command according to the intent; S3, the BIM operation command is sent to the desktop application layer through the message routing and forwarding module of the intelligent agent layer. After being assembled into a standard plug-in protocol command message, the message routing and forwarding module matches the source client identifier carried in the message body in the local memory routing table and performs cross-process bidirectional asynchronous message relay forwarding to the plug-in layer. The plug-in layer parses and calls the SDK interface to perform specific operations. S4, the plugin layer encapsulates the specific operation execution result into a command result message and returns it. The message routing and forwarding module recalls and forwards the message to the desktop application layer based on the source client identifier. The desktop application layer encapsulates the message into a command result feedback message and sends it back to the Supervisor module based on the target client identifier. The Supervisor module performs the result aggregation step to call the large language model for inference and generate a dialogue response, and pushes the dialogue response to the desktop application layer for front-end display.

15. The natural language command interaction method for BIM software according to claim 14, characterized in that, Prior to step S1, the following is also included: The desktop application layer initiates and triggers the Agent discovery module within it, which then uses a degradation strategy to locate or automatically initiate the intelligent proxy layer as an independent backend service process; wherein the degradation strategy includes: The desktop application layer launches and triggers its Agent discovery module, which then searches for or automatically launches the intelligent agent layer using a degradation strategy; wherein the degradation strategy includes: Port file detection: Read the agreed port file to obtain the process identifier and communication port number of the smart agent layer, check the process liveness status corresponding to the process identifier, and send a health check request to the communication port number; If no live smart agent layer process is found in the port file detection, the health check request is sent to the preset default port. Port scanning: If no response is received from the default port detection, a port scan is performed one by one within a preset port range, and the health check request is sent to each scanned port. Automatic startup: If the port scan does not find an available communication port, the independent backend service process of the intelligent proxy layer is automatically started at the operating system level, and the health check request is continuously sent until the intelligent proxy layer activates the WebSocket server module after passing the health check.

Images