A large model-based home intelligent question answering and design method, device and medium

Abstract

The application discloses a kind of based on big model's home intelligent question and design method, equipment and medium, belong to intelligent house technical field, for solving existing building information model software is difficult to continuously execute complex tasks across multiple tools;Whole interactive process is one-way, by user-driven, lack of intelligent and guiding technical problem.The method comprises the following steps: receiving the input information of user based on the unique session identifier created by the plug-in end;Intention analysis processing is carried out on the input information, and the intention analysis and tool calling decision are obtained based on the calling tool corresponding to the intention analysis result;Intention analysis and tool calling decision are analyzed;The feedback state of the plug-in end is judged to the execution result information, and the execution feedback state result is obtained;The result feedback processing of the relevant plug-in end is carried out to the execution feedback state result, and the result feedback information is obtained;The interrupt workflow in intelligent agent is reactivated and controlled, and the final result output information is obtained.

Description

Technical Field

[0001] This application relates to the field of smart homes, and in particular to a home smart question-and-answer and design method, device and medium based on a large model. Background Technology

[0002] With the development of smart homes and digital design, users expect to interact with home design systems in more natural ways, such as through voice or text commands. In professional building information modeling software (such as Revit), traditional add-ons typically exist as toolbar buttons and dialog boxes. Users need to actively search for and click buttons for specific functions, then manually input precise parameters (such as wall height, component dimensions, etc.) in pop-up windows to execute an atomic design operation. This approach requires users to have a deep understanding of the software's functions and design parameters, making it difficult to comprehend vague, intent-based commands like "design a comfortable bedroom for me," or to continuously execute complex tasks across multiple tools. The entire interaction process is unidirectional, user-driven, and lacks intelligence and guidance.

[0003] Traditional smart home control systems or design tools typically rely on preset, fixed command sets, lacking flexibility and unable to understand complex and comprehensive user intents. Furthermore, they cannot obtain more specialized tool parameters based on simple user descriptions. This application analyzes user intents by calling a large model, automatically mapping corresponding parameters based on the semantics of the user's description, achieving more flexible tool invocation. Simultaneously, it checks the invocation order of different tools; for example, point location design should precede wiring design to more rationally meet user needs. Additionally, traditional tools have poor interactivity, unable to interact with users or remember user preferences. This invention collects user interaction history and inputs it into a large model, more accurately capturing user needs and preferences. Summary of the Invention

[0004] This application provides a home intelligent question-answering and design method, device and medium based on a large model, to solve the following technical problems: existing building information modeling software is difficult to continuously execute complex tasks across multiple tools; the entire interaction process is unidirectional and user-driven, lacking intelligence and guidance.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] On one hand, this application provides a home smart question answering and design method based on a large model, including: receiving user input information based on a unique session identifier created by the plug-in; performing intent analysis processing on the input information through a preset large model, and obtaining intent analysis and tool invocation decisions based on the tool invocation corresponding to the intent analysis results; performing decision parsing on the intent analysis and tool invocation decisions to obtain tool execution step information; judging the feedback status of the execution result information by the plug-in based on the tool execution step information to obtain execution feedback status results; performing result feedback processing on the execution feedback status results related to the plug-in to obtain result feedback information; and reactivating the interrupted workflow in the smart agent based on the workflow not-finished status in the result feedback information to obtain the final result output information at the plug-in.

[0007] This application's embodiments, through basic user interaction and question-and-answer sessions, allow users to interact with the system using complex, multi-step natural language commands, significantly lowering the barrier to entry. It also adds dialogue storage functionality, enabling long-term memory and multi-turn dialogue. Leveraging the powerful reasoning capabilities of a large model, the current question, combined with historical conversations, toolsets, a list of labeled room names, and a list of labeled space types, is input into the large model. This accurately parses user needs into an executable tool call sequence, automatically checks tool parameters, and maps rooms to corresponding room names and space types, forming one or more tool execution instructions. Furthermore, the execution status can be managed through a workflow engine (LangGraph), ensuring that multi-step tasks are executed in an orderly and reliable manner. After a tool call is made on the plugin side, an unfinished workflow is activated through an interface call, continuing execution and forming a closed loop.

[0008] In one feasible implementation, receiving user input information based on a unique session identifier created by the plugin terminal specifically includes: creating the unique session identifier for user interaction through the plugin terminal; wherein the plugin terminal includes at least: a chat interface and a voice assistant front-end; receiving the user's input information based on the unique session identifier and the plugin terminal; wherein the input information includes any one or more of the following: natural language questions, instructions, or images.

[0009] In one feasible implementation, the input information is processed through intent analysis using a pre-defined large model. Specifically, this includes: calling the agent's start interface via the plugin, and passing the input information, session ID, image, and a list of standard room names to the agent; querying the local sqlite3 database and obtaining the status record corresponding to the unique session identifier based on the session ID and through the agent; wherein the status record includes at least historical dialogues; if the local sqlite3 database is not found, a new status record is created; activating or creating corresponding workflows based on the status record and the new status record; wherein the status record controls workflow activation, and the new status record controls workflow creation; and using the large model, performing user intent analysis on the input information and the status records in the workflow related to a home design toolset to obtain the intent analysis result; wherein the home design toolset includes at least: arranging walls, ceilings, and floors; generating furniture; and general Q&A.

[0010] In one feasible implementation, based on the invocation tool corresponding to the intent analysis result, an intent analysis and tool invocation decision is obtained, specifically including: if the intent analysis result is daily dialogue information, image question-and-answer information, or professional question-and-answer information, then the invocation tool is determined to be a question-and-answer tool, and a first invocation decision is obtained; if the intent analysis result is furniture design information, then the invocation tool is determined to be a furniture design tool, and through the large model, the tool name and corresponding tool parameters of the furniture design tool are fed back, resulting in a second invocation decision; if the intent analysis result is room design information, then the invocation tool is determined to be a room design tool, and through the large model, the room name input by the user is mapped to a standard room name and a standard space type, resulting in a third invocation decision; based on the first invocation decision, the second invocation decision, and the third invocation decision, the intent analysis and tool invocation decision is obtained.

[0011] In one feasible implementation, the intent analysis and tool invocation decision are parsed to obtain tool execution step information. Specifically, this includes: using the intelligent agent to perform structured parsing of the intent analysis and tool invocation decision output by the large model to obtain JSON data; wherein the JSON data contains one or more tool names to be invoked and corresponding input parameter lists; and parsing the JSON data online to obtain tool execution step information for the execution control of the invoked tool.

[0012] In one feasible implementation, based on the tool execution step information, the feedback status of the execution result information is judged by the plugin end to obtain the execution feedback status result. Specifically, this includes: obtaining the execution result of the calling tool based on the tool execution step information; if the execution result is a tool execution result that needs to wait for feedback from the plugin end, then the current output status in the agent is stored in the sqlite3 database, the current workflow is ended, and the name of the calling tool that needs to be executed by the plugin end and its corresponding parameters are returned; if the execution result is a tool execution result that does not need to wait for feedback from the plugin end, then the execution steps of the calling tool are judged to determine whether they have been completed: if the execution is completed, the tool execution result is returned to the plugin end; if the execution is not completed, the next execution step is executed. After the tool execution result is returned to the plugin, the plugin is given a feedback status judgment: if the plugin needs to invoke the tool and provide data feedback, the plugin's return status is the first execution feedback status result; wherein, the first execution feedback status result includes: waiting for tool execution and returning relevant information about the invoked tool, the relevant information including: tool invocation identifier, tool name, tool invocation parameters, and the overall model's thought process content; if the plugin does not need to invoke the tool and provide data feedback, the plugin's return status is the second execution feedback status result; wherein, the second execution feedback status result includes: workflow completed, output result returned, and the overall model's thought process content; wherein, the execution feedback status result includes: the first execution feedback status result and the second execution feedback status result.

[0013] In one feasible implementation, the execution feedback status result is processed by the plugin to obtain result feedback information. Specifically, this includes: if the execution feedback status result is a workflow completion status, the plugin returns the output result and the thought content of the large model to the user to obtain user result feedback information; if the execution feedback status result is a waiting tool execution status, the plugin calls the corresponding local tool executor to execute the specific task according to the tool name and corresponding parameters in the tool execution step information, and feeds back the specific task execution result, the corresponding session ID, and the corresponding tool call identifier to the agent through the result interface of the agent called by the plugin to obtain agent result feedback information; wherein, the result feedback information includes: the user result feedback information and the agent result feedback information.

[0014] In one feasible implementation, based on the workflow not-finished state in the result feedback information, the interrupted workflow in the agent is reactivated to obtain the final result output information at the plug-in end. Specifically, this includes: if the workflow not-finished state exists in the result feedback information, the plug-in end sends feedback data through the result interface and controls the agent to receive the feedback data through the result interface; based on the session ID, the interrupted workflow in the agent is determined; based on the interrupted workflow, the current execution result is injected, and the interrupted workflow is reactivated until the workflow not-finished state corresponding to the interrupted workflow is changed to the workflow completed state; based on the workflow completed state, the final result output information in the agent is obtained, and the final result output information is returned to the plug-in end.

[0015] Secondly, embodiments of this application also provide a home smart question-answering and design device based on a large model, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, so that the at least one processor can execute a home smart question-answering and design method based on a large model as described in any of the above embodiments.

[0016] Thirdly, embodiments of this application also provide a non-volatile computer storage medium, which is a non-volatile computer-readable storage medium storing at least one program. Each program includes instructions, which, when executed by a terminal, cause the terminal to execute a home smart question-answering and design method based on a large model as described in any of the above embodiments.

[0017] This application provides a home smart question answering and design method, device, and medium based on a large model. Compared with the prior art, the embodiments of this application have the following beneficial technical effects:

[0018] 1. Natural Interaction: It can perform basic interactions and Q&A with users. Users can interact with the system using complex, multi-step natural language commands, which greatly reduces the barrier to entry. At the same time, it adds a dialogue storage function to realize long-term memory and multi-turn dialogue.

[0019] 2. Intelligent Decision Making: Leveraging the powerful reasoning capabilities of the large model, the current problem is input into the large model along with historical conversations, toolsets, lists of labeled room names, lists of labeled space types, etc. The user's needs are accurately parsed into an executable tool call sequence, the tool parameters are automatically checked, and the room is mapped to the corresponding room name and space type, forming one or more tool execution instructions.

[0020] 3. Reliable execution: The execution status is managed through the workflow engine (LangGraph) to ensure that multi-step tasks can be executed in an orderly and reliable manner. After the plug-in tool is called, the unfinished workflow is activated through the interface call and continues to be executed, forming a closed loop.

[0021] 4. Flexible and scalable: The toolset can be easily extended and updated, while the core of the intelligent agent does not need to be modified frequently, which has good maintainability and scalability.

[0022] 5. Complete Decoupling: This architecture completely decouples intelligent decision-making from concrete execution. Plugin-based toolsets can be iterated and expanded independently of the intelligent agent. For example, local tools can be added to support new design specifications or new building components; simple registration on the server side is all it takes for them to be understood and invoked by the larger model. This allows the system to quickly adapt to evolving industry needs and software updates, maintaining long-term technological advancement. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0024] Figure 1 A flowchart illustrating a home smart question-answering and design method based on a large model, provided for embodiments of this application;

[0025] Figure 2 A flowchart of a workflow engine provided in an embodiment of this application;

[0026] Figure 3 A flowchart of a plug-in interface provided in this application embodiment;

[0027] Figure 4 This is a structural diagram of a home smart question-answering and design device based on a large model, provided as an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0029] It should be noted that this application is based on the LangGraph framework to build a home smart agent server. However, most home design tools are Revit plugins. Traditional smart agent frameworks call tools mostly through interfaces, but plugin tools cannot be encapsulated into callable interfaces. Therefore, the traditional workflow method cannot obtain feedback data from plugin tools. This application solves this problem by designing a waiting state and a recovery state for the workflow. When the tool needs to obtain the execution result from the plugin, it enters a waiting state. If the user calls the / result interface and returns data, the workflow state is restored based on the session ID and tool ID, and subsequent steps are continued.

[0030] Meanwhile, tools in traditional Revit add-ins are local function libraries running within the desktop software environment, tightly coupled with the Revit application. They are not designed as services that can be called via network APIs. Therefore, an external, server-based agent cannot directly call these local tools and synchronously wait for their results. This creates a technological gap between intelligent decision-making (on the server) and design execution (on the client). The "wait-and-recover" mechanism proposed in this application cleverly solves this core problem. The add-in is no longer a passive executor waiting for instructions, but becomes an active, state-aware participant in the workflow. After receiving the tool instructions to be executed, it performs complex and potentially time-consuming design tasks locally, and then actively reports the results to the server through the / result interface, thereby driving the entire workflow to continue. This achieves asynchronous collaboration and closed-loop operation of complex tasks between the web service and the local desktop application.

[0031] It should be noted that the intelligent agent server in this application, i.e., the intelligent agent mentioned, includes:

[0032] 1. Create the ` / start` interface, of type POST, to receive client requests. The parameters include the following:

[0033] User input (user_input);

[0034] Standard room name list (room_list);

[0035] Unique session identifier (session_id);

[0036] Image file (img_file);

[0037] After receiving a request, the API queries the database based on the Session ID to retrieve the session history, loads the state table, and if no record is found or no session identifier is provided, creates a new session and initializes the state table. The workflow engine is then activated.

[0038] 2. Construct the ` / result` interface, of type POST, to receive client requests. The parameters include the following:

[0039] Tool call identifier (tool_call_id);

[0040] Unique session identifier (session_id);

[0041] Termination flag (quit_step);

[0042] Result;

[0043] After receiving the request, the interface queries the database based on the Session ID to obtain the session history, loads the status table, checks whether the unique session identifier and tool identifier match the corresponding steps, and activates the workflow engine.

[0044] 3. Build a workflow engine

[0045] A workflow engine is built based on the LangGraph framework, defining edges and nodes, where nodes include:

[0046] (1) Entry node: The entry point of the workflow;

[0047] (2) Analysis node: Invoke the large model to analyze user intent and parse the results;

[0048] (3) Execution node: Detect steps and execute steps sequentially;

[0049] (4) Waiting node: Waiting for the plugin to execute the tool and return the result;

[0050] (5) Processing Result Node: The result of the processing tool's execution;

[0051] The analysis node calls the large model to perform intent analysis and constructs the prompt, which includes the following:

[0052] (1) The current problem {question};

[0053] (2) Historical Dialogue {history};

[0054] (3) The tools that can be called;

[0055] (4) Label the room names {roomlist};

[0056] (5) Label space type: ["dining room", "living room", "kitchen", "master bedroom", "bedroom", "study", "storage room", "cloakroom", "master bathroom", "guest bathroom", "toilet", "fire rescue room", "toilet", "open balcony", "enclosed balcony", "corridor", "entrance hall"];

[0057] (6) Classification of Design Tools:

[0058] (a) Design of ceiling, floor, and walls:

[0059] Generate the ceiling (roofCreater_tools);

[0060] Create floor (floorCreater_tools);

[0061] Create walls (wallCreater_tools)

[0062] (II) MEP-Site Design:

[0063] Generate angle valves (valveLayoutCreator_tools);

[0064] Generate floor drains (floorDrainCreator_tools);

[0065] Generate the distribution box (distributionBoxCreator_tools);

[0066] Create lighting fixtures (lightingFixtureCreator_tools);

[0067] Socket Creator (socketCreator_tools);

[0068] (III) MEP Wiring Design:

[0069] Generate indoor water supply pipes (autoResidentialSupplyPipe_tools);

[0070] Generate indoor hot water pipes (autoResidentialHotPipe_tools);

[0071] Generate lighting wires (autoLightingConnectionCmd_tools);

[0072] (iv) Furniture arrangement:

[0073] Generate furniture layout (furnitureLayoutCmd_tools);

[0074] (V) Drawing from a source:

[0075] Generate the final image (imageGenerate_tools);

[0076] (7) Prompt content: "You are a professional home design AI assistant that can accurately identify the user's intent and call the corresponding tools to complete the user's needs. The capabilities you provide include intelligent question answering, home design (including floor, ceiling and wall design, intelligent arrangement of indoor furniture, MEP electromechanical point arrangement, MEP point connection, rendering of effect, etc.), spatial semantic analysis, etc...."

[0077] It should be noted that the plugin side of this application includes:

[0078] Develop plugins for BIM software such as Revit, serving as clients for user interaction with the intelligent agent. The plugins primarily implement the following functionalities:

[0079] 1. User Interface: Provides a docked window or Ribbon panel containing a chat input box, a file upload button, and a results display area. Users input natural language commands through this interface.

[0080] 2. Session Management: When a user initiates a new conversation, the plugin generates a globally unique session ID and carries this ID in all subsequent communications to maintain the context of multi-turn conversations.

[0081] 3. Request Construction and Sending: The communication module on the plugin side listens for user input and encapsulates it with context information such as session ID and room list extracted from the current project into a POST request to the agent / start interface.

[0082] 4. Response parsing and scheduling: The plugin receives JSON responses from the smart agent.

[0083] (1) If the status is completed, the text content of result.answer will be displayed on the UI.

[0084] (2) If status is required_tool_execution, check if tool_call is empty. If tool_call is empty, print the agent's thinking and answer content. Otherwise, parse the information in tool_call and hand it over to the tool call module for processing.

[0085] 5. Local Tool Execution: The tool invocation module calls the corresponding function in the local toolset to execute the design task based on tool_name and parameters. Progress indicators may appear during execution. After execution, the result (e.g., "Socket created successfully"), along with tool_call_id and session_id, is combined into a request to the / result interface and sent.

[0086] 6. Loop and Closed Loop: The plugin continues this loop of "receiving instructions -> executing -> providing feedback results" until it receives the final response of the completed state, which signifies the complete realization of the user's current intent.

[0087] This application provides a home smart question-answering and design method based on a large model, such as... Figure 1 As shown, the home intelligence question-answering and design method based on a large model specifically includes steps S101-S106:

[0088] S101: Receive user input information based on the unique session identifier created by the plugin.

[0089] Specifically, a unique session identifier for user interaction is first created through the plugin. The plugin includes at least the chat interface and the voice assistant front-end.

[0090] Furthermore, based on the unique session identifier and the plugin, user input information is received. This input information includes one or more of the following: natural language questions, commands, or images. In other words, the plugin (such as a chat interface or voice assistant front-end) creates a unique session ID for the current user interaction and receives the user's natural language questions, commands, or images.

[0091] S102. Through a pre-set large model, perform intent analysis on the input information, and based on the intent analysis results, obtain the intent analysis and tool invocation decision.

[0092] Specifically, through the plugin, the start interface of the agent is called, and the input information, session ID, image, and standard room name list are passed into the agent.

[0093] Furthermore, based on the session ID, and through the intelligent agent, the local sqlite3 database is queried to obtain the state record corresponding to the unique session identifier. The state record includes at least the historical dialogue.

[0094] Furthermore, if the local sqlite3 database is not found, a new status record is created.

[0095] Furthermore, based on the status record and the new status record, the corresponding workflow is activated or created respectively. The status record controls workflow activation, while the new status record controls workflow creation.

[0096] In one embodiment, the plugin calls the ` / start` interface provided by the agent, passing in user input, session ID, image (optional), and a list of standard room names (for space name mapping). The agent server then creates or activates the corresponding workflow based on the session ID: it queries the local sqlite3 database to retrieve the state record corresponding to the session ID, including historical conversations; if no record is found, a new state record is created.

[0097] Furthermore, through a large model, the input information and state records in the workflow are analyzed to determine the user intent of the home design toolset, yielding intent analysis results. The home design toolset includes at least: arranging walls, ceilings, and floors; generating furniture; and general question-and-answer functionality.

[0098] Furthermore, if the intent analysis result is everyday dialogue information, image-based question-and-answer information, or professional question-and-answer information, the tool to be invoked is determined to be a question-and-answer tool, resulting in the first invocation decision. If the intent analysis result is furniture design information, the tool to be invoked is determined to be a furniture design tool, and through the large model, the tool name and corresponding tool parameters of the furniture design tool are fed back, resulting in the second invocation decision. If the intent analysis result is room design information, the tool to be invoked is determined to be a room design tool, and through the large model, the room name entered by the user is mapped to a standard room name and standard space type, resulting in the third invocation decision.

[0099] In one embodiment, user input and historical dialogue information are submitted to a built-in large model (Qwen3-8B) for intent analysis. The large model determines the user's intent based on a predefined set of home design tools (such as arranging walls, floors, and ceilings, generating furniture, and general question-and-answer tools), deciding which tools to invoke and their parameters. If the user is only engaging in daily conversations, image-based question-and-answer, or professional question-and-answer, the question-and-answer tool is invoked; if a home design tool is needed, the large model returns the tool name and corresponding parameters; if the invoked tool requires room information, the large model maps the user-input room name to a standard room name (roomName) and a standard space type (roomType). For example, if a user inputs "Generate furniture for the second bedroom", the large model will determine the user's intent, call the furniture generation tool, and map "second bedroom" to the standard room name "children's room" and the standard space type "bedroom". Finally, the input will be in JSON format, for example: [{"step": 1, "params": {"roomMessage": [{"roomName":"children's room", "roomType":"bedroom"}]}, "tool": "furnitureLayoutCmd_tools"}].

[0100] Furthermore, based on the first call decision, the second call decision, and the third call decision, the intent analysis and tool call decision are obtained.

[0101] S103. Perform decision analysis on intent analysis and tool invocation decision to obtain tool execution step information.

[0102] Specifically, the intent analysis and tool invocation decisions output by the large model are structured and parsed by an intelligent agent to obtain JSON data. This JSON data contains the names of one or more tools to be invoked, along with their corresponding lists of input parameters.

[0103] Furthermore, the JSON data is parsed online to obtain tool execution step information used to invoke tool control. That is, the agent parses the output of the large model into structured JSON data. This JSON data contains the names of one or more tools to be invoked and their corresponding lists of input parameters, yielding the tool execution steps.

[0104] S104. Based on the tool execution step information, determine the feedback status of the execution result information on the plugin side to obtain the execution feedback status result.

[0105] Specifically, the execution results of the invoked tool are obtained based on the tool's execution step information.

[0106] Furthermore, if the execution result is a tool execution result that requires feedback from the plugin, the current output state in the agent is stored in the sqlite3 database, the current workflow ends, and the name and corresponding parameters of the calling tool that needs to be executed by the plugin are returned. If the execution result is a tool execution result that does not require feedback from the plugin, the execution step of the calling tool is checked for completion: if completed, the tool execution result is returned to the plugin. If incomplete, the next execution step continues.

[0107] In one embodiment, if the execution result is "waiting for tool result" (waiting_for_tool_result), the current state is stored in the sqlite3 database, the current workflow ends, and the name and parameters of the tool that the plugin needs to execute are returned. If no plugin feedback is needed, such as in a question-and-answer tool, after obtaining the tool execution result, it checks whether the tool execution steps have been completed. If the execution is complete, the result is returned; otherwise, the next step continues until the end.

[0108] Furthermore, after the tool execution result is returned to the plugin, a feedback status check is performed on the plugin: if the plugin needs to invoke the tool and provide data feedback, the plugin's return status is the first execution feedback status result. This first execution feedback status result includes: waiting for tool execution and returning relevant information about the invoked tool, including: tool invocation identifier, tool name, tool invocation parameters, and the overall model's thought process. If the plugin does not need to invoke the tool and provide data feedback, the plugin's return status is the second execution feedback status result. This second execution feedback status result includes: workflow completed, output results returned, and the overall model's thought process.

[0109] As a feasible implementation, the tool execution results are returned to the plugin. If the plugin needs to invoke the tool and provide data feedback, the status is returned as "Waiting for tool execution" (requires_tool_execution), along with relevant tool call information, including the tool call identifier (tool_call_id), tool name (tool_name), tool call parameters (parameters), and the overall model's thought process (think). If the plugin is not needed, typically for dialogue or question-and-answer results, the status is returned as "Workflow completed" (completed), along with the result (result) and the overall model's thought process (think).

[0110] The execution feedback status results include the above: the first execution feedback status result and the second execution feedback status result.

[0111] S105. Process the execution feedback status result with the relevant plugin to obtain the result feedback information.

[0112] Specifically, if the execution feedback status is that the workflow is completed, the plugin will return the output results and the thought process of the large model to the user, thus obtaining the user's result feedback information.

[0113] Furthermore, if the execution feedback status is "waiting for tool execution," then based on the tool name and corresponding parameters in the tool execution step information, the corresponding local tool executor is invoked to execute the specific task. The specific task execution result, the corresponding session ID, and the corresponding tool invocation identifier are then fed back to the agent through the `result` interface called by the plugin, thus obtaining the agent's result feedback information. This result feedback information includes both user result feedback information and agent result feedback information.

[0114] As a feasible implementation method, when the plugin receives data returned by the agent server, if the status is "workflow completed", it returns the results and the thinking content of the large model to the user. If the status is "waiting for tool execution", it calls the corresponding local tool executor (Plugin) to execute the specific task according to the tool name and parameters in the instruction. After the tool is executed, the plugin calls the agent's result interface and feeds back the execution result (success data or error information) along with the session ID and tool call identifier (tool_call_id) to the agent.

[0115] S106. Based on the workflow not being completed in the result feedback information, reactivate the interrupted workflow in the agent to obtain the final result output information at the plug-in end.

[0116] Specifically, if the feedback information indicates that the workflow is not yet finished, the plugin sends feedback data through the result interface and controls the agent to receive the feedback data through the result interface.

[0117] Furthermore, based on the session ID, the interrupted workflow within the agent is identified.

[0118] Furthermore, based on the interrupted workflow, the current execution result is injected and the interrupted workflow is reactivated until the workflow incomplete state corresponding to the interrupted workflow is changed to the workflow completed state.

[0119] Furthermore, based on the workflow completion status, the final result output information in the intelligent agent is obtained, and the final result output information is returned to the plug-in end.

[0120] As a feasible implementation, if the workflow is not yet finished in the feedback information during step S1055, and the plugin calls the tool and feeds back data through the result interface, the agent receives the feedback through the result interface, locates the interrupted workflow based on the session ID, injects the execution result, and activates the workflow to continue running to the next step. Steps S104, S105, and S106 are executed in a loop until the workflow ends. The large model then generates the final, comprehensive answer, which is the final result output information within the agent. Finally, the agent returns the final result output information to the plugin through the interface, and the plugin presents it to the user, completing the closed loop of this interaction.

[0121] As a feasible implementation method, this application also provides a home intelligent question-answering and design system based on a large model. This system includes: an intelligent agent server (Agent), i.e., the aforementioned intelligent agent: built based on the LangGraph framework, it is the central hub of the entire system and includes:

[0122] 1. / start interface: The entry point for requests, responsible for obtaining session records and initializing workflow.

[0123] The interface includes the following parameters: user input (user_input), a list of standard room names (room_list), a unique session identifier (session_id), and an image file (img_file). User input and the list of standard room names are required.

[0124] Based on the passed-in session identifier, retrieve the session record from the sqlite3 database and load the state table. If there is no record or no session identifier is passed, create a new session and initialize the state table.

[0125] The workflow engine is then activated.

[0126] 2. result interface: Receives the tool execution results from the plugin, checks whether the unique session identifier and tool identifier match the corresponding steps, and activates the workflow engine.

[0127] 3. Workflow Engine: Integrates and calls large models for intent recognition and tool invocation, manages and maintains state, and records the current execution state (such as the tools invoked and the results obtained).

[0128] in, Figure 2 A flowchart of a workflow engine provided for an embodiment of this application, such as Figure 2 As shown, the processing procedure is as follows:

[0129] S1: Start: Workflow begins. Execute S2.

[0130] S2: Determine whether to terminate the workflow: Check the termination flag. If it is, end the workflow and proceed to S9; otherwise, continue executing S3.

[0131] S3: Determine if there are any incomplete workflows: if so, execute S6; otherwise, execute S4.

[0132] S4: Intent Analysis and Result Parsing: The large model is invoked for intent analysis. The output of the large model is parsed in JSON format to obtain a list of execution tools. The initial step is 0. Continue executing S6.

[0133] S5: Process Tool Results: Process the tool execution results returned by the plugin and record the results in the status. Execute S6.

[0134] S6: Determine if there are subsequent steps: If the current step is less than the total number of steps, continue to S7; otherwise, end the workflow and proceed to S9.

[0135] S7: Tool Execution: Execute the corresponding tool according to the name of the tool in the current step and obtain the result, then execute S8.

[0136] S8: Determine if it is necessary to wait for data return from the plugin: Determine if the tool execution result requires the plugin to execute the tool and return data. If yes, record the current state and end the workflow to proceed to S9. If no, proceed to S6 and repeat all steps until the end.

[0137] S9: End.

[0138] As one feasible implementation, the aforementioned plug-in serves as a terminal for direct interaction with the user and with specific tools. It includes:

[0139] (1) User Interaction Module: As a front-end interface embedded in design software such as Revit, it not only provides interfaces such as chat boxes for receiving user natural language and image input, but also intuitively presents the model's responses, design results, and thought processes to the user. This module can transform abstract text results into visual operations, such as highlighting the room being designed in the model, or previewing the effect of furniture arrangement in real time. It is also responsible for presenting the clarifying questions returned by the agent to the user (such as "Which room do you want to design?"), and feeding back the user's selection to the agent, forming a complete human-computer interaction link.

[0140] (2) Tool Invocation Module (Executor): This is the core of the plug-in, acting as both a "translator" and an "executor." It receives structured JSON instructions from the agent server and parses them. Based on the tool_name in the instruction, it invokes the corresponding specific function in the plug-in's local toolset. This module is responsible for converting the parameters in the instructions (such as {"roomName": "living room", "roomType": "living room"}) into precise instructions and geometric information that the Revit API can understand, thereby performing actual creation, modification, or analysis operations in the BIM model. Simultaneously, it possesses robust error handling capabilities, capable of capturing and processing any problems that may arise during execution (such as model constraint conflicts, invalid parameters, etc.), and packaging the execution status (success or failure information) and feeding it back to the agent.

[0141] (3) Communication Module: This module serves as a bridge between the plug-in and the agent server. It strictly adheres to the defined communication protocol and is responsible for managing HTTP(S) communication. It packages the user's initial request, session ID, and context information of the current BIM model (such as the list of available rooms) and securely sends it to the server's / start interface. After the tool completes execution, it sends the execution result, along with the session ID and tool call ID, to the / result interface. This module ensures data integrity, consistency, and state synchronization throughout the asynchronous interaction process.

[0142] (4) Toolset: This is a modular and extensible design capability library within the plugin. It contains a series of local functions that encapsulate specific design tasks, such as "Wall Layout Tool," "Floor Generation Tool," "Furniture Placement Tool," and "Common Sense Q&A Tool." Each tool is implemented by calling Revit plugin tools and has clearly defined input (parameters) and output (results) specifications. This modular design makes adding new design functions simple; a new tool function only needs to be developed and registered in the tool calling module, which can then be dynamically called by the agent, thereby achieving flexible expansion of system functions.

[0143] In one embodiment, Figure 3 A flowchart of a plug-in interface provided in this application embodiment is shown below. Figure 3 As shown, the specific processing procedure is as follows:

[0144] S1: User Interaction and Command Reception: The plugin starts, and the user inputs questions or commands in natural language through the interface, optionally attaching image files. Workflow begins. Execute S2.

[0145] S2: Request Construction and Sending: Create or obtain a unique session ID for the current interaction. Encapsulate the user input, session ID, and necessary context information (such as the list of standard room names in the currently opened model). Call the agent server's / start interface to send the request. Execute S3.

[0146] S3: Wait and receive response: The plugin enters a waiting state, asynchronously waiting for the agent server to return data. After successfully receiving the JSON response from the server, execute S4.

[0147] S4: Response parsing and status assessment: Parse the JSON response and check the core status field, "status".

[0148] If the status is completed (workflow completed), proceed to S9 to end the current interaction; otherwise, execute S5.

[0149] S5: Tool Identifier Check: Check the tool call field tool_call. If it is not empty, execute S6; otherwise, execute S7.

[0150] S6: Parse the tool call instruction: From the `tool_call` field of the response, parse the tool call identifier (`tool_call_id`), tool name (`tool_name`), and tool call parameters (`parameters`). Search for and match the corresponding tool executor in the plugin's local toolset. Call the executor, passing in the parameters, and execute the specific task in the local design software environment (e.g., calling an API to generate walls, arranging furniture, etc.). After execution, obtain the execution result (success data or error message). Execute S8.

[0151] S7: Output Answer: Extract the final answer (e.g., result.answer) from the response and display it to the user on the plugin's user interface. Proceed to S9 to end this interaction.

[0152] S8: Tool execution result feedback: Return the execution result of the local tool to the server, execute S3 to wait and receive the response.

[0153] S9: End.

[0154] In one embodiment, in a home design scenario:

[0155] The user input: "Please help me generate the furniture and walls for the living room and master bedroom."

[0156] The plugin creates a session ID: sess_123 and calls the / start interface, passing in a list of standard room names for the current room (e.g., ["living room", "kitchen", "open balcony", "study", "bedroom", "master bedroom", "master bathroom", "secondary bathroom", "study"]) and the session ID.

[0157] The ` / start` interface receives the request, queries the database based on the session ID to obtain the session state, activates the workflow engine, and then calls the large model. The large model performs intent recognition and maps the user-mentioned "living room" and "master bedroom" to the corresponding standard room names and standard space types, ultimately outputting structured JSON indicating that the two tools must be called in sequence.

[0158] [{"step": 1, "params": {"roomMessage": [{"roomName":Living Room"", "roomType":"Living Room"},{"roomName":Master Bedroom"", "roomType":"Master Bedroom"}]}, "tool": "furnitureLayoutCmd_tools"};

[0159] {"step": 2, "params": {"roomMessage": [{"roomName":"Living Room", "roomType":"Living Room"},{"roomName":Master Bedroom"", "roomType":"Master Bedroom"}]}, "tool": "wallCreater_tools"} ];

[0160] The workflow engine parses the JSON data and executes two tools in sequence. First, the first tool generates furniture, which requires the plugin to execute and return data. Therefore, the workflow engine changes the status to "waiting for plugin execution", stores the status in the database, and ends the workflow.

[0161] The plugin received the following response:

[0162] {

[0163] "session_id": "session_123",

[0164] "status": "requires_tool_execution",

[0165] "tool_call": {

[0166] "tool_call_id": "5ca111c6-45b5-49cb-93b9-a86b2705c905",

[0167] "tool_name": "furnitureLayoutCmd_tools",

[0168] "parameters": {

[0169] "status": "requires_client_execution",

[0170] "message": "Execute the furniture arrangement tool",

[0171] "tool_name": "FurnitureLayoutCmd",

[0172] "params": [

[0173] {

[0174] "roomName": "Living Room"

[0175] "roomType": "Living Room"

[0176] },

[0177] {

[0178] "roomName": "Master Bedroom"

[0179] "roomType": "Master Bedroom"

[0180] } ]

[0182] }

[0183] },

[0184] "think":"Okay, now I need to handle the user's question: 'Please generate the furniture and walls for the living room and master bedroom.' First, based on the user's history, they previously requested the living room and master bedroom floors, and now they want furniture and walls. I need to determine if the user's intent falls under the category of generation tasks, or if it requires calling the appropriate tools.\n\nThe user's question involves the generation of furniture and walls, which falls under the categories of furniture layout tools (furnitureLayoutCmd_tools) and wall generation tools (wallCreater_tools). According to the design tool classification description, furniture layout tools require room information, and so does wall generation tools. Therefore, I need to check if the user has provided sufficient room information.\n\nThe user's question mentions 'living room' and 'master bedroom,' both of which are in the standard space name list. Next, I need to map these room names to standard room names and types. The living room belongs to 'family room,' and the master bedroom belongs to 'master bedroom.' Therefore, the room information should be {\"roomName\": \"living room\", \"roomType\": \"family room\"} and {\"roomName\": \"master bedroom\", "roomType": "Master Bedroom"}. Next, I need to sort the tools according to the order described in the design tool category instructions. First, call the furniture arrangement tool, then the wall generation tool. Each room is processed separately, so the information for both rooms needs to be passed to the corresponding tools. Check that the parameters are complete; the user didn't provide any additional parameters, so they can be called directly. Confirm that the room names and types are correctly mapped, with no missing or incorrect information. Therefore, the generated tool call should be a two-step process, handling the furniture and walls of the living room and master bedroom separately.

[0185] }

[0186] The plugin calls the "Furniture Design Tool" to execute the design, and upon completion, it calls the / result interface to report: "Furniture design complete".

[0187] The ` / result` interface receives a request, queries the database based on the session ID to obtain the session status, verifies the current tool ID, stores the tool's returned results, and activates the `ses_123` workflow. The workflow analyzes the results and determines whether a step is complete; if not, it proceeds to the next step. Based on the current status (wall completed), it decides on the next step and executes the wall generation tool, which requires waiting for the plugin to run.

[0188] The plugin received the following response:

[0189] {

[0190] "session_id": "session_123", # Session ID

[0191] "status": "requires_tool_execution",

[0192] "tool_call": {

[0193] "tool_call_id": "cdb7f5f6-058a-4bfe-8031-d213b7540f6c",

[0194] "tool_name": "wallCreater_tools",

[0195] "parameters": {

[0196] "status": "requires_client_execution",

[0197] "message": "Execute the wall generation tool",

[0198] "tool_name": FurnitureLayoutCmd"",

[0199] "params": [

[0200] {

[0201] "roomName": "Living Room"

[0202] "roomType": "Living Room"

[0203] },

[0204] {

[0205] "roomName": "Master Bedroom"

[0206] "roomType": "Master Bedroom"

[0207] } ]

[0209] }

[0210] }

[0211] }

[0212] The plugin calls the "Wall Generation Tool" to execute and returns a message via the / result interface: "Wall generation complete".

[0213] When the / result interface receives a request, the agent reactivates the workflow, queries the session status, checks whether the steps are completed, and returns the final result if the steps are completed.

[0214] The plugin receives the final result, summarizes all tool execution results, and returns them to the user. For the user's request, "Please help me generate the furniture and walls for the living room and master bedroom," it completes all related automatic designs. The user can view the final design scheme from multiple dimensions, such as floor plans and 3D models.

[0215] Thus, a complex home design task that required multiple tool calls was successfully completed through human-computer intelligent interaction.

[0216] In one embodiment, in an interactive scenario:

[0217] The user typed: "Please generate furniture for me."

[0218] The plugin creates a session ID: sess_456 and calls the / start interface, passing in a list of standard room names for the current room (e.g., ["living room", "kitchen", "open balcony", "study", "bedroom", "master bedroom", "master bathroom", "secondary bathroom", "study"]) and the session ID.

[0219] The ` / start` interface receives a request, queries the database based on the session ID to obtain the session state, activates the workflow engine, and then calls the large model. The large model performs intent recognition and matches the tool for generating furniture. However, checking the parameters reveals that the user did not mention a room name, so it returns the prompt "What is the name of the room you wish to generate?".

[0220] The workflow engine stores the state in the database and then terminates the workflow.

[0221] The plugin received the following response:

[0222] {

[0223] "session_id": "sess_456",

[0224] "status": "completed",

[0225] "result": {

[0226] Answer: Which room would you like to furnish? For example, living room, bedroom, study, etc. If you haven't specified a room, please tell me the name, and I will generate a corresponding furniture arrangement plan for you.

[0227] },

[0228] "think": "Okay, the user is requesting furniture arrangement again, but the previous conversation didn't provide specific room information. I need to check for any missing parameters. The user may not have specified the room type, such as living room or bedroom. Based on standard space names and types, I need to prompt the user to select a room for arrangement. At the same time, the user may want some common furniture suggestions, such as sofas and TV cabinets. Next, I should call the furniture layout tool and ask the user for the specific room name to ensure the parameters are complete. If the user provides room information later, then continue generating the arrangement plan."

[0229] }

[0230] The plugin returns the results to the user. It then awaits further user interaction, initiating another request using the Session ID. The workflow engine loads the historical dialogue along with the user's current input and feeds it into the larger model for the next decision and process.

[0231] Thus, a complex home design task that required user interaction and multiple rounds of dialogue during tool usage was successfully completed.

[0232] In one embodiment, in a question-and-answer scenario:

[0233] The user enters: "Please describe this apartment layout in detail" and selects the 2D drawing of the current room.

[0234] The plugin creates a session ID: sess_789 and calls the / start interface, passing in the current room's 2D drawing and the session ID.

[0235] The ` / start` interface receives a request, queries the database based on the session ID to obtain the session state, activates the workflow engine, and then calls the large model. The large model performs intent recognition, calls an image question-answering tool, and inputs historical sessions, images, and user questions into the multimodal large language model (qwen2.5-VL-7B) to obtain the output of the large model.

[0236] The workflow engine processes the tool call results and checks if there are any follow-up steps. If there are no follow-up steps, it stores the status in the database and ends the workflow.

[0237] The plugin received the following response:

[0238] {

[0239] "session_id": "sess_789",

[0240] "status": "completed",

[0241] "result": {

[0242] Answer: This is a compact two-bedroom, one-living room apartment. Here is a detailed description:\n\n1. **Living Room**: Located in the center of the apartment, it's the main area for family activities. The living room features a sofa and coffee table, providing a comfortable space for relaxation and socializing.\n\n2. **Bedroom 1**: Located to the left of the living room, this is the master bedroom. It includes a bed and wardrobe, with a simple and practical design.\n\n3. **Bedroom 2**: Located to the right of the living room, this is a secondary bedroom. It also includes a bed and wardrobe, and while relatively compact, it is fully functional.\n\n4. **Bathroom**: Located to the left of Bedroom 2, this is a separate bathroom equipped with a toilet and shower.\n\n5. **Kitchen**: Located at the bottom of the apartment, this is a small kitchen with a stove and refrigerator, featuring a simple and convenient design.\n\n6. **Dining Room**: Located to the right of the living room, this is a small dining room with a table and chairs, suitable for meals and gatherings.\n\n7. **Equipment Platform:** Located at the bottom of the apartment, this is an equipment platform used for storing everyday items. **8. **Open Balcony:** Located to the left of Bedroom 2, this is an open balcony providing space for outdoor activities. This apartment layout is compact yet fully functional, making it suitable for small families.

[0243] },

[0244] "think": ""

[0245] }

[0246] The plugin returns the results to the user. It then awaits further user interaction, initiating another request using the Session ID. The workflow engine loads the historical dialogue along with the user's current input and feeds it into the larger model for the next decision and process.

[0247] Thus, an image question-answering task was successfully completed through human-computer intelligent interaction.

[0248] In addition, embodiments of this application also provide a home smart question-answering and design device based on a large model, such as... Figure 4 As shown, the 400 smart home question-answering and design devices based on a large model specifically include:

[0249] At least one processor 401; and a memory 402 communicatively connected to the at least one processor 401; wherein the memory 402 stores instructions executable by the at least one processor 401 to enable the at least one processor 401 to execute:

[0250] Based on the unique session identifier created by the plugin, it receives user input information;

[0251] The input information is processed through a pre-defined large model to analyze intent, and the intent analysis and tool invocation decision is obtained based on the corresponding tool invocation result.

[0252] Decision analysis is performed on intent analysis and tool invocation decisions to obtain information on tool execution steps;

[0253] Based on the tool's execution step information, the plugin side is used to determine the execution result information and obtain the execution feedback status result.

[0254] The execution feedback status result will be processed by the relevant plugin to obtain the result feedback information;

[0255] Based on the workflow not being completed in the feedback information, the interrupted workflow in the agent is reactivated to obtain the final output information at the plug-in end.

[0256] This application's embodiments, through basic user interaction and question-and-answer sessions, allow users to interact with the system using complex, multi-step natural language commands, significantly lowering the barrier to entry. It also adds dialogue storage functionality, enabling long-term memory and multi-turn dialogue. Leveraging the powerful reasoning capabilities of a large model, the current question, combined with historical conversations, toolsets, a list of labeled room names, and a list of labeled space types, is input into the large model. This accurately parses user needs into an executable tool call sequence, automatically checks tool parameters, and maps rooms to corresponding room names and space types, forming one or more tool execution instructions. Furthermore, the execution status can be managed through a workflow engine (LangGraph), ensuring that multi-step tasks are executed in an orderly and reliable manner. After a tool call is made on the plugin side, an unfinished workflow is activated through an interface call, continuing execution and forming a closed loop.

[0257] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0258] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0259] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0260] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0261] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0262] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0263] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0264] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0265] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0266] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0267] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of this specification.

Claims

1. A home smart question-answering and design method based on a large model, characterized in that, The method includes: Based on the unique session identifier created by the plugin, it receives user input information; The input information is processed by intent analysis using a pre-defined large model, and the intent analysis and tool invocation decision is obtained based on the tool invocation decision corresponding to the intent analysis result. The intent analysis and tool invocation decision are analyzed to obtain tool execution step information; Based on the tool execution step information, the plugin side performs a feedback status judgment on the execution result information to obtain the execution feedback status result, specifically including: Based on the tool execution step information, obtain the execution result of the invoked tool; If the execution result is a tool execution result that needs to be fed back from the plugin, then the current output state in the agent is stored in the sqlite3 database, the current workflow is ended, and the name of the calling tool that needs to be executed by the plugin and the corresponding parameters are returned. If the execution result is a tool execution result that does not require waiting for feedback from the plugin, then determine whether the execution steps of calling the tool have been completed: If the execution is complete, the result of the tool execution will be returned to the plugin for control; if the execution is incomplete, the next execution step will continue. After the tool returns the execution result to the plugin, a status check is performed on the plugin: If the plugin needs to invoke a tool and provide feedback data, the plugin's return status is the first execution feedback status result; wherein, the first execution feedback status result includes: waiting for the tool to execute and returning relevant information about the invoked tool, the relevant information including: tool invocation identifier, tool name, tool invocation parameters, and the thinking content of the large model; If the plugin does not need to call tools and provide data feedback, the plugin's return status is the second execution feedback status result; wherein, the second execution feedback status result includes: workflow completed, output results returned, and the thought content of the large model; The execution feedback status result includes: the first execution feedback status result and the second execution feedback status result; The execution feedback status result is processed by the plugin side to obtain result feedback information, specifically including: If the execution feedback status result is a workflow completion status, then through the plugin, the returned output result and the thinking content of the large model are fed back to the user to obtain user result feedback information. If the execution feedback status result is a waiting tool execution status, then according to the tool name and corresponding parameters in the tool execution step information, the local corresponding tool executor is called to execute the specific task, and the specific task execution result, the corresponding session ID and the corresponding tool call identifier are fed back to the intelligent agent through the result interface of the intelligent agent called by the plug-in end, so as to obtain the intelligent agent result feedback information; The result feedback information includes: the user result feedback information and the agent result feedback information; Based on the workflow not being completed in the feedback information, the interrupted workflow in the agent is reactivated to obtain the final result output information at the plug-in end.

2. The home intelligent question-answering and design method based on a large model according to claim 1, characterized in that, Based on the unique session identifier created by the plugin, user input information is received, specifically including: The plugin creates a unique session identifier for user interaction; wherein the plugin includes at least a chat interface and a voice assistant front-end. Based on the unique session identifier and the plugin, the user's input information is received; wherein the input information includes any one or more of the following: natural language questions, instructions, or images.

3. The home intelligent question answering and design method based on a large model according to claim 1, characterized in that, The input information is processed through intent analysis using a pre-defined large model, specifically including: The plugin calls the agent's start interface and passes the input information, session ID, image, and standard room name list to the agent. Based on the session ID, and through the intelligent agent, the local sqlite3 database is queried and the status record corresponding to the unique session identifier is obtained; wherein, the status record includes at least: historical dialogue; If the local sqlite3 database is not found, a new status record is created. Based on the status record and the new status record, the corresponding workflow is activated or created respectively; wherein, the status record is the activation control of the workflow, and the new status record is the creation control of the workflow; Using the large model, the input information and the state records in the workflow are analyzed to determine the user intent of the home design toolset, and the intent analysis results are obtained; wherein, the home design toolset includes at least: arranging walls, ceilings, and floors, generating furniture, and general question and answer.

4. The home intelligent question answering and design method based on a large model according to claim 3, characterized in that, Based on the intent analysis results and the corresponding tool invocation decisions, the intent analysis and tool invocation decisions are obtained, specifically including: If the intent analysis result is daily dialogue information, image question-and-answer information, or professional question-and-answer information, then the calling tool is determined to be a question-and-answer tool, and a first calling decision is obtained; If the intent analysis result is furniture design information, then the calling tool is determined to be a furniture design tool, and through the large model, the tool name and corresponding tool parameters of the furniture design tool are fed back to obtain a second calling decision; If the intent analysis result is room design information, then the calling tool is determined to be a room design tool, and through the large model, the room name input by the user is mapped to a standard room name and a standard space type to obtain a third calling decision; The intent analysis and tool invocation decision are obtained based on the first invocation decision, the second invocation decision, and the third invocation decision.

5. The home intelligent question answering and design method based on a large model according to claim 1, characterized in that, The intent analysis and tool invocation decision are analyzed to obtain tool execution step information, specifically including: The intelligent agent performs structured parsing of the intent analysis and tool invocation decisions output by the large model to obtain JSON data; wherein, the JSON data contains one or more tool names to be invoked and corresponding input parameter lists; The JSON data is parsed online to obtain the tool execution step information used for the execution control of the calling tool.

6. The home intelligent question-answering and design method based on a large model according to claim 1, characterized in that, Based on the workflow not being completed in the feedback information, the interrupted workflow in the agent is reactivated to obtain the final output information at the plug-in end, specifically including: If the workflow is not finished in the result feedback information, the plug-in sends feedback data through the result interface and controls the agent to receive feedback data through the result interface. Based on the session ID, the interrupted workflow in the agent is determined; Based on the interrupted workflow, the current execution result is injected and the interrupted workflow is reactivated until the workflow incomplete state corresponding to the interrupted workflow is changed to the workflow complete state. Based on the workflow completion status, the final result output information in the agent is obtained, and the final result output information is returned to the plugin.

7. A home intelligent question-answering and design device based on a large model, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, enabling the at least one processor to execute a large-model-based smart home question-answering and design method according to any one of claims 1-6.

8. A non-volatile computer storage medium, characterized in that, The storage medium is a non-volatile computer-readable storage medium that stores at least one program, each program including instructions that, when executed by a terminal, cause the terminal to perform a home smart question-answering and design method based on a large model according to any one of claims 1-6.

Images