Device control method and system based on natural language and electronic device

Abstract

The invention relates to the technical field of equipment control, and provides an equipment control method and system based on a natural language and electronic equipment. According to the application, the machine language vector is generated through the intention recognition result of the natural language instruction, and the graph query statement is generated according to the semantic concept after the machine language vector is deconstructed, so that the equipment knowledge graph of the original equipment is queried through the graph query statement to obtain the equipment operation path meeting the semantic concept; the knowledge graph has a mesh topology structure, the equipment knowledge graph is used for replacing a rigid relational database structure, deep requirements in natural language instructions are mined through intention recognition, machine language vectors are obtained, the machine language vectors are deconstructed, and the control of the original equipment is achieved. And generating a graph query statement corresponding to the equipment knowledge graph according to the deconstructed semantic concept, thereby accurately and comprehensively querying the complex context semantic relationship between the nodes, and improving the understanding accuracy of the natural language instruction.

Description

Technical Field

[0001] This application relates to the field of equipment control technology, and in particular to a device control method, system and electronic device based on natural language. Background Technology

[0002] In the field of the Internet of Things (IoT), the "Natural Language to SQL" (NL2SQL) technology approach is commonly used for device control. By parsing the user's natural language commands into structured query languages ​​(SQL) suitable for databases, and then manipulating predefined two-dimensional relational data tables that store device information, the device can be controlled to execute simple and direct natural language commands, making it easier for users to control the device.

[0003] However, due to the unique nature of natural language commands, and the fact that an intelligent environment is a network woven from numerous entities such as devices, spaces, users, time, and scenarios, with rich mesh-like relationships between devices, NL2SQL technology employs a rigid mapping model and a flat data table structure. Relational databases cannot effectively represent and query the complex and multidimensional semantic relationships between entities in an intelligent environment, making it difficult to support deep understanding and reasoning of complex natural language commands. They also lack the ability to reason about deep intentions and dynamic contexts, thus failing to directly convert unstructured, semantically rich natural language commands into precise SQL queries. This results in natural language commands not being correctly understood and executed, and the control results being seriously inconsistent with user expectations. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0005] In view of the shortcomings of the prior art described above, this application provides a device control method, system and electronic device based on natural language, so as to improve the accuracy of understanding natural language instructions and thus accurately control the device.

[0006] This application provides a device control method based on natural language, comprising: acquiring a device knowledge graph of an original device and acquiring natural language instructions; performing intent recognition based on the natural language instructions to generate machine language vectors based on the intent recognition results; deconstructing the machine language vectors to obtain semantic concepts in the machine language vectors, and generating a graph query statement based on the semantic concepts; querying the device knowledge graph using the graph query statement to obtain a device operation path that satisfies the semantic concepts, and controlling the original device according to the device operation path.

[0007] In one embodiment of this application, the device knowledge graph includes entity nodes and node relationships between the entity nodes. The entity nodes include at least one of the following: device nodes, used to carry device attributes of the original device; space nodes, used to carry space information where the original device is located; user nodes, used to carry user information bound to the original device; and scene nodes, used to carry usage scene information associated with the original device.

[0008] In one embodiment of this application, the method further includes: if the device attributes include dynamic attributes, then performing data monitoring on the dynamic attributes; if a change in the dynamic attributes is detected, then updating the dynamic attributes in the device knowledge graph according to the changed dynamic attributes.

[0009] In one embodiment of this application, natural language instructions are obtained in the following manner: receiving user instructions input by a user through a user terminal; if a user instruction in text format is received, then the user instruction is treated as a natural language instruction; if a user instruction in voice format is received, then the user instruction is converted into a natural language instruction in text format using speech recognition technology.

[0010] In one embodiment of this application, before performing intent recognition based on the natural language instruction to generate a machine language vector based on the intent recognition result, the method further includes: if the natural language instruction includes a reference word fragment, then by querying the historical instructions corresponding to the natural language instruction, obtaining the target object corresponding to the reference word fragment, and adding the target object corresponding to the reference word fragment to the natural language instruction; if the natural language instruction includes an omitted fragment, then by querying the historical instructions corresponding to the natural language instruction, obtaining the target object corresponding to the omitted fragment, and adding the target object corresponding to the omitted fragment to the natural language instruction.

[0011] In one embodiment of this application, before performing intent recognition based on the natural language instruction to generate a machine language vector based on the intent recognition result, the method further includes: if the natural language instruction includes qualitative control conditions, converting the qualitative control conditions into standardized parameters corresponding to the original device, and adding the standardized parameters to the natural language instruction.

[0012] In one embodiment of this application, intention recognition is performed based on the natural language instruction to generate a machine language vector based on the intention recognition result, including: acquiring real-time environmental data by collecting data on the device environment where the original device is located, and performing intention recognition on the natural language instruction to determine the target device from the original device based on the intention recognition result; inputting the natural language instruction, the real-time environmental data, and the target device as prompt words into a preset large model to perform intention recognition using the preset large model to obtain a structured machine language vector.

[0013] In one embodiment of this application, generating a graph query statement based on the semantic concept includes: obtaining a statement template table, wherein the statement template table includes multiple instruction intent types and query statement templates corresponding to each instruction intent type; determining the core intent type corresponding to the machine language vector by performing intent recognition on the machine language vector, and matching each query statement template according to the comparison results between the core intent type and each instruction intent type to obtain a target template; mapping the device knowledge graph according to the semantic concept to obtain the graph concept corresponding to the semantic concept, and filling the graph concept into the target template to obtain the graph query statement.

[0014] In one embodiment of this application, querying the device knowledge graph using the graph query statement to obtain a device operation path that satisfies the semantic concept includes: querying the device knowledge graph according to the graph query statement to obtain anchor nodes; traversing the device knowledge graph based on the anchor nodes to obtain one or more candidate paths that satisfy the graph concept; evaluating the quality of different candidate paths according to a preset evaluation dimension to obtain an evaluation score corresponding to each candidate path; and filtering each candidate path according to the evaluation score to obtain the device operation path.

[0015] In one embodiment of this application, based on the anchor node, traversing the device knowledge graph to obtain one or more candidate paths that satisfy the graph concept includes: taking the anchor node as the starting point and traversing along the node relationships between entity nodes to obtain candidate target pairs that satisfy the graph concept, wherein the candidate target pairs are used to characterize the association relationship between the original device and the operation instructions; and combining the candidate target pairs to obtain one or more candidate paths.

[0016] In one embodiment of this application, quality evaluation is performed on different candidate paths according to a preset evaluation dimension to obtain an evaluation score corresponding to each candidate path. This includes: if the preset evaluation dimension includes an execution result dimension, then result prediction is performed based on the candidate path to obtain the execution result corresponding to the candidate path, and the evaluation score corresponding to the candidate path is determined based on the semantic similarity between the execution result and the machine language vector; if the preset evaluation dimension includes a user preference dimension, then user preference settings are summarized based on user history records, and the evaluation score corresponding to the candidate path is determined based on the similarity between the candidate path and the user preference settings; if the preset evaluation dimension includes a cost dimension, then cost prediction is performed based on the candidate path to obtain the execution cost corresponding to the candidate path, and the evaluation score corresponding to the candidate path is determined based on the execution cost; if the preset evaluation dimension includes a logic dimension, then logical deduction is performed from the candidate path to obtain logical anomalies in the candidate path, and the evaluation score corresponding to the candidate path is determined based on the logical anomalies.

[0017] In one embodiment of this application, before controlling the original device according to the device operation path, the method further includes: displaying the device operation path to the user; if a user modification instruction corresponding to the device operation path is received, updating the device operation path according to the user modification instruction and displaying the updated device operation path to the user; if a user confirmation instruction corresponding to the device operation path is received, controlling the original device according to the device operation path.

[0018] In one embodiment of this application, controlling the original device according to the device operation path includes: obtaining a control instruction queue corresponding to the device operation path, wherein the control instruction queue includes multiple device control instructions arranged in execution order; formatting the device control instructions corresponding to the original device according to the preset instruction format of the original device; and sequentially sending the formatted device control instructions to the original device in the execution order to trigger the original device to execute the received device control instructions.

[0019] In one embodiment of this application, after controlling the original device according to the device operation path, the method further includes: obtaining the device attributes of the original device; in response to a display instruction, rendering the device attributes according to a preset rendering type corresponding to the device attributes to obtain a device attribute display interface, and displaying the device attribute display interface to the user; in response to a log instruction, persisting the device operation path and the device attributes to obtain a log record.

[0020] This application also provides a natural language-based device control system, comprising: an acquisition module for acquiring a device knowledge graph of an original device and acquiring natural language instructions; a conversion module for performing intent recognition based on the natural language instructions to generate machine language vectors based on the intent recognition results; a generation module for deconstructing the machine language vectors to obtain semantic concepts in the machine language vectors and generating a graph query statement based on the semantic concepts; and a control module for querying the device knowledge graph through the graph query statement to obtain a device operation path that satisfies the semantic concepts and controlling the original device according to the device operation path.

[0021] This application also provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described method.

[0022] The beneficial effects of this application are: Machine language vectors are generated from the intent recognition results of natural language commands. Based on the semantic concepts derived from the deconstruction of these machine language vectors, graph query statements are generated. These graph query statements are then used to query the device knowledge graph of the original device to obtain device operation paths that satisfy the semantic concepts, thus achieving control of the original device. In this way, because the knowledge graph itself has a mesh topology, replacing the rigid relational database structure with a device knowledge graph allows for the extraction of deep needs from natural language commands through intent recognition. Machine language vectors are obtained, and by deconstructing these vectors and generating graph query statements corresponding to the device knowledge graph based on the semantic concepts derived from the deconstruction, the complex contextual semantic relationships between nodes can be accurately and comprehensively queried. This improves the accuracy of understanding natural language commands and enables precise device control. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] In the attached diagram: Figure 1 This is a flowchart illustrating a natural language-based device control method in an embodiment of this application. Figure 2 This is a schematic diagram of a system architecture in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a device knowledge graph in an embodiment of this application; Figure 4 This is a flowchart illustrating another natural language-based device control method in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a natural language-based device control system in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. Detailed Implementation

[0025] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0027] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0028] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0029] Unless otherwise stated, the term "multiple" means two or more.

[0030] In this application, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0031] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0032] Combination Figure 1 As shown, this application provides a device control method based on natural language, including: Step S101: Obtain the device knowledge graph of the original device and obtain natural language instructions; Step S102: Perform intent recognition based on natural language instructions to generate machine language vectors based on the intent recognition results; Step S103: By deconstructing the machine language vector, the semantic concepts in the machine language vector are obtained, and a graph query statement is generated based on the semantic concepts; Step S104: Query the device knowledge graph using a graph query statement to obtain the device operation path that satisfies the semantic concept, and control the original device according to the device operation path.

[0033] The device control method based on natural language provided in this application generates machine language vectors from the intent recognition results of natural language commands. Then, based on the semantic concepts derived from the deconstruction of these machine language vectors, a graph query statement is generated. This graph query statement is used to query the device knowledge graph of the original device to obtain the device operation path that satisfies the semantic concepts, thus achieving control of the original device. In this way, because the knowledge graph itself has a mesh topology, the device knowledge graph replaces the rigid relational database structure. By mining the deep needs in natural language commands through intent recognition, machine language vectors are obtained. The deconstruction of these machine language vectors and the generation of the corresponding graph query statement based on the semantic concepts obtained from the deconstruction allow for accurate and comprehensive querying of the complex contextual semantic relationships between nodes, improving the accuracy of understanding natural language commands and achieving precise device control.

[0034] Combination Figure 2As shown, this application provides a system architecture for implementing a natural language-based device control method. The system architecture includes an input module 201, a processing module 202, and an execution module 203.

[0035] The input module 201 is used to acquire natural language instructions, and the input module 201 includes a speech recognition module.

[0036] Combination Figure 3 As shown, this application provides a device knowledge graph, wherein the device knowledge graph includes entity nodes and node relationships between entity nodes.

[0037] Optionally, the entity node includes at least one of the following: a device node, used to carry the device attributes of the original device; a space node, used to carry the space information where the original device is located; a user node, used to carry the user information bound to the original device; and a scene node, used to carry the usage scene information associated with the original device.

[0038] In some embodiments, node relationships include being located at, belonging to, bound to, and capable of execution.

[0039] Optionally, the method further includes: if the device attributes include dynamic attributes, then monitoring the dynamic attributes; if a change in the dynamic attributes is detected, then updating the dynamic attributes in the device knowledge graph based on the changed dynamic attributes.

[0040] In some embodiments, device attributes include static attributes (e.g., device brand, device model, operation list, etc.) and dynamic attributes (on / off status, brightness status, temperature status).

[0041] In some embodiments, when dynamic attributes change, the device knowledge graph is updated in real time via the Internet of Things message queue MQTT to ensure the timeliness of the device knowledge graph data.

[0042] In some embodiments, after the original device executes the control command, it can actively report device attributes through communication protocols such as MQTT (Message Queuing Telemetry Transport), or report device attributes in response to polling requests. For example, if the original device is an air conditioning device, the device attributes include device name, device ID (Identifier), belonging space, space ID, space path, device category, device category ID, temperature, set temperature, operating status, operating mode, wind speed, wind direction, etc.

[0043] Optionally, natural language instructions can be obtained by: receiving user instructions input by the user through a user terminal; if a user instruction in text format is received, then treating the user instruction as a natural language instruction; if a user instruction in speech format is received, then converting the user instruction into a natural language instruction in text format using speech recognition technology.

[0044] In some embodiments, users input natural language commands using user terminals such as mobile phones, voice panels, and smart speakers; the user terminal has context preservation capabilities and supports understanding multi-turn natural language commands.

[0045] In some embodiments, a preset speech recognition module is used to perform speech recognition on natural language instructions in speech format, so as to convert the natural language instructions in speech format into text format. At the same time, the speech recognition module supports the recognition of multiple languages ​​and accents, and has noise filtering and self-learning optimization mechanisms to ensure recognition accuracy in complex environments.

[0046] In some embodiments, if a natural language instruction is ambiguous or lacks information, the system will proactively ask the user via user terminals such as mobile phones, voice panels, and smart speakers, thereby improving the clarity of the natural language instruction's intent.

[0047] The processing module 202 is used to infer the device operation path based on natural language instructions, wherein the processing module 202 includes a context processing module.

[0048] Optionally, before performing intent recognition based on the natural language instruction to generate a machine language vector based on the intent recognition result, the method further includes: if the natural language instruction includes a reference word fragment, then by querying the historical instructions corresponding to the natural language instruction, obtaining the target object corresponding to the reference word fragment, and adding the target object corresponding to the reference word fragment to the natural language instruction; if the natural language instruction includes an omitted fragment, then by querying the historical instructions corresponding to the natural language instruction, obtaining the target object corresponding to the omitted fragment, and adding the target object corresponding to the omitted fragment to the natural language instruction.

[0049] In some embodiments, a preset context processing module converts natural language instructions into structured machine language vectors, which contain contextual information such as real-time environmental data.

[0050] In some embodiments, if the natural language instruction includes reference words such as "it" or "that", the context processing module queries the target object referred to by the reference word in conjunction with the historical instruction record, and replaces the reference word in accordance with the target object referred to by the reference word.

[0051] In some embodiments, if the natural language instruction includes an omitted segment, such as "raise it a little higher", then the context processing module queries the target object referred to by the omitted segment in conjunction with the historical instruction record, and adds the target object referred to by the omitted segment to the natural language instruction.

[0052] Optionally, before performing intent recognition based on the natural language instruction to generate a machine language vector based on the intent recognition result, the method further includes: if the natural language instruction includes qualitative control conditions, converting the qualitative control conditions into standardized parameters corresponding to the original device, and adding the standardized parameters to the natural language instruction.

[0053] In some embodiments, the control conditions in the natural language instructions include: instruction actions, such as "on", "off", "increase", "set to", "switch to", etc.; target values / parameters, such as "set the temperature to 25 degrees", "adjust the brightness to half", "change the fan speed to medium", etc.; comparison relationships, such as "a little higher", "a little lower", "brightest", "a little cooler", etc.; time conditions, such as "automatically turn off at 8 pm", "turn off all before leaving get off work", "turn on at 6 am every day"; and continuous conditions, such as "run for two hours" "run continuously all night", etc.

[0054] In some embodiments, qualitative control conditions include fuzzy semantic conditions such as "cool down", "turn on maximum airflow", and "turn up the fan speed". The context processing module queries the operation list and parameter specifications corresponding to the original device, and converts the qualitative control conditions into standardized parameters corresponding to the original device based on the query results. For example, the qualitative control condition "cool down" is converted into "set the temperature value to 25°C" based on the temperature value after the current temperature value is reduced (e.g., if the current temperature value is 26°C, "cool down" is converted into "set the temperature value to 25°C"). "Turn on maximum airflow" is converted into "fan speed level is high".

[0055] Optionally, intent recognition is performed based on natural language instructions to generate machine language vectors based on the intent recognition results, including: acquiring real-time environmental data by collecting data from the device environment in which the original device is located, and determining the target device from the original device based on the intent recognition results by performing intent recognition on natural language instructions; inputting the natural language instructions, real-time environmental data, and target device as prompt words into a preset large model to perform intent recognition using the preset large model to obtain a structured machine language vector.

[0056] In some embodiments, machine language vectors are constrained by device parameter thresholds to avoid semantic over- or ineffective control.

[0057] In some embodiments, real-time environmental data includes device status, environmental sensor data, user schedules, time and location, etc.

[0058] In some embodiments, natural language instructions, real-time environmental data, and target devices are input as prompt words into a preset large model; the preset large model outputs a structured machine language vector based on intent reasoning, the machine language vector including target devices, core instructions, device status, and constraints, etc.

[0059] Optionally, generating a graph query statement based on semantic concepts includes: obtaining a statement template table, wherein the statement template table includes multiple instruction intent types and query statement templates corresponding to each instruction intent type; determining the core intent type corresponding to the machine language vector by performing intent recognition on the machine language vector, and matching each query statement template according to the comparison results between the core intent type and each instruction intent type to obtain a target template; mapping the device knowledge graph according to semantic concepts to obtain graph concepts corresponding to the semantic concepts, and filling the graph concepts into the target template to obtain the graph query statement.

[0060] In some embodiments, semantic recognition is performed on machine language vectors using a pre-defined large model to deconstruct the machine language vectors and obtain the semantic concepts contained in the machine language vectors. For example, semantic concepts include target entities, expected states, spatial locations, execution scenarios, etc. The semantic concepts are then mapped to the device knowledge graph to obtain graph concepts in the device knowledge graph. For example, the semantic concept "watching a movie" is mapped to the scene node "watching mode" in the device knowledge graph, and the semantic concept "living room" is mapped to the spatial node in the device knowledge graph, etc.

[0061] In some embodiments, a corresponding target template is selected according to the core intent type, the graph concept is filled into the target template, and a graph query statement is generated. The purpose of the graph query statement is not to obtain the final device operation path, but to locate anchor nodes in the device knowledge graph and extract the core subgraph.

[0062] Optionally, the device knowledge graph is queried using a graph query statement to obtain device operation paths that satisfy semantic concepts. This includes: querying the device knowledge graph using the graph query statement to obtain anchor nodes; traversing the device knowledge graph based on the anchor nodes to obtain one or more candidate paths that satisfy the graph concepts; evaluating the quality of different candidate paths according to preset evaluation dimensions to obtain evaluation scores for each candidate path; and filtering each candidate path based on the evaluation scores to obtain the device operation path.

[0063] Optionally, based on the anchor node, traversal is performed in the device knowledge graph to obtain one or more candidate paths that satisfy the graph concept, including: taking the anchor node as the starting point and traversing along the node relationships between entity nodes to obtain candidate target pairs that satisfy the graph concept, wherein the candidate target pairs are used to characterize the association between the original device and the operation instructions; and combining the candidate target pairs to obtain one or more candidate paths.

[0064] In some embodiments, the anchor node is used as the starting point, and a directional traversal is performed along the node relationships of the device knowledge graph, with the candidate path constrained by device parameter thresholds. During the traversal, candidate target pairs that satisfy the graph concept are collected. For example, starting from the scene node of the "movie viewing scene", the "executable" node relationships are traversed to find all associated operations. Then, combined with the living room node, all matching device instances in the space are found, thus forming a candidate set containing all potential operations. The candidate target pairs are combined to obtain one or more candidate paths. For example, candidate path A includes {closing the curtains, dimming the lights}, and candidate path B includes {closing the curtains, reducing the brightness of the main light, turning off the downlights}. Based on the evaluation scores corresponding to each candidate path, device operation paths are selected from the candidate paths. The device operation paths are converted into a standardized machine-readable array format, such as JSON (JavaScript Object Notation) format. Each object in the array clearly defines a specific operation instruction, including the target device ID, the execution action, and related parameters.

[0065] Optionally, quality evaluation is performed on different candidate paths according to preset evaluation dimensions to obtain evaluation scores for each candidate path. This includes: if the preset evaluation dimensions include an execution result dimension, then result prediction is performed based on the candidate path to obtain the execution result corresponding to the candidate path, and the evaluation score corresponding to the candidate path is determined based on the semantic similarity between the execution result and the machine language vector; if the preset evaluation dimensions include a user preference dimension, then user preference settings are summarized based on user history records, and the evaluation score corresponding to the candidate path is determined based on the similarity between the candidate path and the user preference settings; if the preset evaluation dimensions include a cost dimension, then cost prediction is performed based on the candidate path to obtain the execution cost corresponding to the candidate path, and the evaluation score corresponding to the candidate path is determined based on the execution cost; if the preset evaluation dimensions include a logic dimension, then logical deduction is performed from the candidate path to obtain logical anomalies in the candidate path, and the evaluation score corresponding to the candidate path is determined based on the logical anomalies.

[0066] The execution module 203 is used to obtain the control instruction queue corresponding to the device operation path, and send the device control instructions in the control instruction queue to the corresponding original device in sequence.

[0067] Optionally, before controlling the original device according to the device operation path, the method further includes: displaying the device operation path to the user; if a user modification instruction corresponding to the device operation path is received, updating the device operation path according to the user modification instruction and displaying the updated device operation path to the user; if a user confirmation instruction corresponding to the device operation path is received, controlling the original device according to the device operation path.

[0068] In some embodiments, the device operation path is displayed to the user, and the device operation path is updated through multiple rounds of interaction with the user to obtain an accurate device control range.

[0069] Optionally, controlling the original device according to the device operation path includes: obtaining a control instruction queue corresponding to the device operation path, wherein the control instruction queue includes multiple device control instructions arranged in execution order; formatting the device control instructions corresponding to the original device according to the preset instruction format of the original device; and sending the formatted device control instructions to the original device in the order of execution to trigger the original device to execute the received device control instructions.

[0070] In some embodiments, a control command queue is generated according to the device operation path. The device control commands in the control command queue are formatted to translate into the exclusive protocols and command formats corresponding to different original devices. Then, the formatted device control commands are sent to the corresponding original devices according to the network interface type such as WIFI (wireless network) and Zigbee protocol, so as to control the original devices to execute the device control commands.

[0071] Optionally, after controlling the original device according to the device operation path, the method further includes: obtaining the device attributes of the original device; in response to the display command, rendering the device attributes according to the preset rendering type corresponding to the device attributes to obtain the device attribute display interface, and displaying the device attribute display interface to the user; in response to the log command, persisting the device operation path and device attributes to obtain log records.

[0072] In some embodiments, if the preset rendering type is string, the device attributes are displayed directly; if the preset rendering type is table, the device attributes are rendered as a table based on the table header information; if the preset rendering type is chart_pie, the device attributes are rendered as a pie chart; if the preset rendering type is chart_line, the device attributes are rendered as a line chart; if the preset rendering type is HTML, it is rendered according to the HTML format; if the preset rendering type is markdown, it is displayed according to the markdown style.

[0073] In some embodiments, device operation paths and device attributes are encapsulated to form device control logs, providing a reference for data analysis and user behavior analysis.

[0074] Combination Figure 4 As shown, this application provides a device control method based on natural language, including: Step S401: Obtain the device knowledge graph of the original device; Step S402: Obtain natural language instructions; Step S403: Perform intent recognition based on natural language instructions to generate machine language vectors based on the intent recognition results; If the natural language instruction includes a reference word fragment, then the historical instructions corresponding to the natural language instruction are queried according to the reference word fragment to obtain the target object corresponding to the reference word fragment, and the target object is added to the natural language instruction. If the natural language command includes qualitative control conditions, then the qualitative control conditions are converted into standardized parameters corresponding to the original equipment, and the standardized parameters are added to the natural language command. Step S404: Obtain the semantic concepts in the machine language vector by deconstructing the machine language vector; Step S405: Map the device knowledge graph according to the semantic concepts to obtain the graph concepts corresponding to the semantic concepts; Step S406: By performing intent recognition on the machine language vector, the core intent type is determined, and the graph concept is filled into the target template matched by the core intent type to obtain the graph query statement; Step S407: Query the device knowledge graph according to the graph query statement to obtain anchor nodes; Step S408: Based on the anchor node, traverse the device knowledge graph to obtain one or more candidate paths that satisfy the graph concept; Step S409: Evaluate the quality of the candidate paths according to the preset evaluation dimensions, obtain the evaluation scores, and filter each candidate path according to the evaluation scores to obtain the equipment operation path. Step S410: Obtain the control instruction queue corresponding to the device operation path, and format the device control instructions in the control instruction queue; Step S411: The formatted device control commands are sent to the original device in sequence, triggering the original device to execute the received device control commands.

[0075] The device control method based on natural language provided in this application generates machine language vectors from the intent recognition results of natural language commands. Then, based on the semantic concepts derived from the deconstruction of these machine language vectors, a graph query statement is generated. This graph query statement is used to query the device knowledge graph of the original device to obtain the device operation path that satisfies the semantic concepts, thus achieving control of the original device. In this way, because the knowledge graph itself has a mesh topology, the device knowledge graph replaces the rigid relational database structure. By mining the deep needs in natural language commands through intent recognition, machine language vectors are obtained. The deconstruction of these machine language vectors and the generation of the corresponding graph query statement based on the semantic concepts obtained from the deconstruction allow for accurate and comprehensive querying of the complex contextual semantic relationships between nodes, improving the accuracy of understanding natural language commands and achieving precise device control.

[0076] Combination Figure 5 As shown, this application provides a natural language-based device control system, including: The acquisition module 501 is used to acquire the device knowledge graph of the original device and to acquire natural language instructions.

[0077] The conversion module 502 is used to perform intent recognition based on natural language instructions, and to generate machine language vectors based on the intent recognition results. The generation module 503 is used to deconstruct the machine language vector to obtain the semantic concepts in the machine language vector, and generate a graph query statement based on the semantic concepts.

[0078] The control module 504 is used to query the device knowledge graph through a graph query statement to obtain the device operation path that meets the semantic concept, and control the original device according to the device operation path.

[0079] The device control system based on natural language provided in this application generates machine language vectors through intent recognition of natural language commands. Based on the semantic concepts derived from the deconstruction of these machine language vectors, a graph query statement is generated. This graph query statement is then used to query the device knowledge graph of the original device to obtain the device operation path that satisfies the semantic concepts, thus achieving control of the original device. In this way, because the knowledge graph itself has a mesh topology, the device knowledge graph replaces the rigid relational database structure. Through intent recognition, deep needs in natural language commands are mined to obtain machine language vectors. By deconstructing these machine language vectors and generating graph query statements corresponding to the device knowledge graph based on the semantic concepts obtained from the deconstruction, a closed-loop logic of command input, intent understanding, solution generation, evaluation and optimization, command execution, and status feedback is constructed to build a knowledge graph synchronized with the real world. This allows for accurate and comprehensive querying of complex contextual semantic relationships between nodes, improving the accuracy of understanding natural language commands and achieving precise device control.

[0080] This application also provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described method.

[0081] Figure 6 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0082] like Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from storage portion 608 into Random Access Memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0083] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0084] The electronic device disclosed in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication between them. The memory is used to store computer programs, the communication interface is used to perform communication, and the processor and the transceiver are used to run the computer programs, so that the electronic device performs the various steps of the above method.

[0085] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated subsamples, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other subsamples, wholes, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes the element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0086] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0087] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some sub-samples may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. Furthermore, the functional units in this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than those disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A device control method based on natural language, characterized in that, include: Obtain the device knowledge graph of the original device and obtain natural language commands; Intent recognition is performed based on the natural language instructions to generate machine language vectors based on the intent recognition results; By deconstructing the machine language vector, the semantic concepts in the machine language vector are obtained, and a graph query statement is generated based on the semantic concepts; The device knowledge graph is queried using the graph query statement to obtain the device operation path that satisfies the semantic concept, and the original device is controlled according to the device operation path.

2. The method according to claim 1, characterized in that, The device knowledge graph includes entity nodes and node relationships between the entity nodes. The entity nodes include at least one of the following: A device node is used to carry the device attributes of the original device. Spatial nodes are used to carry spatial information about the location of the original device; User nodes are used to carry user information bound to the original device; Scene nodes are used to carry the usage scene information associated with the original device.

3. The method according to claim 2, characterized in that, The method further includes: If the device attributes include dynamic attributes, then the dynamic attributes are monitored. If a change in the dynamic attribute is detected, the dynamic attribute in the device knowledge graph is updated according to the changed dynamic attribute.

4. The method according to claim 1, characterized in that, Natural language instructions can be obtained in the following ways: Receive user commands input by the user terminal; If a user instruction in text format is received, the user instruction will be treated as a natural language instruction. If a user command in voice format is received, the user command is converted into a natural language command in text format using speech recognition technology.

5. The method according to claim 1, characterized in that, Before performing intent recognition based on the natural language instructions to generate machine language vectors based on the intent recognition results, the method further includes: If the natural language instruction includes a reference word fragment, then by querying the historical instructions corresponding to the natural language instruction, the target object corresponding to the reference word fragment is obtained, and the target object corresponding to the reference word fragment is added to the natural language instruction; If the natural language instruction includes an omitted segment, then the target object corresponding to the omitted segment is obtained by querying the historical instructions corresponding to the natural language instruction, and the target object corresponding to the omitted segment is added to the natural language instruction.

6. The method according to claim 1, characterized in that, Before performing intent recognition based on the natural language instructions to generate machine language vectors based on the intent recognition results, the method further includes: If the natural language instruction includes qualitative control conditions, then the qualitative control conditions are converted into standardized parameters corresponding to the original device, and the standardized parameters are added to the natural language instruction.

7. The method according to claim 1, characterized in that, Intent recognition is performed based on the natural language instructions to generate machine language vectors based on the intent recognition results, including: By collecting data on the device environment in which the original device is located, real-time environmental data is obtained, and by performing intent recognition on the natural language commands, the target device is determined from the original device based on the intent recognition results. The natural language instructions, the real-time environmental data, and the target device are used as prompt words and input into a preset large model to perform intent recognition and obtain a structured machine language vector.

8. The method according to claim 1, characterized in that, Generate a graph query statement based on the semantic concepts, including: Obtain a statement template table, wherein the statement template table includes multiple instruction intent types and query statement templates corresponding to each instruction intent type; By performing intent recognition on the machine language vector, the core intent type corresponding to the machine language vector is determined, and the query statement templates are matched according to the comparison results between the core intent type and each instruction intent type to obtain the target template. The device knowledge graph is mapped according to the semantic concept to obtain the graph concept corresponding to the semantic concept, and the graph concept is filled into the target template to obtain the graph query statement.

9. The method according to claim 1, characterized in that, By querying the device knowledge graph using the aforementioned graph query statement, device operation paths that satisfy the semantic concepts are obtained, including: The device knowledge graph is queried according to the graph query statement to obtain anchor nodes; Based on the anchor node, the device knowledge graph is traversed to obtain one or more candidate paths that satisfy the graph concept; The quality of different candidate paths is evaluated according to the preset evaluation dimensions to obtain the evaluation score corresponding to each candidate path. Based on the evaluation scores, each candidate path is filtered to obtain the device operation path.

10. The method according to claim 9, characterized in that, Based on the anchor node, a traversal is performed in the device knowledge graph to obtain one or more candidate paths that satisfy the graph concept, including: Starting from the anchor node, the nodes are traversed along the node relationships between entity nodes to obtain candidate target pairs that satisfy the graph concept. The candidate target pairs are used to characterize the association between the original device and the operation instructions. The candidate target pairs are combined to obtain one or more candidate paths.

11. The method according to claim 9, characterized in that, The quality of different candidate paths is evaluated according to preset evaluation dimensions to obtain evaluation scores for each candidate path, including: If the preset evaluation dimension includes the execution result dimension, then the result is predicted according to the candidate path to obtain the execution result corresponding to the candidate path, and the evaluation score corresponding to the candidate path is determined according to the semantic similarity between the execution result and the machine language vector. If the preset evaluation dimension includes the user preference dimension, then the user preference settings are obtained by summarizing the user's historical records, and the evaluation score corresponding to the candidate path is determined based on the similarity between the candidate path and the user preference settings. If the preset evaluation dimension includes the cost dimension, then the cost is predicted based on the candidate path to obtain the execution cost corresponding to the candidate path, and the evaluation score corresponding to the candidate path is determined based on the execution cost. If the preset evaluation dimension includes a logical dimension, then logical deduction is performed from the candidate path to obtain logical anomalies in the candidate path, and the evaluation score corresponding to the candidate path is determined based on the logical anomalies.

12. The method according to any one of claims 1 to 11, characterized in that, Before controlling the original device according to the device operation path, the method further includes: Show the user the operation path of the device; If a user modification instruction corresponding to the device operation path is received, the device operation path is updated according to the user modification instruction, and the updated device operation path is displayed to the user. If a user confirmation instruction corresponding to the device operation path is received, the original device is controlled according to the device operation path.

13. The method according to any one of claims 1 to 11, characterized in that, Controlling the original device according to the device operation path includes: Obtain the control instruction queue corresponding to the device operation path, wherein the control instruction queue includes multiple device control instructions arranged in execution order; The device control commands corresponding to the original device are formatted according to the preset command format of the original device. According to the execution order, the formatted device control commands are sent to the original device in sequence, triggering the original device to execute the received device control commands.

14. The method according to any one of claims 1 to 11, characterized in that, After controlling the original device according to the device operation path, the method further includes: Obtain the device attributes of the original device; In response to the display command, the device attributes are rendered according to the preset rendering type corresponding to the device attributes to obtain the device attribute display interface, and the device attribute display interface is displayed to the user. In response to the log command, the device operation path and the device attributes are persisted to obtain log records.

15. A device control system based on natural language, characterized in that, include: The acquisition module is used to acquire the device knowledge graph of the original device and obtain natural language commands; A conversion module is used to perform intent recognition based on the natural language instructions, and to generate machine language vectors based on the intent recognition results; The generation module is used to deconstruct the machine language vector to obtain the semantic concepts in the machine language vector, and generate a graph query statement based on the semantic concepts; The control module is used to query the device knowledge graph through the graph query statement to obtain the device operation path that satisfies the semantic concept, and control the original device according to the device operation path.

16. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 12.

Images