Terminal control method and device, equipment and storage medium

Abstract

The invention provides a terminal control method and device, equipment and a storage medium. The terminal control method in the application comprises the following steps: collecting multi-modal scene data in a target interaction scene associated with a smart home system; generating a target decision instruction according to a target processing model and the multi-modal scene data; performing private protocol conversion on the target decision instruction to obtain a target control instruction corresponding to a target home terminal; and driving the target home terminal to execute the target control instruction to obtain a terminal control result. According to the technical scheme of the invention, cross-ecological cooperative linkage of all smart homes can be realized, active interaction of the smart home terminals is realized, and the intelligent degree and interaction initiative of the smart home system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things control, in particular to a terminal control method and device, equipment and a storage medium. BACKGROUND

[0002] At present, with the rapid development of Internet of Things technology and smart home technology, more and more users use multi-category smart home terminals to form a whole-house smart system to realize intelligent control of lighting, security, home appliances and environmental monitoring. However, the existing smart home terminals are often based on their own ecology and communication protocols (such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc.), resulting in incompatibility between devices, and users need to switch between multiple Apps for management. Moreover, the existing smart home terminals are in a "command-response" passive interaction mode, which cannot provide active scene-based services according to the environment, resulting in poor compatibility and low intelligence level of the existing smart home terminals, which cannot meet the intelligent needs of users. SUMMARY

[0003] The embodiments of the present application provide a terminal control method, device, equipment and storage medium, aiming to solve the technical problem of poor compatibility and low intelligence level of multi-smart home terminals in the prior art.

[0004] In one aspect, the embodiments of the present application provide a terminal control method, which comprises the following steps:

[0005] Collecting multi-modal scene data in a target interaction scene associated with the smart home system;

[0006] Generating a target decision instruction according to a target processing model and the multi-modal scene data;

[0007] Converting the target decision instruction into a target control instruction corresponding to a target home terminal through a private protocol;

[0008] Driving the target home terminal to execute the target control instruction to obtain a terminal control result.

[0009] In one possible implementation of the present application, the multi-modal scene data comprises scene environment data and object behavior data.

[0010] The collecting of the multi-modal scene data in the target interaction scene associated with the smart home system comprises:

[0011] Collecting scene information of the target interaction scene through a first sensor to obtain scene environment data;

[0012] Collecting object behavior of a target interaction object through a second sensor to obtain object behavior data;

[0013] generate multi-modal scene data of the target interaction scene according to the scene environment data and the object behavior data.

[0014] In a possible implementation manner of the present application, the generating of the target decision instruction according to the target processing model and the multi-modal scene data comprises:

[0015] performing parameter extraction on the multi-modal scene data by using the target agent to obtain target interaction parameters;

[0016] performing scene analysis on the target interaction parameters by using the target processing model to obtain an initial decision instruction;

[0017] performing instruction protocol conversion on the initial decision instruction according to a target control interface to obtain the target decision instruction.

[0018] In a possible implementation manner of the present application, the performing of scene analysis on the target interaction parameters by using the target processing model to obtain an initial decision instruction comprises:

[0019] performing instruction matching according to corresponding decision rule information of the smart home system and the target interaction parameters to obtain a first decision instruction;

[0020] matching the target interaction parameters and historical decision parameters corresponding to a target object to determine a target historical parameter in the historical decision parameters and a second decision instruction corresponding to the target historical parameter;

[0021] performing instruction analysis on the target interaction parameters by using the target processing model to obtain a third decision instruction;

[0022] determining an initial decision instruction according to the first decision instruction, the second decision instruction and the third decision instruction.

[0023] In a possible implementation manner of the present application, the performing of instruction protocol conversion on the initial decision instruction according to a target control interface to obtain the target decision instruction comprises:

[0024] performing instruction splitting on the initial decision instruction to obtain initial decision sub-instructions corresponding to each target home terminal;

[0025] performing conversion on the initial decision sub-instructions according to the target control interface and a terminal identifier of the target home terminal to obtain converted decision sub-instructions;

[0026] summarizing the converted decision sub-instructions to obtain the target decision instruction.

[0027] In a possible implementation of the present application, the private protocol conversion of the target decision instruction to obtain the target control instruction corresponding to the target home terminal comprises:

[0028] determining the target home terminal corresponding to the target decision instruction, and a protocol adaptation framework corresponding to the target home terminal;

[0029] performing private protocol conversion on the target decision instruction by using the protocol adaptation framework to obtain the target control instruction corresponding to the target home terminal.

[0030] In a possible implementation of the present application, the private protocol conversion of the target decision instruction to obtain the target control instruction corresponding to the target home terminal comprises:

[0031] performing instruction information extraction on the target decision instruction by using the protocol adaptation framework to obtain target instruction information;

[0032] mapping the target instruction information by using a target private protocol corresponding to the protocol adaptation framework to obtain the target control instruction corresponding to the target home terminal.

[0033] On the other hand, the present application provides a terminal control device, which comprises:

[0034] a data acquisition module configured to acquire multi-modal scene data in a target interactive scene associated with the smart home system;

[0035] a decision generation module configured to generate a target decision instruction according to a target processing model and the multi-modal scene data;

[0036] a protocol conversion module configured to perform private protocol conversion on the target decision instruction to obtain a target control instruction corresponding to a target home terminal;

[0037] an active interaction module configured to drive the target home terminal to execute the target control instruction to obtain a terminal control result.

[0038] On the other hand, the present application also provides a terminal control device, which comprises:

[0039] one or more processors;

[0040] a memory; and

[0041] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the steps of the terminal control method.

[0042] In another aspect, the present application also provides a computer readable storage medium having stored thereon a computer program, which is loaded by a processor to execute the steps in the terminal control method.

[0043] In the present application, multi-modal scene data in a target interactive scene associated with the smart home system is collected; a target decision instruction is generated according to a target processing model and the multi-modal scene data; the target decision instruction is converted into a target control instruction corresponding to a target home terminal through a private protocol; and the target home terminal is driven to execute the target control instruction to obtain a terminal control result. In a smart home system corresponding to multiple smart home terminals in multiple ecologies, multi-modal scene data in a target interactive scene associated with the smart home system is actively collected, and the multi-modal scene data is analyzed by using a target processing model to generate a target decision instruction. The target decision instruction is converted into a target control instruction corresponding to a private protocol instruction of each smart home terminal, and the target control instruction is used to actively drive the target home terminal to execute a corresponding interactive operation, so as to realize cross-ecology collaborative linkage of each smart home, and to realize active interaction of the smart home terminal, thereby improving the intelligent degree and interaction initiative of the smart home system. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 A scene schematic diagram of the terminal control method of the embodiments of the present application;

[0046] Figure 2 A flowchart of one embodiment of the terminal control method in the embodiments of the present application;

[0047] Figure 3 A flowchart of one embodiment of generating an initial decision instruction in the terminal control method provided by the embodiments of the present application;

[0048] Figure 4 A flowchart of one embodiment of performing private protocol conversion of the instruction in the terminal control method provided by the embodiments of the present application;

[0049] Figure 5 A structural schematic diagram of one embodiment of the terminal control device provided by the embodiments of the present application;

[0050] Figure 6Fig. 1 is a structural schematic diagram of an embodiment of a terminal control device provided by the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the present application.

[0052] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0053] In the present application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" in the present application is not necessarily to be construed as preferred or advantageous over other implementations. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, for purposes of explanation, specific details are set forth. It is apparent to those skilled in the art that the present application can be practiced without using these specific details. In other instances, well-known structures and processes are not described in detail in order to avoid obscuring the description of the present application. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features presented herein.

[0054] At present, with the rapid development of Internet of Things technology and smart home technology, more and more users are using multi-category smart home terminals to form a whole-house smart system to realize intelligent control of lighting, security, home appliances and environmental monitoring. However, existing smart home terminals are often based on their own ecosystems and communication protocols (such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc.), resulting in incompatibility between devices. Users need to switch between multiple apps for management, and the interaction mode of existing smart home terminals is "command-response" passive interaction, which cannot actively provide scenario-based services according to the environment. As a result, the compatibility of existing smart home terminals is poor and the intelligence level is low, which cannot meet the intelligent needs of users.

[0055] Based on this, the present application proposes a terminal control method, device, equipment and computer-readable storage medium to solve the technical problems in the prior art of poor compatibility and low intelligence level of multiple smart home terminals.

[0056] The terminal control method in an embodiment of the present invention is applied to a terminal control device, which is arranged in a terminal control device. The terminal control device is provided with one or more processors, memories, and one or more applications, wherein the one or more applications are stored in the memories and configured to be executed by the processor to implement the terminal control method; wherein the terminal control device can be an intelligent terminal, such as a mobile phone, a tablet computer, a network device, and a smart computer, etc.; optionally, the terminal control device can also be a server, or a service cluster consisting of multiple servers.

[0057] like Figure 1 As shown, Figure 1 This is a schematic diagram of a scenario for the terminal control method according to an embodiment of the present application. The terminal control scenario in this embodiment includes a terminal control device 100 (with a terminal control device integrated therein) and multiple smart home terminals 200. The terminal control device 100 runs a computer-readable storage medium corresponding to the terminal control method to execute the steps of the terminal control method. The smart home terminals 200 are connected to the terminal control device 100 in communication and are configured to receive target control instructions transmitted by the terminal control device to execute corresponding interactive operations.

[0058] It is understandable that Figure 1 The terminal control device in the terminal control method scenario shown, or the devices included in the terminal control device, does not constitute a limitation on the embodiments of the present invention, that is, the number and type of terminal control devices included in the terminal control method scenario, or the number and type of devices included in each device, do not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be regarded as equivalent replacements or derivatives of the technical solution claimed to be protected in the embodiments of the present invention.

[0059] In the embodiment of the present invention, the terminal control device 100 is mainly used to: collect multimodal scene data in the target interaction scene associated with the smart home system; generate a target decision instruction based on the target processing model and the multimodal scene data; perform a private protocol conversion on the target decision instruction to obtain a target control instruction corresponding to the target home terminal; drive the target home terminal to execute the target control instruction to obtain a terminal control result.

[0060] The terminal control device 100 in the embodiment of the present invention can be an independent terminal control device, such as a smart terminal such as a mobile phone, tablet computer, smart TV, network device, server and smart computer, or it can be a terminal control network or terminal control cluster composed of multiple terminal control devices.

[0061] The embodiments of the present application provide a terminal control method, apparatus, device, and computer-readable storage medium, which are described in detail below.

[0062] It will be understood by those skilled in the art that Figure 1 The application environment shown in is only one of the application scenarios related to the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer terminal control devices shown, or terminal control network connection relationships, such as Figure 1 Only one terminal control device is shown. It can be understood that the scenario of the terminal control method can also include one or more terminal control devices, which are not specifically limited here; the terminal control device 100 can also include a memory for storing multimodal scenario data and other data.

[0063] It should be noted that Figure 1 The scenario diagram of the terminal control method shown is only an example. The scenario of the terminal control method described in the embodiment of the present invention is to more clearly illustrate the technical solution of the embodiment of the present invention, and does not constitute a limitation on the technical solution provided by the embodiment of the present invention.

[0064] Based on the scenario of the above terminal control method, various embodiments of the terminal control method disclosed in the present invention are proposed.

[0065] like Figure 2 As shown, Figure 2 This is a flow chart of an embodiment of a terminal control method in an embodiment of the present application. The terminal control method includes the following steps 201 to 204:

[0066] 201. Collect multimodal scene data in a target interaction scene associated with the smart home system;

[0067] The terminal control method in the embodiment is applied to a terminal control device, and the type and quantity of the terminal control device are not specifically limited, that is, the terminal control device can be one or more intelligent terminals as a central control device in a smart home system to drive each intelligent terminal in the smart home system to actively execute each control instruction. In a specific embodiment, the terminal control device is a smart television or other high-performance device that can deploy a target processing model.

[0068] Optionally, the target processing model is a multi-modal large model AI agent, and the multi-modal large model refers to an artificial neural network model with a very large number of parameters. In the field of artificial intelligence, a large model generally refers to a model with hundreds of millions to trillions of parameters. The model usually needs to be trained on a large-scale data set and requires a large amount of computing resources for optimization and adjustment. Large models are usually used to solve complex multi-modal document knowledge extraction, natural language processing, computer vision, and speech recognition tasks.

[0069] Optionally, the multi-modal large model can be a DeepSeek, ChatGPT, BERT, XLNet, ZhiP model, Claude, Moonshot AI model, ChatGLM model, TongYiQianWen model, MiniMax model, XingHuo model, XingZhi model, Llama model, 360GPT model, Qwen model, Baichuan model, Yungue model, vivoLM model, and WenXinYiYan model, etc. multi-modal large-scale language model, and the embodiments of the present application are not limited. Among them, the target processing model can give the terminal control device the ability of intent understanding and scenario decision-making, so that the terminal control device can be used for data analysis on multi-modal scene data, thereby actively and heuristically generating interaction instructions corresponding to the smart home system and the multi-modal scene data, to actively control the smart home system in different ecosystems to perform corresponding interaction operations in collaboration.

[0070] Optionally, the smart home system is an Internet of Things system in which multiple smart home terminals of different ecologies are arranged in a target interaction area. Different ecologies refer to smart home terminals of different brands, and the ecosystems and communication protocols of each smart home terminal are incompatible with each other. Optionally, the smart home terminal is a home terminal that can access a local area network or the Internet and respond to remote interaction instructions sent by a target object or other terminal to perform corresponding interaction operations. Optionally, in one specific embodiment, the smart home terminal can be any smart home device such as a smart air conditioner, a smart refrigerator, a smart television, a smart lamp, a smart curtain, and a smart sound box. Optionally, the smart home terminal can be a cross-brand smart home device between different brands, which causes the smart home terminal to be unable to perform cross-brand and cross-protocol collaborative connection, resulting in the need for the user to control the smart home terminal to perform corresponding operations through multiple different applications.

[0071] Optionally, the target interaction scene is a scene area in which the smart home system deploys each smart home terminal. For example, the target interaction scene can be an indoor or outdoor scene in which a smart home terminal is arranged in a living room, a room, a kitchen, or the like.

[0072] Optionally, the terminal control device can serve as a smart central control device of the smart home system and collect multi-modal scene data in the target interaction scene associated with the smart home system, thereby avoiding the need to purchase an additional central control device for the smart home system and saving control costs of the smart home system.

[0073] Optionally, the multi-modal scene data is data information of various modal types contained in the target interaction scene, which is used to represent home environment information in the target interaction scene and behavior information of a specified object in the target interaction scene to provide data basis for generating a target decision instruction for a subsequent target processing model.

[0074] Optionally, in one specific embodiment, the multi-modal scene data includes scene environment data and object behavior data. The scene environment data refers to current environmental parameters in the target interaction scene, which represents the environmental state of the target interaction scene. For example, in one specific embodiment, the scene environment parameters can be scene temperature parameters, scene humidity parameters, scene illumination parameters, and air quality parameters.

[0075] The object behavior data is data information representing the behavior, activity, and interaction of the target object, which is used to represent the behavior information of the target object in the target interaction scene. Optionally, in one specific embodiment, the object behavior data includes posture behavior data and voice behavior data. The posture behavior data is behavior data representing the posture action of the target object in the target interaction scene. The voice behavior data is voice data information representing the voice triggered by the target object in the target interaction scene.

[0076] Optionally, the target object is a user object or other smart terminal that has the smart home terminal control authority in the target interaction scene. Optionally, in an embodiment, the target object is a user object corresponding to a user identifier bound to each smart home terminal.

[0077] Optionally, the terminal control device can collect scene information of the target interaction scene using a first sensor to obtain scene environment data. The first sensor is a sensor for collecting scene environment parameters in the target interaction scene. Optionally, in an embodiment, the first sensor can be a temperature sensor, a humidity sensor, an illumination sensor, an air quality sensor (e.g., a PM2.5 sensor, a carbon dioxide detection sensor, etc.), or other sensors for collecting multi-modal scene environment parameters in the target interaction scene. The terminal control device collects multi-modal environment information in the target interaction scene in real time using the first sensor to obtain scene environment data.

[0078] Optionally, the terminal control device can also collect object behavior of the target interaction object using a second sensor to obtain object behavior data. The second sensor is a detection sensor for collecting behavior data of the target object in the target interaction scene to obtain object behavior data, and is used to capture multi-modal behavior data such as posture images and voice data of the target object in the target interaction scene. Optionally, the second sensor can be a camera module, an infrared sensor, a motion sensor (e.g., a gyroscope and a body sensor, etc.), a voice sensor, or other sensors capable of collecting behavior of the target object. The object behavior data includes object posture interaction data and object voice interaction data. The object posture interaction data is interaction control data triggered by gesture information and other posture information of the target object. The object voice interaction data is interaction control data sent by the target object through voice triggering. For example, in an embodiment, the terminal control device detects object voice interaction data “the house is too stuffy” from the target object through the second sensor, determines the object voice interaction data as object behavior data, and automatically analyzes multi-modal environment data in a subsequent step to generate corresponding control instructions to link air conditioners, fresh air equipment, windows, and other devices to actively initiate interaction, thereby improving the intelligence of the smart home system.

[0079] Optionally, the terminal control device generates multi-modal scene data of the target interaction scene according to the scene environment data and the object behavior data. That is, the terminal control device and / or the target perception layer of the target processing model can use a signal conditioning circuit to perform pre-processing such as amplification and filtering on the scene environment data and the object behavior data, obtain pre-processed scene environment data and object behavior data, use a digital-to-analog conversion chip to convert analog signals in the scene environment data and the object behavior data into data signals, and encrypt and check the scene environment data and the object behavior data during data transmission to ensure data transmission accuracy, thereby generating multi-modal scene data.

[0080] 202. generating a target decision instruction according to the target processing model and the multi-modal scene data;

[0081] Optionally, the terminal control device generates a target decision instruction according to the target processing model and the multi-modal scene data after obtaining the multi-modal scene data. That is, the terminal control device drives the target decision layer composed of the target processing model and the target intelligent agent (AIAgent), receives the multi-modal scene data using the target decision layer, and performs scene reasoning using the multi-modal scene data to determine the target decision instruction corresponding to the multi-modal scene data. The target processing model has strong semantic understanding and language generation capabilities by pre-training a large amount of text data. In the target decision layer, the target processing model receives the multi-modal scene data extracted by the input processing module, performs semantic analysis and understanding, and converts natural language information into computer-processable semantic representations. At the same time, the target processing model combines pre-set rules and models to reason and predict the target interaction scene, providing more accurate decision basis for subsequent decision reasoning. In the interaction with the user, the target processing model can understand the user's voice instructions and text information, generate natural and fluent replies and suggestions, and realize intelligent interaction.

[0082] The target decision instruction is a general device control instruction based on multi-modal scene data to drive smart home to perform corresponding operations. That is, the target decision instruction is a standardized format instruction defined by the target control interface. For example, in one specific embodiment, the target decision instruction is a UDCI (Unified Device Control Interface) instruction.

[0083] Optionally, after obtaining the multi-modal scene data, the terminal control device extracts parameters from the multi-modal scene data using a target processing model to obtain target interaction parameters. That is, the terminal control device uses the target processing model and the corresponding target agent to analyze the collected multi-modal scene data, thereby extracting key scene information in the multi-modal scene data that can be used as interaction decision data, and determining the key scene information as the target interaction parameters. Optionally, the target interaction parameters are key scene information in the scene environment data and object behavior data that can cause the smart home system to make corresponding function control. For example, in a specific embodiment, the target interaction parameters can be key temperature information, key humidity information, object quantity information, object behavior information, device operation information of each smart home device, indoor brightness information, and air quality information in the multi-modal scene data that can cause the smart home system to perform corresponding interaction operations. Optionally, in a specific embodiment, the terminal control device uses the target agent to collect target interaction parameters as temperature: 37 degrees Celsius; humidity: 75%; indoor: 2 people; TV on time: 5 minutes; indoor brightness: low, and the like.

[0084] Optionally, after obtaining the target interaction parameters, the terminal control device further analyzes the target interaction parameters using the target processing model to obtain the initial decision instructions. That is, the terminal control device can use the target processing model or the target agent to perform multi-layer decision matching on the target interaction parameters, thereby obtaining each initial decision instruction corresponding to the target interaction parameters, and determining the target decision instruction based on each initial decision instruction.

[0085] Optionally, the terminal control device can perform multi-layer decision matching on the target interaction parameters by using the local scene library corresponding decision rule information, the cloud-stored historical decision parameters and corresponding historical decision instructions, and the target processing model, thereby obtaining each initial decision instruction corresponding to the target interaction parameters.

[0086] Optionally, after obtaining the initial decision instructions, the terminal control device further converts the initial decision instructions into a target decision instruction according to a target control interface. That is, the terminal control device converts the obtained initial decision instructions into a standardized format instruction defined by the target control interface using the target control interface, and determines the standardized format instruction as the target decision instruction.

[0087] The target control interface is a control interface for defining a standardized format of a device control instruction. Optionally, in an embodiment, the target control interface can be integrated into the target processing model. The standardized format instruction can be in JSON Schema format. For example, in an embodiment, the initial decision instruction is

turn on air conditioner 26 degrees high fan speed

{"action":"turn on","device":"air onditioner","temperature":26,"fan_speed":"high"}

[0088] Optionally, the terminal control device splits the initial decision instruction using the target processing model to obtain initial decision sub-instructions corresponding to each target home terminal, i.e., the terminal control device splits the initial decision instruction according to the terminal identifier of each target home terminal using the target processing model and the terminal identifier of each target home terminal, thereby obtaining decision instruction information corresponding to each target home terminal, and determining the decision instruction information as the initial decision sub-instruction. The initial decision sub-instruction is a decision sub-instruction that needs to be executed by the corresponding target home terminal. For example, in an embodiment, the initial decision sub-instruction is

turn on air conditioner 26 degrees high fan speed

turn on first indoor lighting high brightness

[0089] Optionally, after obtaining the initial decision sub-instruction corresponding to each target home terminal, the terminal control device converts the initial decision sub-instruction according to the target control interface and the terminal identifier corresponding to the target home terminal to obtain a converted decision sub-instruction. That is, the terminal control device extracts and converts the initial decision sub-instruction to obtain key decision information represented in a computer-processable semantic representation (such as feature encoding, etc.), and inputs the key decision information of the initial decision sub-instruction and the terminal identifier into the field corresponding to the standardized format of the target control interface according to the standardized format of the target control interface, to obtain the converted decision sub-instruction. For example, in an embodiment, the converted decision instruction can be

{"action":"turn on","device":"air onditioner","temperature":26,"fan_speed":"high"}

[0090] Optionally, the terminal control device aggregates the conversion decision sub-instructions to obtain a target decision instruction after obtaining the conversion decision sub-instructions. That is, the terminal control device aggregates the conversion decision sub-instructions corresponding to the same target home terminal after obtaining the conversion decision sub-instructions corresponding to each target home terminal, to obtain a target decision instruction corresponding to each target home terminal.

[0091] 203. converting the target decision instruction into a target control instruction corresponding to the target home terminal through a private protocol;

[0092] Optionally, the terminal control device further converts the target decision instruction into a target control instruction corresponding to the target home terminal through a private protocol using a target agent after obtaining the target decision instruction. That is, the terminal control device converts the target decision instruction into a target control instruction corresponding to the private protocol of the target home terminal using a target agent.

[0093] Optionally, the terminal control device determines the target home terminal corresponding to each target decision instruction and a protocol adaptation framework corresponding to the target home terminal, and converts the target decision instruction into a target control instruction corresponding to the private protocol of the target home terminal through the protocol adaptation framework. The protocol adaptation framework is a protocol conversion framework for converting a target decision instruction in a standardized format into a target control instruction in a format corresponding to the private protocol of the target home terminal. Optionally, in an embodiment, the protocol adaptation framework is a model context protocol (MCP) framework.

[0094] 204. driving the target home terminal to execute the target control instruction to obtain a terminal control result.

[0095] Optionally, the terminal control device sends the target control instruction to the target home terminal after obtaining the target control instruction corresponding to each target home terminal, and drives the target home terminal to execute the target control instruction to obtain a terminal control result.

[0096] Optionally, the terminal control device sends the target control instruction to the target home terminal, and the target home terminal identifies the target control instruction through a target private protocol and converts the target control instruction that is successfully identified into a machine control code corresponding to the target home terminal to drive the target home terminal to execute the target control instruction and obtain a terminal control result. In this way, the user can use a high-performance intelligent device at home to use a target processing model and a target agent as a central control device of an intelligent home system to understand an intention and make a scene-based decision, thereby actively and heuristically controlling multiple intelligent home terminals with multiple independent ecosystems and multiple private protocols, improving the response speed of the device, and improving the intelligence of device interaction.

[0097] In the embodiment, the terminal control device acquires multi-modal scene data in a target interactive scene associated with the smart home system, generates a target decision instruction according to a target processing model and the multi-modal scene data, converts the target decision instruction into a target control instruction corresponding to a target home terminal according to a private protocol, and drives the target home terminal to execute the target control instruction to obtain a terminal control result. In the smart home system with multiple smart home terminals in multiple ecologies, multi-modal scene data in a target interactive scene associated with the smart home system is actively acquired, the multi-modal scene data is analyzed by using a target processing model to generate a target decision instruction, the target decision instruction is converted into a target control instruction corresponding to a private protocol instruction of each smart home terminal, and the target control instruction is used to actively drive the target home terminal to execute a corresponding interactive operation, so that the smart home terminals are actively interacted, the degree of intelligence and the interactive initiative of the smart home system are improved, and cross-ecology collaborative linkage of the smart home terminals is achieved.

[0098] As shown in Figure 3 , the terminal control method provided by the embodiment of the application includes the following steps: Figure 3 As shown in the flowchart of one embodiment of generating an initial decision instruction in the terminal control method provided by the embodiment of the application, Figure 3 In the embodiment, the terminal control method further includes steps 301-304:

[0099] 301. Matching the target interactive parameter with a decision rule information corresponding to the smart home system to obtain a first decision instruction;

[0100] 302. Matching the target interactive parameter with a historical decision parameter corresponding to a target object to determine a target historical parameter in the historical decision parameter and a second decision instruction corresponding to the target historical parameter;

[0101] 303. Analyzing the target interactive parameter by using the target processing model to obtain a third decision instruction;

[0102] 304. Determining an initial decision instruction according to the first decision instruction, the second decision instruction, and the third decision instruction.

[0103] Based on the above embodiment, in the embodiment, the terminal control device further analyzes the target interactive parameter by using a target processing model to obtain the initial decision instruction after acquiring the target interactive parameter. That is, the terminal control device can use the target processing model or a target intelligent agent to perform multi-layer decision matching on the target interactive parameter, thereby obtaining each initial decision instruction corresponding to the target interactive parameter, and determining a target decision instruction based on each initial decision instruction.

[0104] Optionally, the terminal control device can perform multi-layer decision matching on the target interaction parameter through the local scene library, the historical decision parameter and the corresponding historical decision instruction stored in the cloud, and the target processing model, so as to obtain each initial decision instruction corresponding to the target interaction parameter.

[0105] Optionally, the terminal control device can obtain a local scene library pre-stored with preset decision instructions and associated decision rule information corresponding to the smart home system, obtain each decision rule information stored in the local scene library, and perform instruction matching on the target interaction parameter according to the decision rule information corresponding to the smart home system, to obtain a first decision instruction. That is, there are preset scene parameters in the decision rule information, and the terminal control device compares the preset scene parameters with the target interaction parameter, so as to determine the decision rule information with the same or similar preset scene parameters and target interaction parameters as the hit decision information, and determine the preset decision instruction corresponding to the hit decision information as the first decision instruction. Optionally, in a specific embodiment, the target interaction parameter is

temperature: 37 degrees Celsius; humidity: 75%; number of people: 2; TV on time: 5 minutes; indoor brightness: low

turn on the air conditioner, turn on the air purifier, close the doors and windows, and perform display and reminder on the TV

[0106] Optionally, the terminal control device can also obtain a target object corresponding to the target interaction parameter, and obtain historical decision parameters and historical decision instructions corresponding to the target object. The historical decision parameter is a multi-modal scene parameter representing the target object when outputting the historical decision instruction. The historical decision instruction is an interaction operation instruction executed by the target object in the past period.

[0107] Optionally, the terminal control device uploads the target interaction parameter to the cloud server, matches the target interaction parameter with the historical decision parameter, so as to determine the historical decision parameter with a confidence greater than 90% with the target interaction parameter as the target historical parameter associated with the target interaction parameter, and determine the historical decision instruction corresponding to the target historical parameter as the second decision instruction. Optionally, in a specific embodiment, the second decision instruction obtained by the terminal control device is

turn on the air conditioner, turn off the sound box, and turn on the indoor lighting

[0108] Optionally, the terminal control device also uses the target processing model to perform instruction analysis on the target interaction parameter to obtain a third decision instruction, that is, the terminal control device inputs the target interaction parameter and heuristic interaction prompt information into the target processing model, uses the target processing model to perform data analysis and decision determination according to the heuristic interaction prompt information and the target interaction parameter, removes hallucination from the decision information, and generates a third decision instruction corresponding to the target interaction parameter. The heuristic interaction prompt information is a large model prompt used to guide the target processing model to perform scene analysis and decision instruction generation according to the target interaction parameter.

[0109] Optionally, after obtaining the first decision instruction, the second decision instruction, and the third decision instruction, the terminal control device uses the target agent to identify and merge repeated decision instructions of the first decision instruction, the second decision instruction, and the third decision instruction to generate a target decision instruction. That is, the terminal control device uses the target agent to identify abnormal decision instructions in the first decision instruction, the second decision instruction, and the third decision instruction that are irrelevant to the target interaction parameter, and merge repeated decision instructions in the first decision instruction, the second decision instruction, and the third decision instruction, to generate an initial decision instruction, and convert the initial decision instruction into a target decision instruction in a standardized format in a subsequent step.

[0110] In this embodiment, the terminal control device obtains a first decision instruction by performing instruction matching on the target interaction parameter according to the corresponding decision rule information of the smart home system, matches the target interaction parameter and a historical decision parameter corresponding to the target object to determine a target historical parameter in the historical decision parameter and a second decision instruction corresponding to the target historical parameter, uses the target processing model to perform instruction analysis on the target interaction parameter to obtain a third decision instruction, and determines an initial decision instruction according to the first decision instruction, the second decision instruction, and the third decision instruction. This implementation realizes scene analysis and decision generation of the target interaction parameter in multiple dimensions, and improves the accuracy and efficiency of initial decision instruction generation.

[0111] As shown in Figure 4 , the terminal control method provided in this embodiment includes steps 401-402. Figure 4 As shown in the embodiment, Figure 4 , the terminal control method further includes steps 401-402:

[0112] 401, using the protocol adaptation framework to perform instruction information extraction on the target decision instruction to obtain target instruction information;

[0113] 402. Map the target instruction information using the target private protocol corresponding to the protocol adaptation framework to obtain a target control instruction corresponding to the target home terminal.

[0114] Based on the above embodiment, in this embodiment, after obtaining the target decision instruction, the terminal control device also uses the target agent to convert the target decision instruction into a private protocol to obtain the target control instruction corresponding to the target home terminal. In other words, the terminal control device uses the target agent to convert the target decision instruction into a target control instruction corresponding to the private protocol of the target home terminal.

[0115] Optionally, after receiving the target decision instruction, the terminal control device extracts instruction information from the target decision instruction locally or in the cloud to obtain target instruction information. That is, the terminal control device can locally extract instruction information context from the target decision instruction using the protocol adaptation framework. Optionally, to reduce computing power consumption and improve extraction efficiency, the terminal control device uses the protocol adaptation framework to send the target decision instruction to a protocol adaptation server, which then uses the protocol adaptation server to extract instruction information context from the target decision instruction to obtain target instruction information. The target instruction information is the interactive instruction action information within the target decision instruction that controls the target home terminal. For example, the target instruction information may include startup or shutdown instruction information, parameter adjustment instruction information, and mode switching instruction information.

[0116] Optionally, after obtaining the target command information, the terminal control device uses the target private protocol corresponding to the protocol adaptation framework to perform protocol mapping on the target command information, converts the target command information into private command information that can be read by the target home device, and combines the private command information with the private command template to generate the target control command corresponding to the target home terminal.

[0117] In this embodiment, the terminal control device extracts instruction information from the target decision instruction using the protocol adaptation framework to obtain target instruction information. It then maps the target instruction information using the target proprietary protocol corresponding to the protocol adaptation framework to obtain the target control instruction corresponding to the target home terminal. This converts the standardized protocol into the proprietary target control instruction corresponding to each target home terminal, and then uses the target control instruction to actively drive the target home terminal to perform the corresponding interactive operation, thereby achieving cross-ecosystem collaborative linkage among smart homes.

[0118] In order to better implement the terminal control method in the embodiment of the present application, based on the terminal control method, the embodiment of the present application also provides a terminal control device, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an embodiment of a terminal control device provided in an embodiment of the present application. Specifically, the terminal control device 500 includes:

[0119] The data collection module 501 is configured to collect multi-modal scene data in a target interaction scene associated with the smart home system;

[0120] The decision generation module 502 is configured to generate a target decision instruction according to a target processing model and the multi-modal scene data;

[0121] The protocol conversion module 503 is configured to perform private protocol conversion on the target decision instruction to obtain a target control instruction corresponding to a target home terminal;

[0122] The active interaction module 504 is configured to drive the target home terminal to execute the target control instruction to obtain a terminal control result.

[0123] In a possible implementation manner of the embodiment, the multi-modal scene data in the terminal control device includes scene environment data and object behavior data.

[0124] The collection of the multi-modal scene data in the target interaction scene associated with the smart home system includes:

[0125] Scene information of the target interaction scene is collected by using a first sensor to obtain scene environment data.

[0126] Object behavior of a target interaction object is collected by using a second sensor to obtain object behavior data.

[0127] The multi-modal scene data of the target interaction scene is generated according to the scene environment data and the object behavior data.

[0128] In a possible implementation manner of the embodiment, the terminal control device generates a target decision instruction according to a target processing model and the multi-modal scene data, including:

[0129] The target interaction parameters are extracted from the multi-modal scene data by using the target intelligent agent.

[0130] The target interaction parameters are analyzed by using the target processing model to obtain an initial decision instruction.

[0131] The initial decision instruction is converted into a target decision instruction according to a target control interface.

[0132] In a possible implementation manner of the embodiment, the terminal control device analyzes the target interaction parameters by using the target processing model to obtain an initial decision instruction, including:

[0133] According to the smart home system, the corresponding decision rule information and the target interaction parameter are matched to obtain a first decision instruction;

[0134] The target interaction parameter and the historical decision parameter corresponding to the target object are matched to determine a target historical parameter in the historical decision parameter, and a second decision instruction corresponding to the target historical parameter;

[0135] The target processing model is used for instruction analysis on the target interaction parameter to obtain a third decision instruction;

[0136] According to the first decision instruction, the second decision instruction and the third decision instruction, an initial decision instruction is determined.

[0137] In a possible implementation manner of the embodiment, a terminal control device converts the initial decision instruction into a target decision instruction according to a target control interface, including:

[0138] The initial decision instruction is split to obtain each initial decision sub-instruction corresponding to each target home terminal;

[0139] The initial decision sub-instruction is converted according to the target control interface and a terminal identifier of the target home terminal to obtain a converted decision sub-instruction;

[0140] The converted decision sub-instructions are summarized to obtain the target decision instruction.

[0141] In a possible implementation manner of the embodiment, a terminal control device converts the target decision instruction into a target control instruction corresponding to the target home terminal, including:

[0142] A target home terminal corresponding to the target decision instruction is determined, and a protocol adaptation framework corresponding to the target home terminal is determined;

[0143] The target decision instruction is converted into the target control instruction corresponding to the target home terminal by using the protocol adaptation framework.

[0144] In a possible implementation manner of the embodiment, a terminal control device converts the target decision instruction into a target control instruction corresponding to the target home terminal by using the protocol adaptation framework, including:

[0145] The target decision instruction is extracted by using the protocol adaptation framework to obtain target instruction information;

[0146] The target instruction information is mapped by using a target private protocol corresponding to the protocol adaptation framework to obtain the target control instruction corresponding to the target home terminal.

[0147] In the embodiment, the terminal control device collects multi-modal scene data in a target interactive scene associated with the smart home system, generates a target decision instruction according to a target processing model and the multi-modal scene data, converts the target decision instruction into a target control instruction corresponding to a target home terminal according to a private protocol, and drives the target home terminal to execute the target control instruction to obtain a terminal control result. In the smart home system with multiple smart home terminals in multiple ecologies, the multi-modal scene data in the target interactive scene associated with the smart home system is actively collected, the multi-modal scene data is analyzed by using the target processing model to generate the target decision instruction, the target decision instruction is converted into the target control instruction corresponding to the private protocol instruction of each smart home terminal, and the target control instruction is used to actively drive the target home terminal to execute the corresponding interactive operation, so that the smart home terminals in multiple ecologies are cooperatively linked, the smart home terminals actively interact, and the intelligent degree and interactive initiative of the smart home system are improved.

[0148] The embodiment of the present application also provides a terminal control device, as shown in the accompanying drawings. Figure 6 The terminal control device provided in the embodiment of the present application is shown in the accompanying drawings. Figure 6 The terminal control device provided in the embodiment of the present application is shown in the accompanying drawings.

[0149] The terminal control device provided in the embodiment of the present application is shown in the accompanying drawings.

[0150] one or more processors;

[0151] a memory; and

[0152] one or more application programs, wherein the one or more application programs are stored in the memory and configured to execute the steps of the terminal control method in any of the terminal control method embodiments by the processor.

[0153] Specifically, the terminal control device can include a processor 601 with one or more processing cores, a memory 602 with one or more computer readable storage media, a power supply 603, and an input unit 604. Those skilled in the art can understand that the terminal control device structure shown in the accompanying drawings does not constitute a limitation on the terminal control device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them: Figure 6

[0154] ​The processor 601 is the control center of the terminal control device, connects the various parts of the terminal control device by using various interfaces and lines, executes various functions of the terminal control device and processes data by running or executing software programs and / or modules stored in the memory 602 and calling data stored in the memory 602, and thus monitors the terminal control device as a whole. Optionally, the processor 601 can include one or more processing cores; preferably, the processor 601 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 601.

[0155] The memory 602 can be used to store software programs and modules, and the processor 601 executes various functions and data processing by running the software programs and modules stored in the memory 602. The memory 602 can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the terminal control device, etc. In addition, the memory 602 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 602 can also include a memory controller to provide the processor 601 with access to the memory 602.

[0156] The terminal control device also includes a power supply 603 for supplying power to various components, and preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management, etc. through the power management system. The power supply 603 can also include one or more than one direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator, and any other components.

[0157] The terminal control device can also include an input unit 604, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0158] Although not shown, the terminal control device can also include a display unit, etc., which will not be described here. Specifically, in the present embodiment, the processor 601 in the terminal control device will load the executable file corresponding to the process of one or more than one application program into the memory 602 according to the following instructions, and run the application program stored in the memory 602 by the processor 601, so as to realize various functions, such as:

[0159] Collecting multi-modal scene data in a target interaction scene associated with the smart home system;

[0160] Generating a target decision instruction according to a target processing model and the multi-modal scene data;

[0161] Converting the target decision instruction into a target control instruction corresponding to a target home terminal through a private protocol;

[0162] Driving the target home terminal to execute the target control instruction to obtain a terminal control result.

[0163] To this end, the embodiment of the present application provides a computer readable storage medium, which can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. A computer program is stored on the computer readable storage medium, and the computer program is loaded by a processor to execute steps in any terminal control method provided by the embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0164] Collecting multi-modal scene data in a target interaction scene associated with the smart home system;

[0165] Generating a target decision instruction according to a target processing model and the multi-modal scene data;

[0166] Converting the target decision instruction into a target control instruction corresponding to a target home terminal through a private protocol;

[0167] Driving the target home terminal to execute the target control instruction to obtain a terminal control result.

[0168] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the detailed description of other embodiments above, which will not be repeated here.

[0169] In a specific implementation, each of the above units or structures can be implemented as an independent entity, or can be combined as the same or several entities, and the specific implementation of each unit or structure can be referred to the method embodiments above, which will not be repeated here.

[0170] The specific implementation of each operation can be referred to the above embodiments, which will not be repeated here.

[0171] The terminal control method provided by the embodiments of the present application is described in detail above, and the principles and implementation manners of the present application are described by using specific embodiments. The above embodiment descriptions are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A terminal control method characterized by comprising: The application is applied to a smart home system, the smart home system comprises at least one smart home terminal, and the terminal control method comprises the following steps: Collecting multi-modal scene data in a target interaction scene associated with the smart home system; Generating a target decision instruction according to a target processing model and the multi-modal scene data; Converting the target decision instruction into a target control instruction corresponding to a target home terminal through a private protocol; Driving the target home terminal to execute the target control instruction to obtain a terminal control result.

2. The terminal control method according to claim 1, characterized by, The multi-modal scene data comprises scene environment data and object behavior data; The collecting of the multi-modal scene data in the target interaction scene associated with the smart home system comprises the following steps: Collecting scene information of the target interaction scene through a first sensor to obtain the scene environment data; Collecting object behavior of a target interaction object through a second sensor to obtain the object behavior data; Generating the multi-modal scene data of the target interaction scene according to the scene environment data and the object behavior data.

3. The terminal control method according to claim 1, characterized by, The generating of the target decision instruction according to the target processing model and the multi-modal scene data comprises the following steps: Extracting target interaction parameters from the multi-modal scene data through a target intelligent agent; Analyzing the target interaction parameters through the target processing model to obtain an initial decision instruction; Converting the initial decision instruction into a target decision instruction through a target control interface.

4. The terminal control method according to claim 3, characterized by, The analyzing of the target interaction parameters through the target processing model to obtain the initial decision instruction comprises the following steps: Matching the target interaction parameters with corresponding decision rule information of the smart home system to obtain a first decision instruction; Matching the target interaction parameters with historical decision parameters corresponding to a target object to determine a target historical parameter in the historical decision parameters and a second decision instruction corresponding to the target historical parameter; Analyzing the target interaction parameters through the target processing model to obtain a third decision instruction; Determining the initial decision instruction according to the first decision instruction, the second decision instruction and the third decision instruction.

5. The terminal control method according to claim 3, characterized by, The converting of the initial decision instruction into the target decision instruction through the target control interface comprises the following steps: Splitting the initial decision instruction to obtain initial decision sub-instructions corresponding to each target home terminal; Converting the initial decision sub-instructions through the target control interface and terminal identifiers of the target home terminals to obtain converted decision sub-instructions; Summarizing the converted decision sub-instructions to obtain the target decision instruction.

6. The terminal control method according to claim 1, characterized by, The converting of the target decision instruction into a target control instruction corresponding to a target home terminal through a private protocol comprises the following steps: Determining the target home terminal corresponding to the target decision instruction and a protocol adaptation framework corresponding to the target home terminal; Converting the target decision instruction into the target control instruction corresponding to the target home terminal through the protocol adaptation framework.

7. The terminal control method according to claim 6, characterized by, The private protocol conversion of the target decision instruction by using the protocol adaptation framework comprises: The target instruction information is extracted from the target decision instruction by using the protocol adaptation framework, to obtain target instruction information; The target instruction information is mapped by using a target private protocol corresponding to the protocol adaptation framework, to obtain the target control instruction corresponding to the target home terminal.

8. A terminal control device, characterized by comprising: The terminal control device comprises: a data acquisition module configured to acquire multi-modal scene data in a target interactive scene associated with the smart home system; a decision generation module configured to generate a target decision instruction according to a target processing model and the multi-modal scene data; a protocol conversion module configured to perform private protocol conversion on the target decision instruction to obtain a target control instruction corresponding to a target home terminal; an active interaction module configured to drive the target home terminal to execute the target control instruction to obtain a terminal control result.

9. A terminal control device characterized by comprising: The terminal control device comprises: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the steps of the terminal control method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps of the terminal control method in any one of claims 1 to 7.

Images