Control instruction execution method and device, storage medium and electronic equipment

Abstract

The invention discloses a control instruction execution method and device, a storage medium and electronic equipment, and relates to the technical field of smart home. The execution method of the control instruction comprises the steps of predicting a displacement vector of a first coordinate in a second screen according to dynamic change information of a first screen under the condition that the first screen of the intelligent control equipment dynamically changes and a click operation on the first coordinate of the first screen is received; performing coordinate compensation on the first coordinate according to the displacement vector to obtain a second coordinate of the first coordinate in the second screen, and determining a target control corresponding to the second coordinate and a control instruction of home equipment corresponding to the target control; and sending the control instruction to the home equipment to instruct the home equipment to execute the control instruction. By adopting the technical scheme, the problem that the home equipment cannot be accurately controlled to execute the control instruction due to inaccurate judgment on the coordinate position of the target control when the screen of the intelligent control equipment dynamically changes in the related technology is solved.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a method and apparatus for executing control commands, a storage medium, and an electronic device. Background Technology

[0002] In a smart home environment, smart control devices (such as smartphones and tablets) serve as the interface for users to interact with various home appliances (such as smart lights, smart curtains, and smart air conditioners), carrying out key functions such as sending control commands and receiving device status information. However, in practical applications, when users rotate the smart control device from portrait to landscape mode, or use the split-screen function or zoom on the smart control device, the position of the controls changes, causing user clicks to become ineffective or result in misoperation, thus preventing the correct control of home appliances.

[0003] There is currently no effective solution to the problem that the coordinate position of the target control is not accurately determined when the screen of the smart control device changes dynamically, thus making it impossible to accurately control the home device to execute control commands. Summary of the Invention

[0004] This application provides a method and apparatus for executing control commands, a storage medium, and an electronic device to at least solve the problem in the related art where the coordinate position of the target control is not accurately determined when the screen of a smart control device changes dynamically, thus making it impossible to accurately control home devices to execute control commands.

[0005] According to one embodiment of this application, a method for executing control instructions is provided, comprising: when a first screen of a smart control device undergoes dynamic change and a click operation on a first coordinate of the first screen is received, predicting a displacement vector of the first coordinate in a second screen based on dynamic change information of the first screen, wherein the second screen is the dynamically changed first screen, and the dynamic change information is used to characterize the positional changes of multiple controls in the first screen during the process of the first screen changing into the second screen; performing coordinate compensation on the first coordinate based on the displacement vector to obtain a second coordinate of the first coordinate in the second screen, and determining a target control corresponding to the second coordinate and a control instruction for a home appliance corresponding to the target control, wherein the multiple controls include the target control, and the smart control device is used to send the control instruction to the home appliance; and sending the control instruction to the home appliance to instruct the home appliance to execute the control instruction.

[0006] In an optional embodiment, before predicting the displacement vector of the first coordinate in the second screen based on the dynamic change information of the first screen, the method further includes: acquiring interaction data of multiple first controls in the intelligent control device and multiple first dynamic change information of the first screen within a first time period, wherein the interaction data is time-series data, the multiple first controls include the target control, and the first time period is a period of time before the current time; training a long short-term memory network to be trained based on the multiple interaction data and the multiple first dynamic change information to obtain a trained long short-term memory network, wherein the trained long short-term memory network is used to predict the displacement vector.

[0007] In one optional embodiment, predicting the displacement vector of the first coordinate in the second screen based on the dynamic change information of the first screen includes: constructing a first sequence based on the dynamic change information and the first coordinate; inputting the first sequence into the trained long short-term memory network; and determining the displacement vector based on the output of the trained long short-term memory network.

[0008] In an optional embodiment, sending the control command to the home device includes: determining a target communication protocol based on the device name of the home device, and determining a first protocol type and a first version number of the target communication protocol; determining a protocol parsing module matching the target communication protocol in a device protocol library based on the first protocol type and the first version number, wherein the protocol parsing module is configured in the smart control device; and sending the control command to the home device through the protocol parsing module.

[0009] In an optional embodiment, determining a protocol parsing module matching the target communication protocol in the device protocol library based on the first protocol type and the first version number includes: determining whether multiple first protocol parsing modules exist in the device protocol library based on the first protocol type, wherein the second protocol type of the multiple first protocol parsing modules is the same as the first protocol type; if it is determined that the multiple first protocol parsing modules exist in the device protocol library, determining whether a second protocol parsing module exists among the multiple first protocol parsing modules based on the first version number, wherein the second version number of the second protocol parsing module is the same as the first version number; if it is determined that the second protocol parsing module exists among the multiple first protocol parsing modules, identifying the second protocol parsing module as the protocol parsing module.

[0010] In an optional embodiment, after sending the control command to the home device through the protocol parsing module, the method further includes: continuously monitoring the communication status between the protocol parsing module and the home device; if the communication status is interrupted during a second time period and the number of interruptions is greater than a preset value, determining that the performance of the protocol parsing module has degraded, and sending a request command to the target object, wherein the request command is used to request the target object to adjust the protocol parsing module, and the second time period is a period of time after the current time.

[0011] In an optional embodiment, after sending a request instruction to the target object, the method further includes: receiving a version rollback instruction from the target object based on the request instruction, wherein the version rollback instruction is used to instruct the protocol parsing module to be rolled back; executing the version rollback instruction to determine a fourth version number protocol parsing module in the device protocol library based on the third version number of the protocol parsing module, wherein the fourth version number is the previous version number of the third version number; and communicating with the home device through the fourth version number protocol parsing module.

[0012] According to another aspect of the present invention, a control command execution device is also provided, comprising: a prediction module, configured to predict a displacement vector of the first coordinate in a second screen based on dynamic change information of the first screen when a first screen of a smart control device undergoes dynamic change and a click operation on a first coordinate of the first screen is received, wherein the second screen is the dynamically changed first screen, and the dynamic change information is used to characterize the positional changes of a plurality of controls in the first screen during the process of the first screen changing into the second screen; a determination module, configured to perform coordinate compensation on the first coordinate based on the displacement vector to obtain a second coordinate of the first coordinate in the second screen, and determine a target control corresponding to the second coordinate and a control command of a home appliance corresponding to the target control, wherein the plurality of controls includes the target control, and the smart control device is configured to send the control command to the home appliance; and an execution module, configured to send the control command to the home appliance to instruct the home appliance to execute the control command.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the execution method of the above-described control instructions when it is run.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the execution method of the control instructions through the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described method for executing control instructions.

[0016] In this embodiment, when the first screen of the smart control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received, the displacement vector of the first coordinate in the second screen is predicted based on the dynamic change information of the first screen. The second screen is the dynamically changed first screen, and the dynamic change information characterizes the positional changes of multiple controls in the first screen during the transformation from the first screen to the second screen. Coordinate compensation is performed on the first coordinate based on the displacement vector to obtain the second coordinate of the first coordinate in the second screen. The target control corresponding to the second coordinate and the control command for the home appliance corresponding to the target control are then determined. The multiple controls include the target control, and the smart control device sends the control command to the home appliance. The control command is sent to the home appliance to instruct it to execute the control command. This technical solution solves the problem in related technologies where the coordinate position of the target control is inaccurately determined when the smart control device screen dynamically changes, thus preventing accurate control of the home appliance to execute control commands. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the hardware environment for a method of executing control instructions according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a method for executing control instructions according to an embodiment of this application;

[0021] Figure 3 This is a first schematic diagram of a method for executing control instructions according to an optional embodiment of this application;

[0022] Figure 4 This is a second schematic diagram of a method for executing control instructions according to an optional embodiment of this application;

[0023] Figure 5 This is a structural block diagram of a control instruction execution device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] According to one aspect of the embodiments of this application, a method for executing control instructions is provided. This method for executing control instructions is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned method for executing control instructions can be applied to, for example... Figure 1 The hardware environment shown consists of intelligent control device 102 and computer terminal 104. Figure 1As shown, the intelligent control device 102 is connected to the computer terminal 104 via a network. It can be used to provide services (such as application services) to the intelligent control device 102 or clients installed on the intelligent control device 102. A database can be set up on or independently of the intelligent control device 102 to provide data storage services for the intelligent control device 102 and the computer terminal 104. Cloud computing and / or edge computing services can be configured on or independently of the intelligent control device 102 to provide data processing services for the intelligent control device 102 and the computer terminal 104.

[0027] The aforementioned networks may include, but are not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The intelligent control device 102 and computer terminal 104 may not be limited to PCs, mobile phones, tablets, smart air conditioners, smart range hoods, smart refrigerators, smart ovens, smart stoves, smart washing machines, smart water heaters, smart washing equipment, smart dishwashers, smart projectors, smart TVs, smart clothes racks, smart curtains, smart audio-visual equipment, smart sockets, smart speakers, smart speakers, smart fresh air systems, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaners, smart window cleaning robots, smart mopping robots, smart air purifiers, smart steam ovens, smart microwave ovens, smart water heaters, smart air purifiers, smart water dispensers, system platforms, etc.

[0028] This embodiment provides a method for executing control commands, applied to the aforementioned intelligent control device. Figure 2 This is a flowchart of a method for executing control instructions according to an embodiment of this application, the process including the following steps:

[0029] Step S202: When the first screen of the intelligent control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received, the displacement vector of the first coordinate in the second screen is predicted based on the dynamic change information of the first screen. The second screen is the first screen after the dynamic change, and the dynamic change information is used to characterize the position changes of multiple controls in the first screen during the process of the first screen changing into the second screen.

[0030] Step S204: Perform coordinate compensation on the first coordinate according to the displacement vector to obtain the second coordinate of the first coordinate in the second screen, and determine the target control corresponding to the second coordinate and the control command of the home device corresponding to the target control, wherein the plurality of controls include the target control, and the smart control device is used to send the control command to the home device;

[0031] Step S206: Send the control command to the home appliance to instruct the home appliance to execute the control command.

[0032] In this embodiment, when the first screen of the smart control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received, the displacement vector of the first coordinate in the second screen is predicted based on the dynamic change information of the first screen. The second screen is the dynamically changed first screen, and the dynamic change information characterizes the positional changes of multiple controls in the first screen during the transformation from the first screen to the second screen. Coordinate compensation is performed on the first coordinate based on the displacement vector to obtain the second coordinate of the first coordinate in the second screen. The target control corresponding to the second coordinate and the control command for the home appliance corresponding to the target control are then determined. The multiple controls include the target control, and the smart control device sends the control command to the home appliance. The control command is sent to the home appliance to instruct it to execute the control command. This technical solution solves the problem in related technologies where the coordinate position of the target control is inaccurately determined when the smart control device screen dynamically changes, thus preventing accurate control of the home appliance to execute control commands.

[0033] In an optional embodiment, before predicting the displacement vector of the first coordinate in the second screen based on the dynamic change information of the first screen, the method further includes: acquiring interaction data of multiple first controls in the intelligent control device and multiple first dynamic change information of the first screen within a first time period, wherein the interaction data is time-series data, the multiple first controls include the target control, and the first time period is a period of time before the current time; training a long short-term memory network to be trained based on the multiple interaction data and the multiple first dynamic change information to obtain a trained long short-term memory network, wherein the trained long short-term memory network is used to predict the displacement vector.

[0034] In one optional embodiment, predicting the displacement vector of the first coordinate in the second screen based on the dynamic change information of the first screen includes: constructing a first sequence based on the dynamic change information and the first coordinate; inputting the first sequence into the trained long short-term memory network; and determining the displacement vector based on the output of the trained long short-term memory network.

[0035] In smart home control scenarios, to overcome the problem of inaccurate control coordinate position judgment caused by dynamic changes in the screen of smart control devices, interaction data between users and multiple controls (including target controls) over a period of time, as well as dynamic change information of the smart device screen, such as screen rotation, scaling, and split-screen, are extracted from historical operation records. This interaction data and dynamic change information exist in the form of time-series data, recording the correlation between control position changes and user operations. Using this time-series data as a training set, a Long Short-Term Memory (LSTM) network is trained using the backpropagation algorithm. LSTM can learn and remember long-term dependencies in control position changes. The training process iteratively optimizes the network parameters until LSTM can accurately predict control displacement based on historical data. Once LSTM training is complete, when a new dynamic change occurs on the smart control device screen, a sequence is constructed based on the dynamic change information of the first screen and the first coordinate clicked on the first screen. This sequence is then input into the LSTM, which outputs a predicted displacement vector from the first screen to the second screen based on the learned pattern.

[0036] Optionally, assuming that when a user clicks on the first coordinate in the first screen on an Android tablet, the tablet rotates from portrait to landscape mode, the LSTM receives the dynamic change information of the first screen (portrait mode) and the first coordinate (100, 200), predicts the displacement vector (50, -30) of the first coordinate from the first screen to the second screen (landscape mode), and performs coordinate compensation on the first coordinate based on the displacement vector to obtain the second coordinate (150, 170) of the first coordinate in the second screen.

[0037] In an optional embodiment, sending the control command to the home device includes: determining a target communication protocol based on the device name of the home device, and determining a first protocol type and a first version number of the target communication protocol; determining a protocol parsing module matching the target communication protocol in a device protocol library based on the first protocol type and the first version number, wherein the protocol parsing module is configured in the smart control device; and sending the control command to the home device through the protocol parsing module.

[0038] In an optional embodiment, determining a protocol parsing module matching the target communication protocol in the device protocol library based on the first protocol type and the first version number includes: determining whether multiple first protocol parsing modules exist in the device protocol library based on the first protocol type, wherein the second protocol type of the multiple first protocol parsing modules is the same as the first protocol type; if it is determined that the multiple first protocol parsing modules exist in the device protocol library, determining whether a second protocol parsing module exists among the multiple first protocol parsing modules based on the first version number, wherein the second version number of the second protocol parsing module is the same as the first version number; if it is determined that the second protocol parsing module exists among the multiple first protocol parsing modules, identifying the second protocol parsing module as the protocol parsing module.

[0039] First, the target communication protocol used by the home appliance is identified based on its device name, thus determining the specific protocol type and version number (i.e., the first protocol type and first version number). A pre-built device protocol library is queried to check for multiple first protocol parsing modules with the same protocol type as the target communication protocol. If multiple first protocol parsing modules are found, their version numbers are compared to find a second protocol parsing module that perfectly matches the target communication protocol version. This is because even for the same type of communication protocol, different versions may have subtle differences or compatibility issues. Once a second protocol parsing module with a matching version number is confirmed in the device protocol library, it is selected as the protocol parsing module matching the target communication protocol.

[0040] Optionally, suppose a user uses a smart home application to control smart devices from different brands in their home. When the user attempts to control a smart air conditioner named "CoolAC-110," the application first queries the device information database based on the device name "CoolAC-110" to determine that the air conditioner uses "CoolHome Protocol" as its communication protocol, and that the protocol version is 2.1. Next, the application accesses the built-in device protocol library. After searching for the "CoolHomeProtocol" protocol type, it finds multiple first protocol parsing modules corresponding to this type, including CoolHome Protocol Version 2.0, CoolHome Protocol Version 2.1, and even earlier or newer versions. Among these first protocol parsing modules, it filters for second protocol parsing modules that have the same first version number (i.e., 2.1). In this example, "CoolHome Protocol Version 2.1" is the second protocol parsing module that meets the criteria. Ultimately, after confirming the existence of the "CoolHome Protocol Version 2.1" module, it was identified as the protocol parsing module that handles communication with the "CoolAC-110" smart air conditioner.

[0041] In an optional embodiment, after sending the control command to the home device through the protocol parsing module, the method further includes: continuously monitoring the communication status between the protocol parsing module and the home device; if the communication status is interrupted during a second time period and the number of interruptions is greater than a preset value, determining that the performance of the protocol parsing module has degraded, and sending a request command to the target object, wherein the request command is used to request the target object to adjust the protocol parsing module, and the second time period is a period of time after the current time.

[0042] In an optional embodiment, after sending a request instruction to the target object, the method further includes: receiving a version rollback instruction from the target object based on the request instruction, wherein the version rollback instruction is used to instruct the protocol parsing module to be rolled back; executing the version rollback instruction to determine a fourth version number protocol parsing module in the device protocol library based on the third version number of the protocol parsing module, wherein the fourth version number is the previous version number of the third version number; and communicating with the home device through the fourth version number protocol parsing module.

[0043] During the operation of the smart home system, the communication status with home devices is continuously monitored to ensure communication stability and the effectiveness of the protocol parsing module. The number of communication interruptions within a certain period (the second time period) is recorded. If the number of interruptions exceeds a preset value, it is determined that the performance of the currently used protocol parsing module has degraded, failing to accurately parse or process the device's communication protocol, thus affecting the user's control experience of the home devices. At this point, a request command is automatically sent to the target object (such as the system administrator or remote technical support center). The request command includes the type of protocol parsing module with degraded performance, its current version number, and a version rollback request. Upon receiving the request command, the target object analyzes the performance problem of the current protocol parsing module, assesses whether a rollback to a previous stable version is necessary, and then responds with a version rollback command. Upon receiving the version rollback command, the system searches for the previous version of the protocol parsing module (the fourth version) in the device protocol library. Subsequently, the currently degraded protocol parsing module is uninstalled, and the fourth version protocol parsing module is loaded, attempting to restore communication with the home devices using the fourth version protocol parsing module.

[0044] Optionally, taking a smart light using the Zigbee 3.0 protocol as an example, when using the Zigbee 3.0 protocol parsing module V3.1 for control, it was detected that the communication status was interrupted 10 times in the past 12 hours, exceeding the preset interruption value of 5 times. This indicates a performance degradation in the protocol parsing module V3.1, and an automatic request command is sent to the administrator. The request command includes the protocol parsing module type (Zigbee 3.0 protocol parsing module), the current version number (V3.1), and a request for version rollback. After analyzing the request command, the administrator discovered a known compatibility issue in version V3.1 and therefore issued a version rollback command, instructing a rollback to the previous version V3.0. Upon receiving the version rollback command, the V3.1 version of the Zigbee 3.0 protocol parsing module is uninstalled from the device protocol library, and the V3.0 version of the Zigbee 3.0 protocol parsing module is loaded to re-establish communication with the smart light. Since the V3.0 version module is known to be stable and compatible with the smart light, the communication status is restored to normal, and the user can continue to accurately control the smart light's on / off state and brightness adjustment. This avoids device control failures caused by performance issues of the protocol parsing module, ensuring the smooth operation of the smart home system and the continuity of user operation.

[0045] To better understand the process of executing the above control instructions, the following description, in conjunction with optional embodiments, further illustrates the execution method flow of the above control instructions, but is not intended to limit the technical solutions of the embodiments of this application.

[0046] Figure 3 This is a first schematic diagram of a method for executing control instructions according to an optional embodiment of the present invention, as shown below. Figure 3 As shown, it specifically includes the following:

[0047] When a user controls a cross-brand Android smart device (equivalent to a home appliance) by clicking on a first coordinate on the first screen (current screen) in a smart control device (such as a mobile phone or tablet), and the user dynamically changes the first screen (e.g., rotating, scaling, splitting the screen), the first coordinate and the dynamic change information of the first screen are transmitted to the dynamic user interface (UI) adaptation module. This dynamic UI adaptation module includes an LSTM prediction unit and a click coordinate compensation unit. The LSTM prediction unit predicts the displacement vector of the first coordinate from the first screen to the second screen based on the dynamic change information and the first coordinate. The click coordinate compensation unit compensates for the first coordinate based on the displacement vector, obtaining the second coordinate of the first coordinate on the second screen. Optionally, assuming that when a user clicks on the first coordinate in the first screen on an Android tablet, the tablet rotates from portrait to landscape mode, the LSTM receives the dynamic change information of the first screen (portrait mode) and the first coordinate (100, 200), predicts the displacement vector (50, -30) of the first coordinate from the first screen to the second screen (landscape mode), and performs coordinate compensation on the first coordinate based on the displacement vector to obtain the second coordinate (150, 170) of the first coordinate in the second screen.

[0048] After obtaining the first coordinate and its second coordinate on the second screen, the target control corresponding to the second coordinate and the control command for the corresponding home appliance are determined. Sending the control command relies on a multi-protocol conversion middleware, which handles the conversion of different communication protocols between smart control devices and cross-brand Android smart devices. The protocol parsing module loader within the multi-protocol conversion middleware dynamically loads the most suitable protocol parsing module based on the smart appliance's access status and protocol update requirements, enabling flexible protocol conversion. The protocol version management unit monitors the performance of the protocol parsing module. If a newly loaded protocol parsing module experiences compatibility or stability issues, a version rollback mechanism is immediately triggered, reverting to the previous stable version to ensure stable and uninterrupted system communication.

[0049] Therefore, based on the target communication protocol of the home appliance, the protocol parsing module loader determines the matching protocol parsing module in the device protocol library and loads it into the multi-protocol conversion middleware. The middleware then returns the version number of the protocol parsing module to the protocol version management unit. Simultaneously, control commands are sent to the home appliance through the protocol parsing module to instruct it to execute. If multiple attempts to send control commands fail, the protocol version management unit rolls back the protocol parsing module to the previous stable version and sends control commands to the home appliance again. After the commands are executed, the user is notified of the result. The specific interaction flow is as follows: Figure 4 As shown.

[0050] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, operating terminal, or network device, etc.) to execute the methods of the various embodiments of this application.

[0051] Figure 5 This is a structural block diagram of a control instruction execution device according to an embodiment of this application; as shown... Figure 5 As shown, it includes:

[0052] The prediction module 52 is used to predict the displacement vector of the first coordinate in the second screen based on the dynamic change information of the first screen when the first screen of the intelligent control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received. The second screen is the first screen after the dynamic change, and the dynamic change information is used to characterize the position changes of multiple controls in the first screen during the process of the first screen changing into the second screen.

[0053] The determining module 54 is used to perform coordinate compensation on the first coordinate according to the displacement vector to obtain the second coordinate of the first coordinate in the second screen, and to determine the target control corresponding to the second coordinate and the control command of the home device corresponding to the target control, wherein the plurality of controls include the target control, and the smart control device is used to send the control command to the home device;

[0054] The execution module 56 is used to send the control command to the home appliance to instruct the home appliance to execute the control command.

[0055] In this embodiment, when the first screen of the smart control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received, the displacement vector of the first coordinate in the second screen is predicted based on the dynamic change information of the first screen. The second screen is the dynamically changed first screen, and the dynamic change information characterizes the positional changes of multiple controls in the first screen during the transformation from the first screen to the second screen. Coordinate compensation is performed on the first coordinate based on the displacement vector to obtain the second coordinate of the first coordinate in the second screen. The target control corresponding to the second coordinate and the control command for the home appliance corresponding to the target control are then determined. The multiple controls include the target control, and the smart control device sends the control command to the home appliance. The control command is sent to the home appliance to instruct it to execute the control command. This technical solution solves the problem in related technologies where the coordinate position of the target control is inaccurately determined when the smart control device screen dynamically changes, thus preventing accurate control of the home appliance to execute control commands.

[0056] In an optional embodiment, the prediction module 52 is further configured to acquire interaction data of multiple first controls in the intelligent control device and multiple first dynamic change information of the first screen within a first time period, wherein the interaction data is time-series data, the multiple first controls include the target control, and the first time period is a period of time before the current time; and to train the long short-term memory network to be trained based on the multiple interaction data and the multiple first dynamic change information to obtain a trained long short-term memory network, wherein the trained long short-term memory network is used to predict the displacement vector.

[0057] In an optional embodiment, the prediction module 52 is further configured to construct a first sequence based on the dynamic change information and the first coordinates; input the first sequence into the trained long short-term memory network; and determine the displacement vector based on the output of the trained long short-term memory network.

[0058] In an optional embodiment, the execution module 56 is further configured to determine a target communication protocol based on the device name of the home device, and to determine a first protocol type and a first version number of the target communication protocol; determine a protocol parsing module matching the target communication protocol in the device protocol library based on the first protocol type and the first version number, wherein the protocol parsing module is set in the smart control device; and send the control command to the home device through the protocol parsing module.

[0059] In an optional embodiment, the execution module 56 is further configured to determine whether there are multiple first protocol parsing modules in the device protocol library based on the first protocol type, wherein the second protocol type of the multiple first protocol parsing modules is the same as the first protocol type; if it is determined that there are multiple first protocol parsing modules in the device protocol library, determine whether there is a second protocol parsing module among the multiple first protocol parsing modules based on the first version number, wherein the second version number of the second protocol parsing module is the same as the first version number; if it is determined that there is a second protocol parsing module among the multiple first protocol parsing modules, identify the second protocol parsing module as the protocol parsing module.

[0060] In an optional embodiment, the execution module 56 is further configured to continuously monitor the communication status between the protocol parsing module and the home device; if the communication status is interrupted during a second time period and the number of interruptions is greater than a preset value, the execution module determines that the performance of the protocol parsing module has degraded, and sends a request instruction to the target object, wherein the request instruction is used to request the target object to adjust the protocol parsing module, and the second time period is a period of time after the current time.

[0061] In an optional embodiment, the execution module 56 is further configured to receive a version rollback instruction from the target object based on the request instruction, wherein the version rollback instruction is used to instruct the protocol parsing module to be rolled back; execute the version rollback instruction to determine a fourth version number protocol parsing module in the device protocol library based on the third version number of the protocol parsing module, wherein the fourth version number is the previous version number of the third version number; and communicate with the home device through the fourth version number protocol parsing module.

[0062] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0063] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0064] S1, when the first screen of the intelligent control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received, the displacement vector of the first coordinate in the second screen is predicted based on the dynamic change information of the first screen, wherein the second screen is the first screen after the dynamic change, and the dynamic change information is used to characterize the position changes of multiple controls in the first screen during the process of the first screen changing into the second screen.

[0065] S2, perform coordinate compensation on the first coordinate according to the displacement vector to obtain the second coordinate of the first coordinate in the second screen, and determine the target control corresponding to the second coordinate and the control command of the home device corresponding to the target control, wherein the plurality of controls include the target control, and the smart control device is used to send the control command to the home device;

[0066] S3, send the control command to the home appliance to instruct the home appliance to execute the control command.

[0067] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0068] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0069] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0070] S1, when the first screen of the intelligent control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received, the displacement vector of the first coordinate in the second screen is predicted based on the dynamic change information of the first screen, wherein the second screen is the first screen after the dynamic change, and the dynamic change information is used to characterize the position changes of multiple controls in the first screen during the process of the first screen changing into the second screen.

[0071] S2, perform coordinate compensation on the first coordinate according to the displacement vector to obtain the second coordinate of the first coordinate in the second screen, and determine the target control corresponding to the second coordinate and the control command of the home device corresponding to the target control, wherein the plurality of controls include the target control, and the smart control device is used to send the control command to the home device;

[0072] S3, send the control command to the home appliance to instruct the home appliance to execute the control command.

[0073] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0074] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0075] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0076] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0077] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0078] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0079] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for executing control instructions, characterized in that, include: When the first screen of the intelligent control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received, the displacement vector of the first coordinate in the second screen is predicted based on the dynamic change information of the first screen. The second screen is the first screen after the dynamic change, and the dynamic change information is used to characterize the position changes of multiple controls in the first screen during the process of the first screen changing into the second screen. The first coordinate is compensated according to the displacement vector to obtain the second coordinate of the first coordinate in the second screen, and the target control corresponding to the second coordinate and the control command of the home device corresponding to the target control are determined. The plurality of controls include the target control, and the smart control device is used to send the control command to the home device. Send the control command to the home appliance to instruct the home appliance to execute the control command.

2. The method for executing control commands according to claim 1, characterized in that, Before predicting the displacement vector of the first coordinate in the second screen based on the dynamic change information of the first screen, the method further includes: The interaction data of multiple first controls in the intelligent control device and multiple first dynamic change information of the first screen are acquired within a first time period. The interaction data is time-series data, the multiple first controls include the target control, and the first time period is a period of time before the current time. The Long Short-Term Memory (LSTM) network to be trained is trained based on multiple interaction data and multiple first dynamic change information to obtain a trained LSM network, wherein the trained LSM network is used to predict the displacement vector.

3. The method for executing control commands according to claim 2, characterized in that, Predicting the displacement vector of the first coordinate in the second screen based on the dynamic change information of the first screen includes: A first sequence is constructed based on the dynamic change information and the first coordinates; The first sequence is input into the trained Long Short-Term Memory (LSTM) network, and the shift vector is determined based on the output of the trained LSTM network.

4. The method for executing control instructions according to claim 1, characterized in that, Sending the control command to the home appliance includes: The target communication protocol is determined based on the device name of the home appliance, and the first protocol type and first version number of the target communication protocol are determined accordingly. Based on the first protocol type and the first version number, a protocol parsing module matching the target communication protocol is determined in the device protocol library, wherein the protocol parsing module is set in the intelligent control device; The control command is sent to the home appliance through the protocol parsing module.

5. The method for executing control commands according to claim 4, characterized in that, Based on the first protocol type and the first version number, a protocol parsing module is determined in the device protocol library to match the target communication protocol, including: Based on the first protocol type, determine whether there are multiple first protocol parsing modules in the device protocol library, wherein the second protocol type of the multiple first protocol parsing modules is the same as the first protocol type; If it is determined that the plurality of first protocol parsing modules exist in the device protocol library, it is determined whether there is a second protocol parsing module among the plurality of first protocol parsing modules according to the first version number, wherein the second version number of the second protocol parsing module is the same as the first version number; If it is determined that the second protocol parsing module exists among the plurality of first protocol parsing modules, the second protocol parsing module is identified as the protocol parsing module.

6. The method for executing control commands according to claim 4, characterized in that, After sending the control command to the home appliance through the protocol parsing module, the method further includes: Continuously monitor the communication status between the protocol parsing module and the home device; If the communication status is interrupted during the second time period, and the number of interruptions exceeds a preset value, it is determined that the performance of the protocol parsing module has degraded, and a request instruction is sent to the target object. The request instruction is used to request the target object to adjust the protocol parsing module. The second time period is a period of time after the current time.

7. The method for executing control instructions according to claim 6, characterized in that, After sending a request instruction to the target object, the method further includes: Receive a version rollback instruction from the target object based on the request instruction, wherein the version rollback instruction is used to instruct the protocol parsing module to be rolled back; Execute the version rollback instruction to determine the fourth version number of the protocol parsing module in the device protocol library based on the third version number of the protocol parsing module, wherein the fourth version number is the version number preceding the third version number; The protocol parsing module of the fourth version number communicates with the home device.

8. A control command execution device, characterized in that, include: The prediction module is used to predict the displacement vector of the first coordinate in the second screen based on the dynamic change information of the first screen when the first screen of the intelligent control device undergoes dynamic changes and a click operation on the first coordinate of the first screen is received. The second screen is the first screen after the dynamic change, and the dynamic change information is used to characterize the position changes of multiple controls in the first screen during the process of the first screen changing into the second screen. The determination module is used to perform coordinate compensation on the first coordinate according to the displacement vector to obtain the second coordinate of the first coordinate in the second screen, and determine the target control corresponding to the second coordinate and the control command of the home device corresponding to the target control, wherein the plurality of controls include the target control, and the smart control device is used to send the control command to the home device; An execution module is used to send the control command to the home appliance to instruct the home appliance to execute the control command.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images