Cross-device interaction method and device
Through the cross-device interaction method, the cumbersome APP usage and data security issues under the diversified IoT devices are solved, and a low-cost and secure cross-device control experience is achieved.
Patent Information
- Application Number
- CN202110400031.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-04-14
AI Technical Summary
In the case of diversified IoT devices, users need to install APPs from multiple manufacturers, which are cumbersome to use, high operation and maintenance costs, and data transmission on the public network is likely to cause leakage and illegal collection.
Through cross-device interaction methods, business logic and UI are separated, business logic and UI are separated across devices, reducing development difficulties, reducing user usage costs, and improving data security. All interactions are implemented through UI, and the main control device does not need to expose the business API. .
It reduces the development difficulty and operation and maintenance costs of manufacturers, improves user experience, and enhances data security. Users can control the equipment of different manufacturers without downloading multiple APPs from different manufacturers.
Smart Images

Figure CN115202546B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to a cross-device interaction method and device. Background Art
[0002] The development of the Internet of Things (IoT) has provided great convenience to people's lives. At present, users can remotely control IoT devices through applications (APPs) installed on terminal devices. Taking smart home devices as an example, Figure 1 As shown, the user can open the smart home APP installed on the terminal device, and the smart home APP will obtain and display various data of the associated smart home devices from the cloud platform. The user can also issue control instructions to the smart home devices through the smart home APP, and the smart home APP will send the control instructions to the smart home devices through the cloud platform to achieve control of the smart home devices.
[0003] However, when users purchase IoT devices from multiple manufacturers, they need to install multiple manufacturers’ apps on their terminal devices, which is cumbersome. Manufacturers face different APP platforms (such as / / web / applets, etc.), it is also necessary to maintain multiple APP operation and maintenance teams, which has high operation and maintenance costs. In addition, the transmission of data from IoT devices on the public network is prone to data leakage and illegal collection. Summary of the invention
[0004] The embodiments of the present application provide a cross-device interaction method and device to achieve cross-device business logic and UI separation, reduce the development difficulty of manufacturers, reduce user usage costs, and improve data security.
[0005] In the first aspect, an embodiment of the present application provides a cross-device interaction method, the method comprising: the electronic device displays a corresponding UI in response to user interface UI information from a main control device, the UI includes at least a first function option, and the first function option is associated with at least one user operation type; the electronic device sends user operation information to the main control device in response to a first user operation based on the first function option in the UI, the user operation information includes at least identification information of the first function option and the type of the first user operation, and the user operation information is used to trigger the main control device to perform a control operation or management operation corresponding to the user operation information. Optionally, the user operation information also includes user input information obtained based on the first user operation. The main control device may be an IoT device, etc.
[0006] In the embodiment of the present application, the business code and UI code are all run in the main control device, and the manufacturer does not need to develop multiple APPs for different APP platforms, which can reduce the development difficulty and save costs. In addition, the main control device does not need to expose the business API to the outside, and all interactions are through the UI, which improves the security of the data. At the same time, users do not need to download APPs from different manufacturers on electronic devices. Only one display application is needed to control the main control devices of different manufacturers, which is conducive to reducing the user's usage cost.
[0007] In a possible design, the UI information and the user operation information are information in a domain-specific language DSL format.
[0008] With the above design, DSL (such as JSON format code, etc.) can be used to transmit UI and events (such as user operation information) between devices, which is conducive to the parsing and identification of UI and events between devices.
[0009] In one possible design, the method further includes: before the electronic device responds to the UI information from the main control device, it also includes: the electronic device establishes a communication connection with the main control device in response to a second user operation of selecting the main control device from the list of devices to be connected.
[0010] In the above design, the main control device can send its own UI information to the electronic device after detecting that a connection has been established with the electronic device, which helps to achieve separation of business logic and UI across the main control device and the electronic device.
[0011] In one possible design, after the electronic device sends the user operation information, it also includes: the electronic device receives UI update information from the main control device, and the UI update information at least includes the change information of the UI after the main control device performs the control operation or management operation corresponding to the user operation information; the electronic device updates the UI in response to the UI update information.
[0012] In the above design, after the main control device executes the control operation or management operation corresponding to the user operation information, the electronic device updates the displayed UI to the UI after the main control device executes the corresponding control operation or management operation according to the UI update information sent by the main control device, which is conducive to maintaining the consistency between the business logic of the main control device and the UI displayed by the electronic device.
[0013] In a second aspect, an embodiment of the present application provides a cross-device interaction method, the method comprising: a main control device sends user interface UI information to an electronic device, the UI corresponding to the UI information includes at least a first function option, and the first function option is associated with at least one user operation type; the main control device receives user operation information from the electronic device, the user operation information is obtained by the electronic device based on a first user operation of the first function option in the UI, and the user operation information at least includes identification information of the first function option and the type of the first user operation; the main control device performs a control operation or management operation corresponding to the operation information. Optionally, the user operation information also includes user input information obtained based on the first user operation.
[0014] In a possible design, the UI information and the user operation information are information in a domain-specific language DSL format.
[0015] In a possible design, the main control device sends UI information to the electronic device, including: after the main control device establishes a communication connection with the electronic device, sending the UI information to the electronic device.
[0016] In one possible design, after the main control device executes the control operation or management operation corresponding to the operation information, it also includes: the main control device sends UI update information to the electronic device, and the UI update information includes at least the change information of the UI after the main control device executes the control operation or management operation corresponding to the user operation information.
[0017] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor and a display screen; wherein the processor is connected to the memory and the display screen; one or more computer programs are stored in the memory, and the one or more computer programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes the method described in the first aspect or any possible design of the first aspect.
[0018] In a fourth aspect, an embodiment of the present application further provides a main control device, comprising: a memory and a processor; wherein the processor is connected to the memory; one or more computer programs are stored in the memory, and the one or more computer programs include instructions, and when the instructions are executed by the main control device, the main control device executes the method described in the above second aspect or any possible design of the second aspect.
[0019] In a fifth aspect, an embodiment of the present application further provides an electronic device, which includes functional modules capable of executing the above-mentioned first aspect or any possible design method of the first aspect; these functional modules can be implemented through hardware, or corresponding software can be implemented through hardware.
[0020] In the sixth aspect, an embodiment of the present application also provides a main control device, which includes functional modules that can execute the above-mentioned second aspect or any possible design method of the second aspect; these functional modules can be implemented through hardware, or the corresponding software can be implemented through hardware.
[0021] In the seventh aspect, an embodiment of the present application also provides a computer storage medium, including computer instructions. When the computer instructions are executed on a computer, the computer executes the method described in the first aspect or any possible design of the first aspect, or executes the method described in the second aspect or any possible design of the second aspect.
[0022] In an eighth aspect, an embodiment of the present application further provides a program product, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any possible design of the first aspect, or to execute the method described in the second aspect or any possible design of the second aspect.
[0023] In the ninth aspect, an embodiment of the present application provides a chip for calling and running a computer program stored in a memory from the memory, executing the above-mentioned first aspect or any possible design method of the first aspect, or executing the above-mentioned second aspect or any possible design method of the second aspect.
[0024] The technical effects that can be achieved in the above-mentioned second to ninth aspects can refer to the technical effects that can be achieved in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a schematic diagram of the smart home device control solution;
[0026] Figure 2 A schematic diagram of the architecture of a cross-device interaction system provided in an embodiment of the present application;
[0027] Figure 3 One of the schematic diagrams of the structure of an electronic device provided in an embodiment of the present application;
[0028] Figure 4 One of the electronic device interface schematic diagrams provided in the embodiment of the present application;
[0029] Figure 5A schematic diagram of a display application interface provided in an embodiment of the present application;
[0030] Figure 6 One of the view schematic diagrams provided in the embodiment of the present application;
[0031] Figure 7 One of the UI schematic diagrams provided in the embodiment of the present application;
[0032] Figure 8 The second UI schematic diagram provided for the embodiment of the present application;
[0033] Fig. 9 A schematic diagram of the architecture of an electronic device and a smart switch provided in an embodiment of the present application;
[0034] Fig.10 A schematic diagram of a cross-device interaction method provided in an embodiment of the present application;
[0035] Fig.11 The second view diagram provided for the embodiment of the present application;
[0036] Fig.12 A second structural diagram of an electronic device provided in an embodiment of the present application;
[0037] Fig.13 A schematic diagram of the structure of the main control device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] The rise of IoT devices such as smart lights, smart sockets, and smart water meters has brought many conveniences to people's lives. However, after users purchase IoT devices from multiple manufacturers, they need to install multiple manufacturers' APPs on their terminal devices, which is cumbersome to use. Manufacturers also need to maintain multiple APP operation and maintenance teams for different APP platforms, which has high operation and maintenance costs. In addition, the transmission of data from IoT devices on the public network is prone to data leakage and illegal collection.
[0039] In view of this, the embodiment of the present application provides a cross-device interaction solution. By splitting the business logic and the user interface (UI), the application can be displayed and interacted across devices. Users can use any electronic device with a screen, such as a watch or a tablet, to control IoT devices. There is no need for manufacturers to maintain multiple APP operation and maintenance teams, and users do not need to download multiple APPs from different manufacturers, which reduces the difficulty of manufacturer development and improves user experience. In addition, all interactions are implemented through the UI, and IoT devices do not need to expose business application programming interfaces (APIs) to the outside world, which can improve the security of IoT device data.
[0040] like Figure 2 The following is a schematic diagram of the architecture of a possible cross-device interaction system provided by an embodiment of the present application. The system includes multiple master control devices (such as smart switches, smart lights, smart refrigerators, etc.) and electronic devices. The electronic devices can be equipped with different operating systems and have the same display application installed. The electronic devices and master control devices can communicate through WiFi, Bluetooth, cellular networks, etc. By splitting the business logic and UI of the master control device, displaying the UI of the master control device on the electronic device, and transmitting the UI and events (such as operations on functional options in the UI) between the master control device and the electronic device, it is possible to achieve interaction across the master control device and the electronic device, and control the master control device.
[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. In order to facilitate understanding by those skilled in the art, some terms in the embodiments of the present application will first be explained.
[0042] 1) Main control device: The main control device in the embodiment of the present application may be an IoT device, for example, it may be a smart home device such as a smart light, a smart socket, a smart switch, a smart air conditioner, a smart refrigerator, or other IoT devices such as a smart traffic light, a smart water meter, and a road side unit (RSU). In addition, the main control device in the embodiment of the present application may also be a device such as a television. It should be understood that the main control device in the embodiment of the present application may be a device without a screen or a device with a screen.
[0043] 2) JavaScript Object Notation (JSON) is a lightweight data exchange format. It is based on a subset of the JS specification developed by the European Computer Association and uses a text format that is completely independent of the programming language to store and represent data. The simplicity and clear hierarchical structure make JSON an ideal data exchange language. It is easy for people to read and write, and it is also easy for machines to parse and generate, and it effectively improves network transmission efficiency.
[0044] 3) Event-driven in the computer field. The so-called event-driven, in simple terms, means that the computer performs corresponding response operations based on the generated events. From the event perspective, the basic structure of an event-driven program consists of an event collector, an event sender, and an event processor. The event collector is specifically responsible for collecting all events, including events from user operations (such as mouse and keyboard events, etc.), events from hardware (such as clock events, etc.), and events from software (such as operating systems, events generated by application logic processing, etc.). The event sender is responsible for distributing the events collected by the collector to the target object. The event processor performs specific event response work.
[0045] 4) Electronic devices: The electronic devices in the embodiments of the present application may be portable terminal devices, such as mobile phones, tablet computers, portable computers, wearable electronic devices (such as smart watches, smart helmets, smart glasses, smart bracelets, etc.), VR devices, augmented reality (AR) devices, etc. For example, portable terminals include but are not limited to devices equipped with Hongmeng Or other operating systems. In addition, the electronic device of the embodiment of the present application may not be a portable terminal device, such as a display device with a screen such as a television, a personal computer (PC), etc.
[0046] For example, Figure 3 FIG. 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Figure 3 As shown, the electronic device includes a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a fingerprint sensor, a touch sensor, etc., and the audio module 170 may include a speaker, a microphone, an earphone interface, etc.
[0047] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices, or two or more different processing units may be integrated into one device.
[0048] The processor 110 may also be provided with a memory for storing computer programs and / or data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store computer programs and / or data that have just been used or are cyclically used by the processor 110. If the processor 110 needs to use the computer program and / or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0049] In some embodiments, the processor 110 may include one or more interfaces. For example, the processor 110 includes a universal serial bus (USB) interface 130 and a subscriber identity module (SIM) interface 195. For another example, the processor 110 may also include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), and / or a general-purpose input / output (GPIO) interface, etc.
[0050] It is understandable that the interface connection relationship between the modules illustrated in the embodiments of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0051] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge an electronic device, and can also be used to transfer data between an electronic device and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as augmented reality (AR) devices, etc.
[0052] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device can support 2 or N SIM card interfaces, where N is a positive integer greater than 2. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device and cannot be separated from the electronic device.
[0053] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of an electronic device. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.
[0054] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193 and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0055] The wireless communication function of the electronic device can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem, and the baseband processor.
[0056] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0057] The mobile communication module 150 can provide solutions for wireless communication including 2G / 3G / 4G / 5G / (subsequent evolution standards such as 6G) etc. applied to electronic devices. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA) etc.
[0058] The wireless communication module 160 includes wireless communication solutions that can be applied to electronic devices, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.
[0059] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).
[0060] The electronic device realizes the display function through the GPU, the display screen 194 and the application processor. The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), and the like. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0061] The electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor. ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0062] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.
[0063] The internal memory 121 includes a running memory and a built-in memory. Among them, the running memory can be used to store computer programs and / or data, etc. The processor 110 executes various functional applications and data processing of the electronic device by running the computer programs stored in the running memory. For example, the running memory may include a high-speed random access memory. The built-in memory can also be referred to as a built-in external memory, etc., which can be used to store computer programs and / or data. For example, the built-in memory can store operating systems, applications, etc. Electronic devices usually load computer programs and / or data in the built-in memory into the running memory, so that the processor 110 runs the corresponding computer programs and / or data to achieve corresponding functions. In addition, the internal memory 121 may include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0064] The electronic device can implement audio functions such as music playing and recording through the audio module 170 and the application processor.
[0065] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device may receive key input and generate key signal input related to user settings and function control of the electronic device.
[0066] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.
[0067] A touch sensor is also called a "touch panel". The touch sensor can be arranged on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor is used to detect touch operations acting on or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. A visual output related to the touch operation can be provided through the display screen. In other embodiments, the touch sensor can also be arranged on the surface of the electronic device, which is different from the position of the display screen 194.
[0068] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0069] In addition, it should be understood that in the embodiments of the present application, "at least one" refers to one or more. "Multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items.
[0070] In this application, "exemplary", "in some embodiments", "in other embodiments", etc. are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete way.
[0071] It should be pointed out that the words "first", "second", etc. involved in the embodiments of the present application are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.
[0072] The following takes the master control device as a smart switch in an IoT device as an example to introduce the cross-device interaction method of the present application.
[0073] For example, the application supporting cross-device interaction on the electronic device is a display application and the main control device is a smart switch. Figure 4 As shown in A in FIG. 1 , the main interface 410 is displayed on the display screen of the electronic device. When the user needs to control the smart switch, the user can click the icon 411 of the display application on the main interface 410. In response to the user clicking the icon 411 of the display application, the electronic device displays the interface of the display application on the display screen. For example, the interface of the display application can be as follows: Figure 4The interface 420 shown in B in FIG. The display interface includes three function options: WiFi 421, Bluetooth 422, and account 423, which can be implemented by controls. Among them, the function option WiFi 421 is used to control the display application to search for the main control device in the same WiFi network as the electronic device, and enter the management interface of the main control device in the same WiFi network as the electronic device; the function option Bluetooth 422 is used to control the display application to search for the main control device within the Bluetooth coverage of the electronic device, and enter the management interface of the main control device within the Bluetooth coverage of the electronic device; the function option account 423 is used to control the display application to communicate with the display application server through the cellular network, obtain the main control device associated with the user's personal account from the display application server, and enter the management interface of the main control device associated with the user's personal account. It can be understood that the interface 420 of the display application can also include other function options, such as function options for controlling the exit of the display application, function options for configuring the user's personal account, etc.
[0074] The user clicks Figure 4 Take the function option WiFi421 in the interface 420 shown in B as an example. Figure 5 As shown, the display application (that is, the electronic device) responds to the user clicking the function option WiFi 421, searches for the master device in the same WiFi network as the electronic device, and enters the management interface 510 of the master device in the same WiFi network as the electronic device. If the smart switch is already in the device list 511, the user can directly click the "smart switch" option in the device list 511, and the display application can respond to the user clicking the "smart switch" option and establish a WiFi connection with the smart switch. If the smart switch is not in the device list 511 of the master device, the user can click the search button 512, and the display application can respond to the user clicking the search button 512, re-search the master device in the same WiFi as the electronic device, and update the master device in the device list 511.
[0075] A service including UI code is running in the smart switch. For example, the UI code running in the smart switch includes information such as background color of rgba(30,144,255,0.7); function option (such as component) style of "button", attribute of "closed", supported (or associated) operation type of "click", etc. The specific UI code may be as follows:
[0076]
[0077]
[0078] When the smart switch determines to establish a communication connection with the electronic device or receives UI reporting information from the electronic device, the UI parsing engine in the smart switch can generate a domain specific language (DSL) from the UI (ie, UI code) description of the smart switch and send it to the electronic device.
[0079] As an example, the smart switch can generate JSON format code from the hml, Css, and Js parts of the UI code, where the hml and Css parts mainly describe the interface information of the UI of the smart switch, and the Js part mainly describes the operation types (such as single click, double click, etc.) supported by one or more functional options (such as components) in the UI of the smart switch. The following is an example of JSON format code generated (i.e. converted) from the above UI code description of the smart switch.
[0080]
[0081] After receiving the UI information from the smart switch, the display application on the electronic device can draw the UI of the smart switch in units of views. For example, one layer on the UI corresponds to one view, and the display application draws the graphics of each view, and then synthesizes the graphics of all views into one UI. Specifically, when constructing the view tree, the display application can consider the positional relationship between different views. For example, the UI information received by the display application is the above JSON format code. The display application parses the above JSON format code and constructs a view tree such as Figure 6 As shown, view1 corresponds to layer 1, view1 is located on the background, the background can be used as a separate view, namely viewroot, and view1 is the child node of viewroot. From the above JSON format code, we can see that the style (also called type) corresponding to view1 is "button", the attribute is "closed", the supported operation type is "click (onclick)", etc., the style corresponding to viewroot is "background (div)", the background color is "512819123", etc.
[0082] The display application lays out and renders each view in the view tree according to the view tree to synthesize a UI. Taking viewroot as an example, the display application obtains the color corresponding to 512819123, that is, rgba(30, 144, 255, 0.7), according to the corresponding style of "background (div)" and the background color of "512819123", and sets the background color to rgba(30, 144, 255, 0.7). Among them, rgb in rgba corresponds to red (red, R), green (green, G), and blue (blue, B), and a represents the alpha channel (alpha), which usually refers to transparency. Figure 7 The figure shows the UI displayed after the display application performs layout rendering according to the UI information.
[0083] When a user performs operations such as single-clicking or double-clicking on the UI displayed on an electronic device, the display application can determine the function option corresponding to the user operation on the UI based on the location of the operation, and determine whether the user operation is an operation type supported by the function option. If so, the display application generates a DSL with the identification information of the function option and the type of user operation (such as single-click, double-click, etc.), and sends it to the smart switch.
[0084] As an example, Figure 7 As shown, after the user clicks on a function option with a style of "button" on the UI displayed by the electronic device, the display application determines that the function option corresponding to the operation on the UI is a function option with a style of "button" according to the position of the operation, and the click operation belongs to the operation type supported by the function option. The display application generates a DSL with the identification information of the function option (such as id) and the type of user operation, such as "click operation (such as onclick)", and sends it to the smart switch. The specific DSL can be as follows.
[0085] {
[0086] “id”:“2-onclick”
[0087] }
[0088] After receiving the DSL, the smart switch parses the DSL and calls the processing function corresponding to the click operation on the function option with the style of "button", executes the opening instruction corresponding to the click operation on the function option with the style of "button", turns on the smart switch, and updates the property of the function option with the style of "button" to "turned on".
[0089]
[0090] Furthermore, the UI parsing engine of the smart switch converts the updated attribute information of the function option with the style of "button" into DSL and sends it to the electronic device. As shown below, the DSL includes information that the attribute of the function option with the style of "button" with id 2 is updated to "opened".
[0091]
[0092]
[0093] like Figure 8 As shown, the display application updates the property of the function option with the style of "button" in the UI from "closed" to "opened" according to the received DSL.
[0094] In another possible implementation, after the smart switch updates the property of the function option with the style of "button" to "turned on", the overall information of the updated UI of the smart switch can also be sent to the display application, and the display application updates the UI according to the updated UI information of the smart switch.
[0095] like Fig. 9 The diagram shows an architecture diagram of a smart switch and an electronic device provided by an embodiment of the present application. The interface layout description includes a code describing the interface information of the UI of the smart switch, and the logic code includes a code describing the event information of one or more functional options in the UI of the smart switch (such as the operation of the functional options). The code in the interface layout description is parsed by a UI parsing engine (such as a JS engine), and the smart switch can obtain the interface information of the UI of the smart switch. The smart switch can also obtain the interface information in the logic code (such as the operation type supported by one or more functional options). By encoding the interface information of the UI of the smart switch and the operation type supported by one or more functional options in the UI of the smart switch, it is converted into a DSL and sent to the display application of the electronic device. In addition, the smart switch can also send resources related to the UI, such as media (music, video, color, etc.) and images (such as pictures of functional options, etc.) to the display application of the electronic device.
[0096] The display application parses the received DSL to obtain the interface information of the UI of the smart switch and the operation types supported by one or more function options in the UI of the smart switch, and displays the UI of the smart switch through layout rendering.
[0097] When the display application detects a user operation on a target function option in the displayed UI, if the user operation belongs to the operation type supported by the target function option, the type of the user operation and the identification information of the target function option are encoded, converted into DSL and sent to the smart switch. The smart switch parses the received DSL, calls the processing function corresponding to the user operation on the target function option, executes the instruction corresponding to the user operation on the target function option, and encodes the attribute information of the updated target function option, converts it into DSL and feeds it back to the display application, so that the display application in the electronic device updates the displayed UI.
[0098] It should be understood that the steps performed by the smart switch in the implementation of the present application can also be implemented through an application in the smart switch, such as through a switch application installed in the smart switch.
[0099] The following embodiments introduce possible implementations of cross-device interaction performed by electronic devices and a main control device. Fig.10 A schematic diagram of a device interaction method provided by an embodiment of the present application is shown. Fig.10 As shown, the process of the method includes:
[0100] S1001: In response to a second user operation of selecting a master device from a list of devices to be connected, the electronic device establishes a communication connection with the master device.
[0101] In some embodiments, the electronic device can search for a master device in the same WiFi network as the electronic device through WiFi, Bluetooth, etc., or search for a master device within the Bluetooth coverage of the electronic device, or log in to the user's personal account to obtain the master device associated with the user's personal account through the server, and can establish a communication connection with the master device selected by the user according to the second user operation of selecting the master device from the list of devices to be connected. Taking the electronic device searching for a master device in the same WiFi network as the electronic device as an example, Figure 5 The device list 511 in FIG. 5 is an example of a possible list of devices to be connected.
[0102] As an example, Figure 5 As shown, after the user selects a master control device (such as a smart switch) that is in the same WiFi network as the electronic device, the electronic device can establish a communication connection with the master control device selected by the user through the WiFi network.
[0103] S1002: The main control device sends UI information to the electronic device, and the electronic device receives the UI information, wherein the UI corresponding to the UI information includes at least a first function option (such as a component), and the first function option is associated with at least one user operation type (such as single click, double click, etc.).
[0104] For example, when the master device detects that a communication connection is established with an electronic device, the UI parsing engine in the master device can generate DSL from the UI (i.e., UI code) description of the master device and send it to the electronic device. The specific implementation can refer to the implementation of the above-mentioned smart switch sending UI information to the electronic device, which will not be described in detail.
[0105] S1003: The electronic device displays a corresponding UI in response to the UI information.
[0106] In some implementations, after receiving UI information from the master device, the electronic device may first construct a view tree based on the UI information. Specifically, when constructing the view tree, the electronic device may consider the positional relationship between different views, where each view in the view tree represents a function option or background in the UI, and the view corresponding to the background may also be represented by viewroot. Fig.11 As shown, view1 corresponds to layer 1, view2 corresponds to layer 2, and view3 corresponds to layer 3. If view1, view2, and view3 are all located on viewroot, then view1, view2, and view3 can be child nodes of viewroot. If the positions of view1, view2, and view3 do not overlap, then view1, view2, and view3 can be the same layer. The electronic device renders each node (view) of the view tree in sequence according to the layout of the view tree to obtain a UI.
[0107] S1004: The electronic device sends user operation information to the main control device in response to a first user operation based on the first function option in the UI, and the main control device receives the user operation information.
[0108] The user operation information includes at least identification information of the first function option and a type of the first user operation, and the type of the first user operation belongs to a user operation type associated with (or supported by) the first function option.
[0109] As an example, when a user performs a first user operation on a first function option in the UI (such as a single-click operation), if the type of the first user operation belongs to the user operation type associated with the first function option, the electronic device can generate a DSL with the identification information of the first function option and the type description of the user's first user operation, and send it to the main control device.
[0110] In some possible implementations, the first user operation may also be a text input operation or a voice input operation, and the user operation information also includes user input information acquired by the electronic device based on the first user operation.
[0111] As an example, taking the first function option as a text box and the user performing a text input operation as an example, the user operation information sent by the electronic device to the main control device may also include information input by the user in the text box.
[0112] S1005: The main control device executes a control operation or a management operation corresponding to the operation information.
[0113] After receiving the operation information, the main control device can call the processing function corresponding to the user operation on the first function option, execute the instruction corresponding to the first user operation on the first function option, and perform the control operation or management operation corresponding to the operation information.
[0114] S1006: The main control device sends UI update information to the electronic device, and the electronic device receives the UI information.
[0115] The UI update information at least includes information about changes in the UI after the main control device performs a control operation or a management operation corresponding to the user operation information.
[0116] It should be understood that the UI update information may only include the UI change information after the main control device executes the control operation or management operation corresponding to the user operation information, such as the attribute information after the first function option is updated, and may also include the overall information of the new UI after the main control device executes the control operation or management operation corresponding to the user operation information. This application does not limit this.
[0117] S1007: The electronic device updates the UI in response to the UI update information.
[0118] As an example, see Figure 7 and Figure 8 , when the user clicks on the function option with the style of "button", the electronic device converts the identification information of the function option and the information of the single-click operation into DSL, and sends it to the smart switch (i.e., the main control device). After receiving the DSL, the smart switch parses the DSL, calls the processing function corresponding to the single-click operation on the function option with the style of "button", executes the opening instruction corresponding to the single-click operation on the function option with the style of "button", turns on the smart switch, updates the attribute of the function option with the style of "button" to "turned on", and converts the UI update information including updating the attribute of the function option with the style of "button" to "turned on" into DSL, and sends it to the electronic device. Based on the received DSL, the electronic device updates the attribute of the function option with the style of "button" in the UI from "turned off" to "turned on".
[0119] In the embodiments provided by the present application, the method provided by the embodiments of the present application is introduced from the perspective of the electronic device and the main control device as the execution subject. In order to realize the functions in the method provided by the embodiments of the present application, the electronic device or the main control device may include a hardware structure and / or a software module, and the above functions are realized in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether a function of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.
[0120] The following describes the device provided in the embodiments of the present application in conjunction with the accompanying drawings to implement the above method embodiments of the present application.
[0121] The present application also provides an electronic device, such as Fig.12 As shown, it includes one or more processors 1201 and one or more memories 1202. The memory 1202 stores one or more computer programs, and when the one or more computer programs are executed by the processor 1201, the electronic device executes the cross-device interaction method provided in the embodiment of the present application.
[0122] The electronic device may also include a display screen 1205 for displaying a graphical user interface, such as an interface of a media application. For example, the display screen 1205 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc.
[0123] Furthermore, in some embodiments, the electronic device may also include a camera 1203 and a microphone 1204 .
[0124] In addition, the electronic device in the embodiment of the present application may also include a speaker, a touch sensor, etc., which is not limited in the embodiment of the present application.
[0125] The present application also provides a main control device, such as Fig.13As shown, it includes one or more processors 1301 and one or more memories 1302. The memory 1302 stores one or more computer programs, and when the one or more computer programs are executed by the processor 1301, the main control device executes the cross-device interaction method provided in the embodiment of the present application.
[0126] Furthermore, in some embodiments, the main control device may also include a camera 1303 , a microphone 1304 , and a display screen 1305 .
[0127] In addition, the main control device in the embodiment of the present application may also include a speaker, a touch sensor, etc., which is not limited in the embodiment of the present application.
[0128] The connection medium between the processor, memory, camera, microphone and display screen is not limited in the embodiments of the present application. For example, the processor, memory, camera, microphone and display screen may be connected via a bus, which may be divided into an address bus, a data bus, a control bus, etc.
[0129] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0130] In the embodiment of the present application, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), etc., or a volatile memory (volatile memory), such as a random access memory (RAM). The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.
[0131] As used in the above embodiments, the term "when..." or "after..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrase "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.
[0132] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc. In the absence of conflict, the solutions of the above embodiments can be used in combination.
[0133] Note: A portion of this patent application document contains material which is subject to copyright protection. The copyright owner reserves all rights reserved except for the production of copies of the material in the patent file or patent record in the Patent Office.
Claims
1. A cross-device interaction method, characterized in that: include: The electronic device displays a corresponding UI in response to the user interface UI information from the main control device, wherein the UI includes at least a first function option, and the first function option is associated with at least one user operation type; The electronic device sends user operation information to the main control device in response to a first user operation based on the first function option in the UI, wherein the user operation information at least includes identification information of the first function option and a type of the first user operation, and the user operation information is used to trigger the main control device to perform a control operation or a management operation corresponding to the user operation information; After the electronic device sends the user operation information, the method further includes: The electronic device receives UI update information from the main control device, where the UI update information at least includes change information of the UI after the main control device performs a control operation or a management operation corresponding to the user operation information; The electronic device updates the UI in response to the UI update information; Among them, the main control device is an Internet of Things (IoT) device without a screen, and all business codes and UI codes run in the IoT device. The IoT device only sends the UI information or UI update information to the electronic device so that the electronic device displays the UI or updates the UI.
2. The method according to claim 1, characterized in that The user operation information also includes user input information acquired based on the first user operation.
3. The method according to claim 1 or 2, characterized in that The UI information and the user operation information are information in a domain specific language DSL format.
4. The method according to claim 1 or 2, characterized in that: Before the electronic device responds to the UI information from the main control device, the electronic device further includes: In response to a second user operation of selecting the master device from the list of devices to be connected, the electronic device establishes a communication connection with the master device.
5. A cross-device interaction method, characterized in that: include: The main control device sends user interface UI information to the electronic device, where the UI corresponding to the UI information includes at least a first function option, and the first function option is associated with at least one user operation type; The main control device receives user operation information from the electronic device, where the user operation information is acquired by the electronic device based on a first user operation of the first function option in the UI, and the user operation information at least includes identification information of the first function option and a type of the first user operation; The main control device performs a control operation or a management operation corresponding to the operation information; After the main control device executes the control operation or management operation corresponding to the operation information, the method further includes: The main control device sends UI update information to the electronic device, wherein the UI update information at least includes change information of the UI after the main control device performs a control operation or a management operation corresponding to the user operation information; Among them, the main control device is an Internet of Things (IoT) device without a screen, and all business codes and UI codes run in the IoT device. The IoT device only sends the UI information or UI update information to the electronic device so that the electronic device displays the UI or updates the UI.
6. The method according to claim 5, characterized in that The user operation information also includes user input information acquired based on the first user operation.
7. The method according to claim 5 or 6, characterized in that The UI information and the user operation information are information in a domain specific language DSL format.
8. The method according to claim 5 or 6, characterized in that The main control device sends UI information to the electronic device, including: After establishing a communication connection with the electronic device, the main control device sends UI information to the electronic device.
9. An electronic device, characterized in that: The electronic device comprises: A memory, a processor and a display screen, wherein the processor is connected to the memory and the display screen; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, cause the electronic device to perform the following steps: In response to user interface UI information from the main control device, display a corresponding UI, wherein the UI includes at least a first function option, and the first function option is associated with at least one user operation type; In response to a first user operation based on the first function option in the UI, sending user operation information to the main control device, the user operation information at least including identification information of the first function option and a type of the first user operation, the user operation information being used to trigger the main control device to perform a control operation or a management operation corresponding to the user operation information; When the instruction is executed by the electronic device, after the electronic device sends the user operation information, it further performs the following steps: Receiving UI update information from the master control device, wherein the UI update information at least includes change information of the UI after the master control device performs a control operation or a management operation corresponding to the user operation information; In response to the UI update information, updating the UI; Among them, the main control device is an Internet of Things (IoT) device without a screen, and all business codes and UI codes run in the IoT device. The IoT device only sends the UI information or UI update information to the electronic device so that the electronic device displays the UI or updates the UI.
10. The electronic device according to claim 9, characterized in that: The user operation information also includes user input information acquired based on the first user operation.
11. The electronic device according to claim 9 or 10, characterized in that: The UI information and the user operation information are information in a domain specific language DSL format.
12. The electronic device according to claim 9 or 10, characterized in that: When the instruction is executed by the electronic device, before the electronic device responds to the UI information from the main control device, the electronic device further performs the following steps: In response to a second user operation of selecting the master device from the list of devices to be connected, a communication connection is established with the master device.
13. A main control device, characterized in that: The main control device comprises: A memory and a processor, wherein the processor is connected to the memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the main control device, cause the main control device to perform the following steps: Sending user interface UI information to an electronic device, wherein the UI corresponding to the UI information includes at least a first function option, and the first function option is associated with at least one user operation type; receiving user operation information from the electronic device, where the user operation information is obtained by the electronic device based on a first user operation of the first function option in the UI, and the user operation information includes at least identification information of the first function option and a type of the first user operation; Execute a control operation or a management operation corresponding to the operation information; When the instruction is executed by the main control device, after the main control device executes the control operation or management operation corresponding to the operation information, the following steps are further performed: Sending UI update information to the electronic device, wherein the UI update information at least includes change information of the UI after performing a control operation or a management operation corresponding to the user operation information; Among them, the main control device is an Internet of Things (IoT) device without a screen, and all business codes and UI codes run in the IoT device. The IoT device only sends the UI information or UI update information to the electronic device so that the electronic device displays the UI or updates the UI.
14. The main control device according to claim 13, characterized in that: The user operation information also includes user input information acquired based on the first user operation.
15. The main control device according to claim 13 or 14, characterized in that: The UI information and the user operation information are information in a domain specific language DSL format.
16. The main control device according to claim 13 or 14, characterized in that: When the main control device sends UI information to the electronic device, the following steps are specifically performed: After establishing a communication connection with the electronic device, UI information is sent to the electronic device.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program, and when the computer program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 4, or execute the method according to any one of claims 5 to 8.
18. A chip, characterized in that: The chip is coupled to the memory, so that the chip calls the computer program stored in the memory when running to implement the method as described in any one of claims 1-4, or implements the method as described in any one of claims 5-8.
Citation Information
Patent Citations
Control device and device for intelligent household electrical appliance and control terminal
CN105116734A
Cross-device interaction method and terminal device
CN111523095A
Multi-device interaction method and system
CN111666075A
UI part display method and electronic equipment
CN111666119A
Equipment control method, intelligent household electrical appliance, intelligent terminal and intelligent electronic equipment
CN111817930A
