Electronic device operation guidance information recording method, acquisition method and terminal device

Abstract

The present application is applicable to the field of terminal technology, and provides a method for recording and obtaining operation guide information of an electronic device, and a terminal device. The above-mentioned method for recording operation guide information of an electronic device includes: the first terminal device responds to the received electronic device type identification, simulating the operation interface and / or physical hardware of the first electronic device corresponding to the electronic device type identification; the first terminal device receives operation information based on the operation interface and / or physical hardware; the first terminal device obtains operation element information generated according to the operation information; the first terminal device records the operation element information and generates operation guide information, and the operation guide information includes the electronic device type identification. The above-mentioned method enables the first terminal device to obtain the operation element information of the first electronic device without going through the first electronic device.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a method for recording and obtaining electronic equipment operation guidance information and a terminal device. Background Art

[0002] At present, with the increasing number of types of electronic devices, users' operations on electronic devices have become increasingly complicated. Especially for users who are not good at operating electronic devices, how to operate electronic devices simply and conveniently has become a difficult problem.

[0003] At present, the operation interface of an electronic device can be displayed on the terminal so that the user can operate on the operation interface. Then, the terminal sends the user's operation to the electronic device for execution to generate operation guidance information. However, the above method requires the terminal to establish a communication connection with the electronic device, and how to record the operation guidance information for the electronic device that cannot establish a communication connection with the terminal device is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a method for recording and obtaining operation guide information of an electronic device, and a terminal device, which are used to enable a terminal device to obtain operation element information of an electronic device without going through the electronic device.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, a method for recording operation guidance information of an electronic device is provided, the method comprising: a first terminal device simulates an operation interface and / or physical hardware of a first electronic device corresponding to a received electronic device type identifier in response to the received electronic device type identifier; the first terminal device receives operation information based on the operation interface and / or the physical hardware; the first terminal device obtains operation element information generated according to the operation information; the first terminal device records the operation element information to generate operation guidance information, wherein the operation guidance information includes the electronic device type identifier.

[0007] In the embodiment of the present application, the first terminal device simulates the operation interface and / or physical hardware of the electronic device. Then the first terminal device receives the operation information based on the operation interface and / or physical hardware, and obtains the operation element information generated according to the operation information. After that, the first terminal device records the operation element information to generate operation guidance information. In the embodiment of the present application, the first terminal device first simulates the operation interface and / or physical hardware of the first electronic device. Since the functions and attributes of the operation interface and / or physical hardware simulated by the first terminal device are the same as the functions and attributes of the operation interface and / or physical hardware of the first electronic device itself, the operation information based on the operation interface and / or physical hardware obtained by the first terminal device is the same as the operation information generated by the first electronic device when the first electronic device executes the same operation information. Therefore, the operation element information generated by the first terminal device based on the operation information is the same as the operation element information generated by the first electronic device when the operation information is executed, so that the first terminal device can obtain the operation element information of the first electronic device without going through the first electronic device. In addition, in the embodiment of the present application, the first electronic device does not need to share its own operation interface with the terminal device, so the risk of leaking the privacy of the user of the first electronic device is avoided.

[0008] In combination with the first aspect, in some embodiments, the first terminal device simulates the operating interface and / or physical hardware of the electronic device corresponding to the electronic device type identifier in response to the received electronic device type identifier, including: the first terminal device requests to establish a communication connection with the first electronic device; when the first terminal device fails to successfully establish a communication connection with the first electronic device, the first terminal device simulates the operating interface and / or the physical hardware of the first electronic device.

[0009] In combination with the first aspect, in some embodiments, the first terminal device simulates the operating interface and / or the physical hardware of the electronic device, including: the first terminal device calls the virtual device model of the first electronic device, and simulates the operating interface and / or the physical hardware of the first electronic device through the virtual device model.

[0010] Exemplarily, the first terminal device can simulate the operating interface and / or physical hardware of the first electronic device in an AR / VR manner, and then use human motion recognition technology to receive operating information based on the operating interface and / or physical hardware input, and obtain operating element information generated based on the operating information.

[0011] The virtual device model may include the physical features, software features, operating component features of the first electronic device, and a 3D / 2D device appearance / operation interface simulation model for display and operation in a virtual environment.

[0012] For example, a virtual device model may include device ID, device type, connection type, recordable operations, hardware information, software information, 2D / 3D model data, etc. Among them, the device ID is the identifier of the virtual device model; the device type is the device type of the first electronic device corresponding to the virtual device model; the connection type is the communication connection method of the virtual device model, such as WIFI, Bluetooth, NFC, local area network, distributed device network, etc.; the recordable operations are used to limit the operations that can be recorded in the operations of the virtual device model, which can be used to prevent privacy information leakage; the hardware information represents the physical characteristics of the first electronic device corresponding to the virtual device model; the software information represents the software characteristics of the application that can be run in the virtual device model; the 2D / 3D model data is the appearance / operation interface simulation model of the first electronic device corresponding to the virtual device model, etc.

[0013] In combination with the first aspect, in some embodiments, the method includes: when the first terminal device successfully establishes a communication connection with the first electronic device, the first terminal device obtains operation element information of the first electronic device from the first electronic device.

[0014] In combination with the first aspect, in some embodiments, the first terminal device obtains operation element information of the first electronic device from the first electronic device, including: the first terminal device sends an operation element information acquisition request to the first electronic device, the operation element information acquisition request is used to request the first electronic device to obtain the operation element information generated by the first electronic device in response to the received operation information, and send the operation element information to the first terminal device.

[0015] In combination with the first aspect, in some embodiments, there are multiple electronic device type identifiers, each of which corresponds to a first electronic device, each of which corresponds to multiple groups of operation element information, and each group of operation element information corresponds to a generation time and an electronic device type identifier;

[0016] The method further includes: the first terminal device sends a time synchronization request to each first electronic device corresponding to the plurality of electronic device type identifiers, the time synchronization request being used to request the first electronic device to adjust the time of the first electronic device to be consistent with the time of the first terminal device;

[0017] The first terminal device records the operation element information and generates operation guide information, including: the first terminal device determines the operation element information corresponding to each first electronic device according to the electronic device type identifier corresponding to each group of the operation element information; the first terminal device packages the operation element information corresponding to each first electronic device according to the chronological order of generation time of the operation element information, and generates operation guide information corresponding to each first electronic device.

[0018] In combination with the first aspect, in some embodiments, the operation guidance information also includes text information, and the text information is: the first electronic device converts the audio information obtained for operation guidance together with the operation information within the time range corresponding to the acquisition of the operation information into text information.

[0019] In combination with the first aspect, in some embodiments, the operation element information includes interface operation element information for the operation interface of the first electronic device, and / or hardware operation element information for the physical hardware of the first electronic device; wherein, the interface operation element information includes the operation interface layout and the operation element information corresponding to the state change of the operation interface.

[0020] Exemplarily, the operation page layout may include application name, page name, button name, position, etc. The state change of the operation interface may include button click, input box content change, focus change, etc. Based on the above state change, the corresponding operation element information can be obtained. For example, the terminal device can obtain the above operation element information through the view service of the UI interface. Specifically, the first terminal device of the Android platform can obtain the above interface operation element information through the local message monitoring function of the accessibility service.

[0021] In combination with the first aspect, in some embodiments, the method also includes: within a time range corresponding to when the first terminal device obtains the operation information, the first terminal device obtains audio information used to perform operation guidance together with the operation information; the first terminal device records the audio information in the operation guidance information.

[0022] In combination with the first aspect, in some embodiments, the first terminal device records the audio information in the operation guidance information, including: the first terminal device converts the audio information into text information, and records the text information in the operation guidance information.

[0023] Among them, in order to avoid the risk of leaking user voiceprint information, the first terminal device can convert the audio information into text information through NLP technology and record it in the operation guidance information. In the subsequent process, the text information can be converted into audio information for playback through TTS technology.

[0024] Exemplarily, the above text information may include an electronic device type identifier and time information, wherein the electronic device type identifier is used to characterize the first electronic device corresponding to the text information, and the time information is used to characterize the time when the first terminal device acquires the text information.

[0025] In some embodiments, each operation element information corresponds to a generation time, an electronic device type identifier and an application identifier. The above-mentioned electronic device operation guide information recording method may also include: the first terminal device obtains the electronic device type identifier and application identifier corresponding to the operation element information, establishes a link relationship between the operation element information and the corresponding electronic device type identifier and application identifier, and displays the electronic device type identifier and the application identifier.

[0026] Exemplarily, the first terminal device can divide the operation element information according to the electronic device type identifier and the application identifier, divide the operation element information corresponding to each electronic device type identifier according to the application identifier, and establish a link relationship between the operation element information and the electronic device type identifier and the application identifier, and display the electronic device type identifier and the application identifier in a list or chart, so that the user can intuitively understand the first electronic device and application corresponding to the operation guide information. At the same time, after receiving the operation information of clicking the electronic device type identifier and / or the application identifier, the first terminal device can retrieve the corresponding operation element information based on the above-mentioned link relationship and replay it through the corresponding virtual device model, so that the user of the first terminal device can check whether the operation is complete. After receiving the input inspection completion information, the first terminal device can package the above-mentioned operation element information and publish it to the cloud for security inspection. Among them, the cloud can perform a security check on the operation element information involving the specified APP in the operation element information, and insert a security prompt for the operation element information that changes the system sensitive settings.

[0027] In a second aspect, a method for obtaining operation guidance information of an electronic device is provided, the method comprising: a second terminal device obtains operation guidance information for performing operation guidance on a second electronic device, the operation guidance information including an electronic device type identifier and operation element information; the second terminal device simulates an operation interface and / or physical hardware of the second electronic device corresponding to the electronic device type identifier; the second terminal device responds to the operation element information based on the operation interface and / or the physical hardware, and generates prompt information for performing operation guidance on the second electronic device.

[0028] In the above-mentioned method for recording operation guidance information of an electronic device, after the second terminal device obtains the operation guidance information for operating the second electronic device, it simulates the operation interface and / or physical hardware of the second electronic device corresponding to the electronic device type identifier, and responds to the above-mentioned operation guidance information based on the operation interface and / or physical hardware. Since the functions and attributes of the operation interface and / or physical hardware simulated by the second terminal device are the same as the functions and attributes of the operation interface and / or physical hardware of the second electronic device itself, the operation information obtained by the second terminal device based on the operation interface and / or physical hardware is the same as the operation information generated by the second electronic device when the second electronic device performs the same operation information. Therefore, the operation element information generated by the second terminal device based on the operation information is the same as the operation element information generated by the second electronic device when the second electronic device executes the operation information. Therefore, the operation guidance effect of the second terminal device responding to the operation guidance information through the simulated operation interface and / or physical hardware is the same as the operation guidance effect of the second electronic device responding to the operation guidance information, so that the second terminal device can respond to the operation guidance information without going through the second electronic device, thereby being able to guide the user to operate the electronic device.

[0029] In combination with the second aspect, in some embodiments, the second terminal device simulates the operating interface and / or physical hardware of the second electronic device corresponding to the electronic device type identifier, including: when the second electronic device is an electronic device that does not have an operation guidance information response unit, the second terminal device simulates the operating interface and / or the physical hardware of the second electronic device; wherein the operation guidance information response unit includes a display unit and / or an audio playback unit.

[0030] In some embodiments, the electronic device operation guidance information recording method may further include: the second terminal device determines, based on the electronic device type identifier, whether the second electronic device corresponding to the electronic device type identifier has an operation guidance information response unit.

[0031] In combination with the second aspect, in some embodiments, the second terminal device simulates the operating interface and / or the physical hardware of the second electronic device, including: the second terminal device calls the virtual device model of the second electronic device, and simulates the operating interface and / or the physical hardware of the second electronic device through the virtual device model.

[0032] The virtual device model of the second electronic device can be stored in the second terminal device or in the cloud server, which is not limited in the present embodiment. The virtual device model can include the physical features, software features, operating component features of the second electronic device, and a 3D / 2D device appearance / operation interface simulation model for display and operation in a virtual environment, etc.

[0033] In conjunction with the second aspect, in some embodiments, there are multiple electronic device type identifiers, each of the electronic device type identifiers corresponds to a second electronic device, each of the second electronic devices corresponds to an operation guidance information, each operation guidance information includes multiple groups of operation element information, and each group of the operation element information corresponds to a generation time;

[0034] The responding to the operation guidance information based on the operation interface and / or the physical hardware includes: the second terminal device parses the operation guidance information corresponding to each of the second electronic devices in turn to obtain multiple groups of operation element information corresponding to each of the second electronic devices; the second terminal device responds to each group of operation element information in sequence through the operation interface and / or physical hardware of the second electronic device corresponding to each group of operation element information in the order of generation time corresponding to each group of operation element information.

[0035] In some embodiments, the operation element information includes interface operation element information for the operation interface of the second electronic device and / or hardware operation element information for the physical hardware of the second electronic device; the interface operation element information includes the operation interface layout and operation element information corresponding to the state change of the operation interface.

[0036] The second terminal device responds to the operation guidance information based on the operation interface and / or the physical hardware. Specifically, it can be: the second terminal device determines the position of the operation element information in the operation interface according to the operation interface layout in the operation element information, and displays the prompt information drawn based on the operation element information at this position.

[0037] Exemplarily, when there is a display interface that needs to be operated in the operation element information corresponding to a certain operation guidance step in the operation guidance information, the second terminal device can extract the button element information in the corresponding operation element information and determine the position of the button element information in the operation interface, and display the prompt information drawn based on the operation element information in the simulated operation interface according to the position, for example, in the form of a highlight box or arrow guide in the simulated operation interface; in the case where there is text information, the terminal device can use TTS to convert the text information into audio information for playback.

[0038] In combination with the second aspect, in some embodiments, the method includes: when the second electronic device is an electronic device having an operation guidance information response unit, the second terminal device sends the operation guidance information to the second electronic device.

[0039] In combination with the second aspect, in some embodiments, the operation guidance information includes interface operation element information for the user interface of the second electronic device, hardware operation element information for the physical hardware of the second electronic device, and text information converted from audio information used for operation guidance; wherein, the interface operation element information is used to instruct the display unit of the second electronic device to display the interface operation element information, the hardware operation element information is used to instruct the audio playback unit of the second electronic device to play the hardware operation element information, and the text information is used to instruct the audio playback unit of the second electronic device to convert the text information into audio information for playback.

[0040] In some embodiments, the above-mentioned method for recording operation guidance information of an electronic device may further include: the second terminal device responds to the received operation instruction, sends the operation instruction to the corresponding second electronic device, so that the second electronic device executes the operation instruction; wherein, the operation instruction may be an operation instruction based on the operation guidance information received by the second terminal device after the second terminal device responds to the operation guidance information based on the operation interface and / or physical hardware.

[0041] It can be understood that after the second terminal device responds to the operation guidance information based on the operation interface and / or physical hardware, the user can input operation instructions for operating the second electronic device on the operation interface and / or physical hardware simulated by the second terminal device, and the second terminal device sends the operation instructions to the corresponding second electronic device, so that the second electronic device can be operated through the second terminal device without the user having to perform operations on the second electronic device.

[0042] Exemplarily, the second terminal device can simulate the operating interface and / or physical hardware of the second electronic device in an AR / VR manner, then obtain operating instructions through human motion recognition technology, and send the operating instructions to the corresponding second electronic device so that the second electronic device executes the operating instructions.

[0043] In a third aspect, a method for recording operation guidance information of an electronic device is provided, the method comprising: a first electronic device receives an operation element information acquisition request sent by a first terminal device; the first electronic device responds to the operation element information acquisition request, receives input operation information, and acquires operation element information generated based on the operation information; the first electronic device sends the operation element information to the first terminal device.

[0044] In the above-mentioned method for recording operation guidance information of an electronic device, after the first electronic device receives the operation element information acquisition request sent by the first terminal device, it responds to the operation element information acquisition request, receives the input operation information and obtains the operation element information generated based on the operation information, and sends the operation element information to the first terminal device, so that the first terminal device records the operation element information to generate operation guidance information corresponding to the first electronic device. Since the operation element information does not involve content that may leak the user privacy of the electronic device compared to the operation interface image in the traditional technology, it can reduce the risk of leaking the user privacy in the process of guiding the operation of the electronic device; and the operation guidance element information does not involve the size of the operation interface, so it can be better suitable for electronic devices with different operation interface sizes compared to the operation interface image in the traditional technology; and the operation element information requires less space than the operation interface image, so the first electronic device has lower requirements on the network in the process of transmitting the operation element information to the first terminal device, which can reduce the data transmission delay.

[0045] In combination with the third aspect, in some embodiments, the first electronic device receives a time synchronization request sent by the first terminal device, and adjusts the time of the first electronic device to be consistent with the time of the first terminal device based on the time synchronization request; the first electronic device sends the operation element information to the first terminal device, including: the first electronic device aggregates the operation element information generated by the first electronic device and sends it to the first terminal device.

[0046] Exemplarily, the operation element information may include a data packet header, a node ID, a node type, an operation ID, a timestamp, operation data, and a data packet tail. The node ID represents the first electronic device corresponding to the operation element information; the node type represents the device type of the first electronic device; the operation ID is an operation identifier for the first electronic device or a virtual device model, wherein the data structure corresponding to the operation corresponding to the operation ID can be extracted from the first electronic device or the virtual device model according to the combination of the node type and the operation ID, and the above combination can distinguish the possible differences in the definition of the same type of operation by the first electronic device with different operation interface sizes; the timestamp is used to define the time when the operation is executed.

[0047] In a fourth aspect, a method for responding to operation guidance information of an electronic device is provided, the method comprising: a second electronic device receives operation guidance information sent by a second terminal device, the operation guidance information comprising multiple groups of operation element information, each group of operation element information corresponding to a generation time; the second electronic device responds to each group of operation element information in sequence based on the generation time of each group of operation element information.

[0048] In the above-mentioned method for recording operation guidance information of an electronic device, after the second electronic device receives the operation guidance information sent by the second terminal device, it responds to each group of operation element information in the operation guidance information in sequence based on the generation time of each group of operation element information in the operation guidance information. Since the operation element information does not involve content that may leak the user privacy of the electronic device compared to the operation interface image in the traditional technology, it can reduce the risk of leaking the user privacy in the process of guiding the operation of the electronic device; and the operation guidance element information does not involve the size of the operation interface, so it can be better suitable for electronic devices with different operation interface sizes compared to the operation interface image in the traditional technology.

[0049] In combination with the fourth aspect, in some embodiments, the operation guidance information includes interface operation element information for the user interface of the second electronic device, hardware operation element information for the physical hardware of the second electronic device, and one or more of text information converted from audio information used for operation guidance; the second electronic device responds to each group of operation element information in sequence, including: the second electronic device displays the interface operation element information through a display unit in the order of generation time of each group of operation element information, and / or plays the hardware operation element information through an audio playback unit, and / or converts the text information into audio information for playback through an audio playback unit.

[0050] In some embodiments, the operation element information includes interface operation element information for the operation interface of the second electronic device and / or hardware operation element information for the physical hardware of the second electronic device; the interface operation element information includes the operation interface layout and operation element information corresponding to the state change of the operation interface.

[0051] The above-mentioned electronic device responds to the operation guidance information, which can specifically be: the second electronic device determines the position of the operation element information in the operation interface of the electronic device according to the operation interface layout in the operation element information, and displays the prompt information generated based on the operation element information at this position.

[0052] Exemplarily, when there is a display interface that needs to be operated in the operation element information corresponding to a certain operation guidance step in the operation guidance information, the second electronic device can extract the key element information in the corresponding operation element information and determine the position of the key element information in the operation interface, and display the prompt information drawn based on the operation element information in the simulated operation interface according to the position, for example, in the form of a highlight box or arrow guidance in the operation interface; in the case where there is text information, the second electronic device can use TTS to convert the text information into audio information for playback.

[0053] In a fifth aspect, a device for recording operation guidance information of an electronic device is provided, the device comprising: a first simulation unit, for simulating, in response to a received electronic device type identifier, an operation interface and / or physical hardware of a first electronic device corresponding to the electronic device type identifier; a first receiving unit, for receiving operation information based on the operation interface and / or physical hardware; a first acquisition unit, for acquiring operation element information generated based on the operation information; and a recording unit, for recording the operation element information to generate operation guidance information.

[0054] In a sixth aspect, a device for obtaining operation guidance information of an electronic device is provided, the device comprising: a second acquisition unit, used to obtain operation guidance information for operating a second electronic device, the operation guidance information including an electronic device type identifier; a second simulation unit, used to simulate an operation interface and / or physical hardware of the second electronic device corresponding to the electronic device type identifier; and a first response unit, used to respond to the operation guidance information based on the operation interface and / or the physical hardware.

[0055] In the seventh aspect, a device for recording operation guidance information of an electronic device is provided, the device comprising: a second receiving unit, used to receive an operation element information acquisition request sent by a first terminal device; a second responding unit, used to respond to the operation element information acquisition request, receive input operation information, and acquire operation element information generated based on the operation information; and a sending unit, used to send the operation element information to the first terminal device.

[0056] In an eighth aspect, a device for responding to operation guidance information of an electronic device is provided, the device comprising: a third receiving unit, used to receive operation guidance information sent by a second terminal device, the operation guidance information comprising multiple groups of operation element information, each group of operation element information corresponding to a generation time; a third responding unit, used to respond to each group of operation element information in sequence based on the generation time of each group of operation element information.

[0057] In the ninth aspect, an embodiment of the present application provides a chip system, which may include a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps in each embodiment of the method described in the first aspect above, or the steps in each embodiment of the method described in the second aspect above, or the steps in each embodiment of the method described in the third aspect above, or the steps in each embodiment of the method described in the fourth aspect above. The chip system may be a single chip or a chip module composed of multiple chips.

[0058] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps in the various embodiments of the method described in any one of the first aspect, or implements the steps in the various embodiments of the method described in the second aspect, or implements the steps in the various embodiments of the method described in the third aspect, or implements the steps in the various embodiments of the method described in the fourth aspect.

[0059] In the eleventh aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the electronic device executes any method described in the first aspect above, and / or implements any method described in the second aspect, and / or implements any method described in the third aspect, and / or implements any method described in the fourth aspect.

[0060] In the twelfth aspect, an embodiment of the present application provides a terminal device, which includes: one or more processors, a memory and a display screen; the memory and the display screen are coupled to the one or more processors, the memory is used to store computer program code, and the computer program code includes computer instructions; when the one or more processors execute the computer instructions, the terminal device executes any one of the methods described in the first aspect above, or implements any one of the methods described in the second aspect above.

[0061] In the thirteenth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors, a memory and a display screen; the memory and the display screen are coupled to the one or more processors, the memory is used to store computer program code, and the computer program code includes computer instructions; when the one or more processors execute the computer instructions, the terminal device executes any one of the methods described in the third aspect above, or implements any one of the methods described in the fourth aspect above.

[0062] It can be understood that the electronic device operation guidance information recording device described in the fifth to eighth aspects, the chip system described in the ninth aspect, the computer-readable storage medium described in the tenth aspect, the computer program product described in the eleventh aspect, the terminal device described in the twelfth aspect, and the electronic device described in the thirteenth aspect are all used to execute the methods provided in the first to fourth aspects. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of an application scenario of the method for recording operation guidance information of an electronic device provided in an embodiment of the present application;

[0064] Figure 2 A schematic diagram of another application scenario of the method for recording electronic device operation guidance information provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of the hardware architecture of a terminal device provided in an embodiment of the present application;

[0066] Figure 4 A software structure diagram of a terminal device provided in an embodiment of the present application;

[0067] Figure 5 A schematic diagram of the architecture of a terminal device provided in an embodiment of the present application;

[0068] Figure 6 A flowchart of a method for recording electronic device operation guidance information provided by an embodiment of the present application;

[0069] Figure 7 A schematic diagram of the operation interface of the first terminal device provided in an embodiment of the present application;

[0070] Figure 8 A schematic diagram of a scenario of an operation interface of a first terminal device provided in an embodiment of the present application;

[0071] Fig. 9 A schematic diagram of a scenario of an operation interface of a first terminal device provided in an embodiment of the present application;

[0072] Fig.10 A flowchart of a method for recording electronic device operation guidance information provided by an embodiment of the present application;

[0073] Fig.11 A flowchart of a method for recording electronic device operation guidance information provided by an embodiment of the present application;

[0074] Fig.12 A flowchart of a method for obtaining electronic device operation guidance information provided in an embodiment of the present application

[0075] Fig.13 A schematic diagram of the operation interface of the second terminal device provided in an embodiment of the present application;

[0076] Fig.14 A schematic diagram of a scenario of an operation interface of a second terminal device provided in an embodiment of the present application;

[0077] Fig.15 A schematic diagram of a scenario of an operation interface of a second terminal device provided in an embodiment of the present application;

[0078] Fig.16 A schematic diagram of the structure of the electronic device operation guidance information recording device provided in the embodiment of the present application

[0079] Fig.17 A schematic diagram of the structure of an apparatus for obtaining operation guidance information of an electronic device provided in an embodiment of the present application;

[0080] Fig.18 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0082] At present, with the increasing number of types of electronic devices, users' operations on electronic devices have become increasingly complicated. Especially for users who are not good at operating electronic devices, how to operate electronic devices simply and conveniently has become a difficult problem.

[0083] For the above problem, an operation interface of an electronic device can be displayed on the terminal device so that the user can operate on the operation interface. Then, the terminal device sends the user's operation to the electronic device for execution to generate operation guidance information.

[0084] For example, terminal device B (the assisted party) can request terminal device A (the assisting party) to assist terminal device B in operating and guiding certain programs or settings in terminal device B through the Family Care Application (Application, APP). Specifically, after terminal device B establishes a communication connection with terminal device A through the Family Care APP, terminal device B can send an operation assistance request to terminal device A through the APP. After terminal device A agrees to the operation assistance request of terminal device B, terminal device B sends the operation interface image of terminal device B to terminal device A. The user operates on terminal device A according to the operation interface image, and then terminal device A sends the user's operation information for the operation interface image to terminal device B. After that, terminal device B can perform corresponding operations according to the operation information implemented by the user on terminal device A, so that terminal device A can assist terminal device B in operating the programs or settings in terminal device B.

[0085] However, the premise of the above scheme is that a real-time synchronous communication connection is established between terminal device A and terminal device B. However, if terminal device A and terminal device B cannot establish a communication connection (for example, terminal device B is not in working state, so terminal device A and terminal device B cannot establish a communication connection), then terminal device A cannot generate operation guidance information for certain programs or settings in terminal device B according to the above scheme. In addition, since the operation interface image of terminal device B is sent to terminal device A, and the operation interface image may contain the privacy information of the user of terminal device B, there may be a risk of user privacy leakage, and the transmission of the operation interface image has high requirements on the quality of network connection.

[0086] Based on the above problems, in the embodiment of the present application, the terminal device simulates the operation interface and / or physical hardware of the electronic device. Then the terminal device receives the operation information based on the operation interface and / or physical hardware, and obtains the operation element information generated according to the operation information. After that, the terminal device records the operation element information to generate operation guidance information. In the embodiment of the present application, the terminal device first simulates the operation interface and / or physical hardware of the electronic device. Since the functions and attributes of the operation interface and / or physical hardware simulated by the terminal device are the same as the functions and attributes of the operation interface and / or physical hardware of the electronic device itself, the operation information based on the operation interface and / or physical hardware obtained by the terminal device is the same as the operation information generated by the electronic device when the electronic device executes the same operation information. Therefore, the operation element information generated by the terminal device based on the operation information is the same as the operation element information generated by the electronic device when the electronic device executes the operation information, so that the terminal device can obtain the operation element information of the electronic device without going through the electronic device. In addition, in the embodiment of the present application, the electronic device does not need to share its own operation interface with the terminal device, so the risk of leaking the privacy of the user of the electronic device is avoided.

[0087] like Figure 1 As shown, Figure 1 A scenario is shown in which a method for recording electronic device operation guidance information provided in an embodiment of the present application is applicable, and the scenario includes: terminal device A, terminal device B, and multiple electronic devices a.

[0088] There is an asynchronous communication connection between terminal device A and terminal device B, for example, terminal device A and terminal device B can perform asynchronous message transmission.

[0089] In which, when the user of terminal device B is not good at operating electronic devices, terminal device B can request terminal device A to record the operation guidance information of electronic device a to assist the user of terminal device B to operate electronic device a.

[0090] The terminal device A may be a device capable of recording operation guidance information. The terminal device B may be a device that responds to the operation guidance information to display prompt information for operating the electronic device a.

[0091] The above-mentioned terminal device A can actively decide to record the operation guidance information, or terminal device A can decide to record the operation guidance information based on the trigger of terminal device B when successfully establishing a connection with terminal device B. The embodiments of the present application do not limit this.

[0092] The above-mentioned multiple electronic devices a may include a personal computer (PC), a TV (television), a speaker, a router, a smart refrigerator and other electronic devices.

[0093] In a possible implementation, terminal device B may request terminal device A to assist in guiding terminal device B to operate electronic device a through a related application. After receiving the request from terminal device B, terminal device A records the operation guidance information of various electronic devices a according to the request. Then, terminal device A transmits the operation guidance information of various electronic devices a to terminal device B.

[0094] Terminal device A can obtain the operation element information of each electronic device a among multiple electronic devices a through terminal device A, and / or obtain the operation element information of the electronic device from electronic device a, record the operation element information of each electronic device a to obtain the operation guidance information of each electronic device a, and send the operation guidance information of each electronic device a. For example, terminal device A can send the operation guidance information of each electronic device a to the server or terminal device B.

[0095] Terminal device B can obtain operation guidance information for operating each electronic device a among multiple electronic devices a from a server or terminal device A, and then generate prompt information for operating the electronic device a through terminal device B in response to the operation guidance information of each electronic device a, and / or generate prompt information for operating the electronic device a through the corresponding electronic device a.

[0096] Exemplarily, the terminal device may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or other electronic devices. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0097] like Figure 2 As shown, Figure 2 Another scenario in which the method for recording electronic device operation guidance information provided by the embodiment of the present application is applicable is shown. Figure 1 The differences between the scenarios shown are: Figure 2The scene shown includes multiple electronic devices b and multiple electronic devices a. The electronic device a or the electronic device b may be one of multiple electronic devices such as a PC, a TV, a speaker, a router, a smart refrigerator, etc. The types of some electronic devices in the multiple electronic devices a are the same as the types of the multiple electronic devices. For example, the multiple electronic devices a include a PC, and the multiple electronic devices b may include a PC and a TV. The multiple electronic devices b may be electronic devices that communicate with the terminal device A, and the multiple electronic devices a may be electronic devices that communicate with the terminal device B.

[0098] exist Figure 2 In the scenario shown, taking the example of the second terminal device requesting the first terminal device to generate operation guidance information for electronic device a, if the first terminal device determines that the electronic device b communicating with the first terminal device is of the same type as the electronic device a, the first terminal device can simulate the operation interface and / or physical hardware of the electronic device b. In other words, the first terminal device can use the simulated operation interface and / or physical hardware of the electronic device b as the operation interface and / or physical hardware of the electronic device a. Figure 1 In the scenario shown, the first terminal device simulates the operating interface and / or physical hardware of the electronic device a.

[0099] For example, Figure 3 1 is a schematic diagram of the hardware architecture of the terminal device 100 provided in the embodiment of the present application. The structures of the terminal device A and the terminal device B can refer to the description of the terminal device 100, which will not be repeated here. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0100] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0101] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0102] The controller may be the nerve center and command center of the terminal device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0103] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0104] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0105] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0106] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0107] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to implement the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0108] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0109] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0110] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0111] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0112] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0113] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0114] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0115] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0116] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0117] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0118] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0119] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0120] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0121] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0122] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0123] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0124] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0125] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0126] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0127] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0128] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0129] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0130] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0131] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0132] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0133] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0134] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0135] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0136] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0137] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0138] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0139] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0140] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0141] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0142] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0143] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0144] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0145] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0146] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0147] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0148] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0149] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0150] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0151] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0152] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0153] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0154] Figure 4 It is a software structure block diagram of the terminal device 100 according to an embodiment of the present application.

[0155] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0156] The application layer can include a series of application packages.

[0157] like Figure 4 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0158] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0159] like Figure 4 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0160] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0161] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0162] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0163] The phone manager is used to provide communication functions of the terminal device 100, such as management of call status (including connection, disconnection, etc.).

[0164] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0165] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0166] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0167] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0168] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0169] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0170] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0171] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0172] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0173] A 2D graphics engine is a drawing engine for 2D drawings.

[0174] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0175] The following is an illustrative description of the software and hardware workflow of the terminal device 100 in conjunction with the capture and photo shooting scene.

[0176] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0177] like Figure 5 As shown, Figure 5 A schematic diagram of the architecture of a terminal device is provided. The architectures of terminal device A and terminal device B can be referenced. Figure 4 The terminal device may include a device management module, a connection management module, a data processing module, and an operation playback module. Figure 3 The processor 110, the internal memory 121, the external memory interface 120 and the USB interface 130 correspond to each other, and the connection management module can be connected to Figure 3The processor 110 corresponds to the wireless communication module 160, and the data processing module can be connected to Figure 3 Corresponding to the processor 110 in Figure 3 Corresponding to the processor 110, display screen, and speaker 170A.

[0178] Specifically, the connection management module includes a local device connection management unit and a virtual device connection management unit, wherein the virtual device connection management unit is used to manage the connection between the remote device and the virtual device model, and the remote device data is mapped to the current main device that performs the recording operation through the device model.

[0179] The data processing module includes a local operation monitoring unit, a virtual device monitoring unit, and an operation data packaging unit, which are used to record operation events during the electronic device operation recording process and provide data aggregation processing, authentication packaging and other operations for operation events.

[0180] The operation playback module includes a local playback unit and a receiving end playback unit, which are used for viewing the recorded operation process and performing related operations such as prompt parsing and distribution on the execution end.

[0181] Among them, the local playback unit may not be set in the terminal device B.

[0182] The device management module includes a physical device model management unit, a virtual device model management unit and a virtual device simulation component. The physical device model and the virtual device model contain the physical characteristics of the actual product, software version characteristics, recordable operation component characteristics and may include 3D / 2D device appearance / operation interface simulation models that are displayed and operated in a virtual environment.

[0183] The following is a description of the electronic device operation guidance method provided in an embodiment of the present application.

[0184] See also Figure 6 , Figure 6 1 is a flow chart of a method for recording operation guidance information of an electronic device provided in an embodiment of the present application, the method comprising:

[0185] Step S101: In response to a received electronic device type identifier, a first terminal device simulates an operation interface and / or physical hardware of a first electronic device corresponding to the electronic device type identifier.

[0186] For example, the first terminal device may be Figure 1 or Figure 2 The terminal device A shown in FIG. Figure 1 or Figure 2The physical hardware of the first electronic device may refer to physical hardware inherent to the first electronic device, such as a switch button, volume +, volume down, etc., of a speaker, an antenna and a switch button of a router.

[0187] The number of the above electronic device type identifications may be one or more.

[0188] It should be understood that, in the embodiment of the present application, before step S101, the method provided in the embodiment of the present application further includes: the first terminal device obtains one or more electronic device type identifiers.

[0189] In a possible implementation, the first terminal device in the embodiment of the present application may obtain one or more electronic device type identifiers in the following manner: the first terminal device receives a signal from the second terminal device (for example, Figure 1 or Figure 2 For example, terminal device B may send a request message to terminal device A through a related application. The request message is used to request terminal device A to record operation guidance information. The operation guidance information is used to guide the user of terminal device B on how to operate the one or more electronic devices, and the request may carry the type identification of the one or more electronic devices.

[0190] In another possible implementation, in the embodiment of the present application, the first terminal device may obtain one or more electronic device type identifiers in the following manner: the first terminal device receives a first operation input by a user, and the first operation is used to determine the one or more electronic device type identifiers.

[0191] See also Figure 7 As shown in (a) of FIG. 1 , the first terminal device displays at least the following through the display interface: Figure 7 The operation guide APP200 shown in (a) in FIG. 1 is shown in FIG. 1 . The first terminal device detects that the user clicks Figure 7 After the operation guide APP200 shown in (a) of FIG. 1 is opened, the first terminal device opens the operation guide APP200 and displays the following Figure 7 The interface shown in (b) or Figure 7 The interface shown in (d) in the figure. Figure 7 As shown in (b) of FIG. 1 , the first terminal device displays information of one or more electronic devices for the user to select. Figure 7 As shown in (c) of FIG. 1 , the user can click on controls corresponding to some or all of the electronic devices in the display interface of the first terminal device. The first terminal device detects that the user clicks Figure 7 In the “confirm” control shown in (c) of FIG. 1 , the first terminal device determines that the “confirm” control is triggered, and the first terminal device can Figure 7The selected "personal computer", "speaker A", "television", "router", "refrigerator" and the like in (c) are determined as one or more electronic device type identifiers selected by the user. Afterwards, the first terminal device simulates the operation interface and / or physical hardware of the electronic device selected by the user, such as Figure 7 As shown in (f) in FIG. 1 , the first terminal device may simulate the operation interface and / or physical hardware of one or more electronic devices selected by the user by clicking in the first operation interface area 201 of the first terminal device.

[0192] like Figure 7 As shown in (d) in FIG. 1 , the first terminal device displays the electronic device controls corresponding to the one or more electronic device type identifiers carried in the request sent by the second terminal device. Figure 7 As shown in (e) of FIG. 1 , after confirming that the second terminal device requests assistance in operating and guiding the above electronic device, the user clicks the "Confirm" control. The first terminal device responds to the one or more electronic devices confirmed by the user, simulating the operation interface and / or physical hardware of the one or more electronic devices, such as Figure 7 As shown in (f) in FIG. 1 , the first terminal device may simulate the operation interface and / or physical hardware of one or more electronic devices confirmed by the user in the first operation interface area 201 of the first terminal device.

[0193] For example, take the first electronic device as a personal computer. Figure 8 As shown in (a) of FIG. 1 , the first terminal device displays an operation interface 201 of a personal computer simulated by the first terminal device. The user can click on each control in the operation interface 201 of the simulated personal computer, for example, the user clicks on the "control panel" control. After the first terminal device responds to the click operation on the "control panel" control, it simulates Figure 8 The user can use the operation interface of the personal computer shown in (b) of FIG. Figure 8 In the operation interface of the personal computer shown in (b), the user clicks on each control, for example, the user clicks on the "Programs and Functions" control. The first terminal device responds to the click on the "Programs and Functions" control and simulates the following Figure 8 The operating interface of the personal computer shown in (c) in FIG. Figure 8 In the operation interface of the personal computer shown in (c) in FIG. 1 , a user clicks on each control. For example, a user clicks on the "Foxmail" control. The first terminal device simulates the following in response to the click on the "Foxmail" control: Figure 8 The operating interface of the personal computer shown in (d) in FIG. Figure 8Click the "Uninstall" control in the operation interface of the personal computer shown in (d).

[0194] For example, take the first electronic device as a router. Fig. 9 As shown, the first terminal device displays the physical hardware of the router simulated by the first terminal device, and the user can operate one or some physical hardware of the router on the simulated physical hardware of the router, for example, the user can adjust the angle of the antenna of the router simulated by the first terminal device.

[0195] Step S102: The first terminal device receives operation information based on the operation interface and / or physical hardware.

[0196] After the first terminal device simulates the operation interface and / or physical hardware of the electronic device, the user can perform operations on the operation interface and / or physical hardware. For example, the user clicks, slides, etc. on the operation interface, adjusts, rotates, etc. the physical hardware, and the corresponding operation information may include operation content, location information, etc.

[0197] For example, if the first electronic device is a speaker, the physical hardware of the first electronic device may be a switch button, volume +, volume down, etc. of the speaker. If the first electronic device is a router, the physical hardware of the first electronic device may be the antenna, switch button, network cable interface, etc. of the router. The user can adjust the buttons or antennas by operating the simulated physical hardware in the first terminal device. The first terminal device can obtain operation information corresponding to the physical hardware by detecting the user's operation on the simulated operation interface and / or physical hardware.

[0198] Step S103: The first terminal device obtains operation element information generated according to the operation information.

[0199] In an application scenario, for the operation interface simulated in the first operation interface area 201 of the first terminal device, the first terminal device receives operation information input based on the operation interface, and obtains operation element information generated according to the operation information. Exemplarily, the operation may be a touch operation or a touch operation set on a certain application on the operation interface, and the operation element information may include interface operation element information for the operation interface, and / or hardware operation element information for the physical hardware of the electronic device, and the interface operation element information may include the layout of the operation interface and operation element information corresponding to the state change of the operation interface.

[0200] Exemplarily, the operation page layout may include application name, page name, button name, position, etc. The state change of the operation interface may include button click, input box content change, focus change, etc. Based on the above state change, the corresponding operation element information can be obtained. For example, the first terminal device can obtain the above operation element information through the view service of the user interface (UI). Specifically, for the first terminal device of the Android platform, the above interface operation element information can be obtained through the local message monitoring function of the accessibility service.

[0201] In an application scenario, for the physical hardware simulated in the first operation interface area 201 of the first terminal device, the first terminal device receives operation information based on the input of the physical hardware, and obtains operation element information corresponding to the operation information in the system message generated by the above operation. Exemplarily, the physical hardware can be physical hardware such as buttons of a speaker and antenna of a router. The user's operation on the physical hardware simulated by the first terminal device can adjust the above buttons or antennas. The first terminal device can monitor the system message generated by the above operation and obtain the hardware operation element information corresponding to the above physical hardware from the system message.

[0202] Exemplarily, the first terminal device can simulate the operating interface and / or physical hardware of the first electronic device in an AR (Augmented Reality) / VR (Virtual Reality) manner, and then combine human motion recognition technology to receive operating information based on the operating interface and / or physical hardware input, and obtain operating element information generated based on the operating information.

[0203] Step S104: The first terminal device records the operation element information and generates operation guidance information, wherein the operation guidance information includes an electronic device type identifier.

[0204] The first terminal device may record the operation element information corresponding to the electronic device type identifier, and generate the operation guidance information corresponding to the electronic device type identifier, wherein the operation guidance information includes the corresponding electronic device type identifier. When there are multiple electronic device type identifiers, the first terminal device records the operation element information corresponding to each electronic device type identifier, and generates the operation guidance information corresponding to each electronic device type identifier.

[0205] In the above-mentioned method for recording operation guide information of an electronic device, the first terminal device simulates the operation interface and / or physical hardware of the first electronic device. Since the functions and attributes of the operation interface and / or physical hardware simulated by the first terminal device are the same as the functions and attributes of the operation interface and / or physical hardware of the first electronic device itself, the operation information based on the operation interface and / or physical hardware obtained by the first terminal device is the same as the operation information generated by the first electronic device when the first electronic device performs the same operation. Therefore, the operation element information generated by the first terminal device based on the operation information is the same as the operation element information generated by the first electronic device when the first electronic device executes the operation information, so that the first terminal device can obtain the operation element information of the first electronic device without going through the first electronic device. In addition, in the embodiment of the present application, the first electronic device does not need to share its own operation interface with the terminal device, so the risk of leaking the privacy of the user of the first electronic device is avoided.

[0206] Exemplarily, the electronic device type identifier may be one or more, for example, the electronic device type identifier may include one or more of the electronic device type identifiers corresponding to a PC, an electronic device type identifier corresponding to a TV, an electronic device type identifier corresponding to a smart refrigerator, an electronic device type identifier corresponding to a speaker, and an electronic device type identifier corresponding to a router.

[0207] In the case of multiple electronic device type identifiers, the first terminal device simulates the operation interface and / or physical hardware of the first electronic device corresponding to the electronic device type identifier, which may be:

[0208] The first terminal device simulates the operating interface and / or physical hardware of the first electronic device corresponding to each electronic device type identifier in sequence according to a preset order, so as to obtain corresponding operating element information based on the operating information corresponding to each first electronic device, and finally records the operating element information corresponding to each first electronic device, and generates operation guide information corresponding to each first electronic device.

[0209] For example, each operation element information may correspond to a generation time and an electronic device type identifier, and the generation time may be based on the time of the first terminal device. Based on this, the first terminal device may determine the operation element information corresponding to each first electronic device according to the generation time and electronic device type identifier corresponding to the operation element information; the first terminal device packages the operation element information corresponding to each electronic device, and generates operation guide information corresponding to each electronic device.

[0210] In the embodiment of the present application, the terminal device simulates the operation interface and / or physical hardware of the electronic device. Then the terminal device receives the operation information based on the operation interface and / or physical hardware, and obtains the operation element information generated according to the operation information. After that, the terminal device records the operation element information to generate operation guidance information. In the embodiment of the present application, the terminal device first simulates the operation interface and / or physical hardware of the electronic device. Since the functions and attributes of the operation interface and / or physical hardware simulated by the terminal device are the same as the functions and attributes of the operation interface and / or physical hardware of the electronic device itself, the operation information based on the operation interface and / or physical hardware obtained by the terminal device is the same as the operation information generated by the electronic device when the electronic device executes the same operation information. Therefore, the operation element information generated by the terminal device based on the operation information is the same as the operation element information generated by the electronic device when the electronic device executes the operation information, so that the terminal device can obtain the operation element information of the electronic device without going through the electronic device. In addition, in the embodiment of the present application, the electronic device does not need to share its own operation interface with the terminal device, so the risk of leaking the privacy of the user of the electronic device is avoided.

[0211] See also Fig.10 , Fig.10 1 is a flow chart of a method for recording operation guidance information of an electronic device provided in an embodiment of the present application, the method comprising:

[0212] Step S201: A first terminal device requests to establish a communication connection with a first electronic device corresponding to an electronic device type identifier.

[0213] Among them, the first terminal device can establish a communication connection with the first electronic device through various methods such as near field communication (such as WIFI (Wireless-Fidelity), Bluetooth, NFC (Near Field Communication, short-range wireless communication technology), etc.), local area network, distributed device network, etc.

[0214] Step S202: When the first terminal device fails to successfully establish a communication connection with the first electronic device, the first terminal device simulates the operation interface and / or physical hardware of the first electronic device.

[0215] Among them, step S201 and step S202 are a possible implementation of step S101.

[0216] For the case of multiple first electronic devices, for example, taking electronic devices such as PC, TV and speakers as examples, the operation order of the above-mentioned first electronic devices is: PC, TV and speakers, then the process of the first terminal device simulating the operation interface and / or physical hardware of the above-mentioned three electronic devices is: the first terminal device first simulates the operation interface and / or physical hardware of the PC, and receives the operation information input by the user in the simulated operation interface and / or physical hardware of the PC. After the first terminal device completes the simulation of the operation interface and / or physical hardware of the PC, the first terminal device simulates the operation interface and / or physical hardware of the TV, and receives the operation information input by the user in the simulated operation interface and / or physical hardware of the TV. After completing the simulation of the operation interface and / or physical hardware of the TV. The first terminal device simulates the operation interface and / or physical hardware of the speaker, and receives the operation information input by the user based on the simulated operation interface and / or physical hardware of the speaker.

[0217] It should be noted that the first terminal device may also simulate the operating interface and / or physical hardware of a PC, TV, and speaker in sequence, and then receive the operating information input by the user in the simulated operating interface and / or physical hardware of different electronic devices in sequence. The embodiment of the present application is not limited to this.

[0218] As a possible implementation, the first terminal device in the embodiment of the present application can simulate the operation interface and / or physical hardware of the first electronic device in the following manner: the first terminal device can call the virtual device model of the first electronic device. Then, the first terminal device simulates the operation interface and / or physical hardware of the first electronic device through the virtual device model.

[0219] Among them, the virtual device model of the first electronic device can be stored in the first terminal device, and can also be stored in the cloud server, so that the first terminal device can obtain the virtual device model of the first electronic device from the cloud server. Of course, the virtual device model of the first terminal device can also be stored in the first electronic device, so that the first terminal device can also obtain the virtual device model of the first electronic device from the first electronic device. Since the first terminal device simulates the operating interface and / or physical hardware of the first electronic device based on the virtual device model of the first electronic device, the first electronic device does not need to share its own operating interface with the first terminal device.

[0220] For example, the virtual device model may include physical features, software features, operating component features of the first electronic device, and a 3D / 2D device appearance / operation interface simulation model for display and operation in a virtual environment.

[0221] For example, a virtual device model may include the information shown in Table 1:

[0222] Table 1 Information contained in the virtual device model

[0223]

[0224] Referring to Table 1, the virtual device model may include a device identity (Identity, ID), a device type, a connection type, recordable operations, hardware information, software information, 2D / 3D model data, etc. Among them, the device ID is the identifier of the virtual device model. The device type is the device type of the first electronic device corresponding to the virtual device model. The connection type is the communication connection method of the virtual device model, such as WIFI, Bluetooth, NFC, local area network, distributed device network, etc.; the recordable operations are used to limit the operations that can be recorded in the operations of the virtual device model, which can be used to prevent privacy information leakage; the hardware information represents the physical characteristics of the first electronic device corresponding to the virtual device model; the software information represents the software characteristics of the application that can be run in the virtual device model. 2D / 3D model data is the appearance / operation interface simulation model of the first electronic device corresponding to the virtual device model, etc.

[0225] The first terminal device calls the virtual device model of the first electronic device, so that the operation interface and / or physical hardware of the first electronic device can be simulated in the first terminal device, that is, the physical characteristics, software characteristics, operation component characteristics, and 3D / 2D device appearance / operation interface simulation model of the first electronic device can be converted to the first terminal device for operation, and the operation element information generated by the first terminal device based on the operation information is the same as the operation element information generated by the first electronic device executing the operation information, so that the first terminal device can obtain the operation element information of the first electronic device without going through the first electronic device.

[0226] Steps S203 to 205 refer to steps S102 to 104 and will not be described in detail here.

[0227] Step S203 may be implemented in the following manner: the first terminal device receives operation information input by the user based on the operation interface and / or physical hardware.

[0228] See also Fig.11 , Fig.11 : is a flowchart of a method for recording operation guidance information of an electronic device provided in an embodiment of the present application. The method may include:

[0229] Step S301: See step S201, which will not be described again here.

[0230] Step S302: When the first terminal device successfully establishes a communication connection with the first electronic device, the first terminal device sends a time synchronization request to the first electronic device.

[0231] There may be multiple electronic device type identifiers, each of which corresponds to a first electronic device. Each first electronic device corresponds to multiple groups of operation element information. Each group of operation element information corresponds to a generation time and an electronic device type identifier.

[0232] The time synchronization request is used to request the first electronic device to adjust the time of the first electronic device to be consistent with the time of the first terminal device. Each first electronic device responds to the time synchronization request and adjusts the time of the first electronic device to be consistent with the time of the first terminal device.

[0233] In one application scenario, there may be multiple electronic device type identifiers, each of which corresponds to a first electronic device. The generation time of the operation element information of each first electronic device is determined based on the time of the corresponding first electronic device. Therefore, the reference time of the operation element information of each first electronic device may be different. The first electronic device is required to send the operation element information corresponding to each operation information to the first terminal device in real time. The first terminal device packages and groups each group of operation element information according to the time of receiving the operation element information and the electronic device type identifier corresponding to the operation element, and generates operation guide information corresponding to each first electronic device, which may result in low data transmission efficiency.

[0234] Based on this problem, in step S302, the time of each first electronic device is adjusted to be consistent, so that each first electronic device does not need to send the operation element information to the first terminal device in real time, thereby improving data transmission efficiency.

[0235] Step S303: The first terminal device sends an operation element information acquisition request to the first electronic device.

[0236] The operation element information acquisition request is used to request the first electronic device to acquire the operation element information generated by the first electronic device in response to the received operation information, and to send the operation element information to the first terminal device.

[0237] For example, when the first terminal device successfully establishes a communication connection with the first electronic devices such as PC, TV, smart refrigerator, speaker and router, the first terminal device sends an operation element information acquisition request to each first electronic device respectively. The user can perform preset operations on the first electronic device, and each first electronic device responds to the operation element information acquisition request, receives the operation information input by the user and obtains the operation element information generated by the first electronic device according to the operation information, and the first electronic device then sends the operation element information to the first terminal device.

[0238] Exemplarily, the operation element information may be recorded in the format shown in Table 2:

[0239] Table 2 Operation element information record format

[0240] Header Node ID Node Type Operation ID Time Stamp Data Tail Packet Header Node ID Node Type Operation ID Timestamp Operational Data Packet tail

[0241] Referring to Table 2, the specific format of the operation element information may include a data packet header, a node ID, a node type, an operation ID, a timestamp, operation data, and a data packet tail. Among them, the node ID represents the electronic device corresponding to the operation element information, which can be set by the first terminal device during the process of communicating and connecting with each first electronic device, and can correspond to the device ID in Table 1; the node type represents the device type of the electronic device, which can be synchronized with the device type in Table 1. For example, when the device type in Table 1 is PC, the node type in the operation element information generated by the PC device is the same as the device type in Table 1; the operation ID is an operation identifier for the electronic device or virtual device model, wherein the data structure corresponding to the operation corresponding to the operation ID can be extracted from the electronic device or virtual device model according to the combination of the node type and the operation ID. The above combination can distinguish the possible differences in the definition of the same type of operation by electronic devices with different operation interface sizes; the timestamp is used to define the time when the operation is executed, and can be used for recording the operation guide information.

[0242] Among them, for user privacy information involved in interface operation elements, such as account number, password, fingerprint, face and other authentication information, the electronic device can match and identify through a pre-established high-risk operation blacklist, and convert the process involving privacy information into a pre-set operation ID. For example, for information such as account number and password, the process of entering the account number and password can be converted into a pre-set operation ID to prevent the leakage of user privacy information.

[0243] Step S304: The first terminal device obtains the operation element information sent by the first electronic device.

[0244] Step S305: See step S104, which will not be described again here.

[0245] In a possible embodiment, corresponding to time synchronization processing of the electronic device, the first terminal device records the operation element information and generates operation guidance information, which may specifically include: the first terminal device determines the operation element information corresponding to each first electronic device according to the generation time and electronic device type identifier corresponding to each group of operation element information; the first terminal device packages the operation element information corresponding to each first electronic device, and generates operation guidance information corresponding to each first electronic device.

[0246] Among them, after the time of the first electronic device is adjusted to be consistent with the time of the first terminal device, the first terminal device can determine the operation element information corresponding to each first electronic device according to the electronic device type identifier corresponding to the operation element information, and the first terminal device packages the operation element information corresponding to each first electronic device in the order of generation time of the operation element information, and generates operation guide information corresponding to each first electronic device, thereby improving data transmission efficiency.

[0247] Furthermore, after the first electronic device adjusts the time of the first electronic device to be consistent with the time of the first terminal device, the first electronic device can aggregate the operation element information in the first electronic device and send it to the first terminal device together. The first terminal device packages the operation element information sent by each first electronic device and generates operation guidance information corresponding to each first electronic device. In an application scenario, during the operation of the electronic device on the first electronic device, the user may also provide additional explanations of the operation information by voice. Therefore, the operation guidance information sent by the first electronic device to the first terminal device may also include text information. The text information is: the first electronic device converts the audio information obtained for operation guidance together with the operation information within the time range corresponding to the operation information.

[0248] In an application scenario, when a user performs an operation on a first terminal device based on an operation interface and / or physical hardware of a simulated first electronic device, the operation information may be supplemented by voice at the same time. In this regard, the method may further include:

[0249] Within a time range corresponding to when the first terminal device acquires the operation information, the first terminal device acquires audio information used for performing operation guidance together with the operation information;

[0250] The first terminal device records the audio information in the operation guidance information.

[0251] Among them, in order to avoid the risk of leaking user voiceprint information, the first terminal device can convert the audio information into text information through NLP (Natural Language Processing) technology and record it in the operation guidance information. The second terminal device can convert the text information into audio information for playback through TTS (Text to Speech) technology.

[0252] Exemplarily, the above text information may include an electronic device type identifier and time information, wherein the electronic device type identifier is used to identify the electronic device corresponding to the text information, and the time information is used to identify the time when the terminal device acquires the text information.

[0253] For example, after the first terminal device obtains the user's audio information "Please adjust the routing antenna to XX angle", it converts the audio information into text information through NLP technology. The text information includes: electronic device type identification (XX model routing), time information (the time when the audio information is obtained) and "Please adjust the routing antenna to XX angle". When the second terminal device plays back the operation guidance information of the XX type routing, the second terminal device can convert "Please adjust the routing antenna to XX angle" into audio information through TTS technology, play the audio information at the time corresponding to the above time information, and then combine it with other operation guidance information to realize the operation guidance of the routing.

[0254] In one application scenario, after the first terminal device obtains the operation guidance information of the electronic device, the first terminal device can display the operation guidance information so that the user of the first terminal device can intuitively understand the electronic device and application corresponding to the operation guidance information.

[0255] For example, each operation element information may correspond to a generation time, an electronic device type identifier and an application identifier. The first terminal device obtains the electronic device type identifier and application identifier corresponding to the operation element information, establishes a link relationship between the operation element information and the corresponding electronic device type identifier and application identifier, and displays the electronic device type identifier and application identifier in the first terminal device.

[0256] Among them, the first terminal device can divide the operation element information according to the electronic device type identifier and the application identifier, divide the operation element information corresponding to a certain electronic device type identifier according to the application identifier, and establish a link relationship between the operation element information and the electronic device type identifier and the application identifier, and display the electronic device type identifier and the application identifier in a list or chart, so that the user can intuitively understand the electronic device and application corresponding to the operation guide information. At the same time, after receiving the operation information of clicking the electronic device type identifier and / or the application identifier, the first terminal device can retrieve the corresponding operation element information based on the above-mentioned link relationship and play it back through the corresponding virtual device model, so that the user of the first terminal device can check whether the recorded operation guide information is complete.

[0257] After receiving the input check completion information, the first terminal device can package the above operation element information and publish it to the cloud for security check. The cloud can perform security checks on the operation element information involving the specified APP in the operation element information, and insert security prompts for the operation element information that changes the system sensitive settings.

[0258] It should be noted that, after the first terminal device records the operation element information and generates the operation guidance information, the first terminal device can upload the operation guidance information to the cloud or send it to the second terminal device.

[0259] See also Fig.12 , Fig.12 1 is a flow chart of a method for recording operation guidance information of an electronic device provided in an embodiment of the present application, the method comprising:

[0260] Step S401: The second terminal device obtains operation guidance information for guiding the operation of the second electronic device, where the operation guidance information includes an electronic device type identifier and operation element information.

[0261] The number of the above electronic device type identifications may be one or more.

[0262] For example, the second terminal device may be Figure 1 or Figure 2 The terminal device B shown in FIG. 1 is a terminal device B. The second electronic device in this step and the first electronic device described above can be electronic devices of a one-to-one corresponding type. For details, see Figure 1 or Figure 2 The relevant contents in will not be repeated here.

[0263] In a possible implementation, step S401 may be implemented in the following manner: the second terminal device obtains the above-mentioned operation guidance information from the cloud.

[0264] In another possible implementation manner, step S401 may be implemented in the following manner: the second terminal device obtains the operation guidance information from the first terminal device.

[0265] Step S402: The second terminal device simulates the operation interface and / or physical hardware of the second electronic device corresponding to the electronic device type identifier.

[0266] Step S403: The second terminal device generates prompt information for operating the second electronic device based on the operation interface and / or the physical hardware response to the operation element information in the operation guidance information.

[0267] Among them, the second terminal device can extract and parse the operation element information one by one according to the execution order of each operation element information in the operation guidance information, and respond to the parsed operation element information based on the operation interface and / or physical hardware simulated by the second terminal device.

[0268] See also Fig.13 As shown in (a) of FIG. 1 , the second terminal device displays at least the following through the display interface: Fig.13 The operation guidance APP300 shown in (a) in FIG.

[0269] The second terminal device detects that the user clicks Fig.13 After the operation guide APP300 shown in (a) of FIG. 1 is opened, the second terminal device opens the operation guide APP300 and displays the following Fig.13 The interface shown in (b).

[0270] like Fig.13 As shown in (b), the second terminal device responds to the electronic device type identifier in the operation guidance information and displays controls of one or more electronic devices corresponding to the electronic device type identifier for the user to click. Fig.13 As shown in (c) of FIG. 1 , the user can click on the controls of one or more electronic devices in the display interface of the second terminal device, and then click on the "confirm" control to select the clicked one or more electronic devices. In response to the user clicking on the selected one or more electronic devices, the second terminal device simulates the operation interface and / or physical hardware of the one or more electronic devices selected by the user, such as Fig.13 As shown in (d), the second terminal device can simulate the operation interface and / or physical hardware of one or more electronic devices selected by the user by clicking in the second operation interface area 301 of the second terminal device to display the operation guidance information of the electronic devices.

[0271] For example, take the second electronic device as a personal computer. Fig.14 As shown in (a) of FIG. 1 , the second terminal device displays the simulated personal computer operation interface 301 and the prompt message "Please click "Control Panel"" displayed in the operation interface 301. After a certain period of time (for example, 5 seconds), the second terminal device displays Fig.14 The operation interface of the personal computer shown in (b) and the prompt message "Please click "Programs and Features"", after a certain period of time (for example, 5 seconds), the second terminal device displays Fig.14 The operation interface of the personal computer shown in (c) and the prompt message "Please click "Foxmail"", after a certain period of time (for example, 5 seconds), the second terminal device displays Fig.14 The operation interface of the personal computer and the prompt message "Please click "Uninstall"" shown in (d) in the figure. The user can uninstall the Foxmail program in the personal computer according to the above prompt message.

[0272] For example, take the second electronic device as a router. Fig.15 As shown, the second terminal device displays the physical hardware of the simulated router and a prompt message of "Please adjust the angle of the antenna in the direction of XX", and the user can adjust the angle of the antenna of the router according to the prompt message.

[0273] Among them, the second terminal device can draw a highlighted border on the edge of the button that needs to be operated, and attach relevant operation prompts, such as text or arrow pointing, or sliding animation logo, where the operation guide can be a pre-recorded animation. The user is guided to operate the physical button by the guide logo displayed on the screen interface. If the application corresponding to the operation element information is not detected in the electronic device, the electronic device can download the application from the network and install it.

[0274] In the above-mentioned method for obtaining operation guidance information of an electronic device, after the second terminal device obtains the operation guidance information for operating the second electronic device, it simulates the operation interface and / or physical hardware of the second electronic device corresponding to the electronic device type identifier, and responds to the above-mentioned operation guidance information based on the operation interface and / or physical hardware, and the second terminal device simulates the functions and properties of the operation interface and / or physical hardware of the second electronic device. The second terminal device and the second electronic device both produce the same result when executing the same operation information. Therefore, the operation guidance effect of the second terminal device responding to the operation guidance information through the simulated operation interface and / or physical hardware is the same as the operation guidance effect of the second electronic device responding to the operation guidance information, so that the second terminal device can respond to the operation guidance information without going through the second electronic device, thereby being able to guide the user to operate the second electronic device.

[0275] As a possible embodiment, the above step S402 may specifically include:

[0276] In the case that the second electronic device is an electronic device without an operation guidance information response unit, the second terminal device simulates the operation interface and / or physical hardware of the second electronic device; wherein the operation guidance information response unit includes a display unit and / or an audio playback unit.

[0277] The second terminal device can determine whether the second electronic device corresponding to the electronic device type identifier has an operation guidance information response unit based on the electronic device type identifier. For ease of description, the electronic device having the operation guidance information response unit is referred to as the third electronic device, and the electronic device not having the operation guidance information response unit is referred to as the fourth electronic device.

[0278] Exemplarily, taking the second electronic devices such as PC, TV, smart refrigerator, speaker and router as an example, the third electronic devices such as PC, TV, smart refrigerator and speaker have a display unit and / or audio playback unit in the operation guidance information response unit, and the fourth electronic devices such as router do not have the operation guidance information response unit. Therefore, for the operation guidance information corresponding to the fourth electronic device, the second terminal device simulates the operation interface and / or physical hardware of the fourth electronic device, and responds to the operation guidance information corresponding to the fourth electronic device based on the operation interface and / or physical hardware.

[0279] For example, the second terminal device can call the virtual device model of the second electronic device, and simulate the operating interface and / or physical hardware of the second electronic device through the virtual device model. Among them, the virtual device model of the second electronic device can be stored in the terminal device or in the cloud server, which is not limited in the embodiments of the present application. The virtual device model can include the physical characteristics of the electronic device, software characteristics, operating component characteristics, and a 3D / 2D device appearance / operation interface simulation model for display and operation in a virtual environment, etc. For a detailed description of the virtual device model, please refer to Table 1 and related content.

[0280] In some embodiments, the operation guidance information may be operation guidance information corresponding to multiple second electronic devices, so the above-mentioned electronic device type identifier may be multiple, each electronic device type identifier corresponds to a second electronic device, each second electronic device corresponds to an operation guidance information, each operation guidance information includes multiple groups of operation element information, and each group of operation element information corresponds to a generation time.

[0281] Correspondingly, responding to the operation guidance information based on the operation interface and / or physical hardware in step S403 may specifically include: the second terminal device parses the operation guidance information corresponding to each second electronic device in turn to obtain multiple groups of operation element information corresponding to each second electronic device; the second terminal device responds to each group of operation element information in sequence through the operation interface and / or physical hardware of the second electronic device corresponding to each group of operation element information in the order of generation time corresponding to each group of operation element information.

[0282] Exemplarily, taking electronic devices such as PC, TV and speakers as examples, the order of generation time of the operating element information of the above electronic devices is the operating element information corresponding to the PC, the operating element information corresponding to the TV and the operating element information corresponding to the speakers. The second terminal device first simulates the operating interface of the PC, and responds to the operating element information corresponding to the PC through the simulated operating interface of the PC to display the corresponding prompt information. The second terminal device then simulates the operating interface of the TV, and responds to the operating element information corresponding to the TV through the simulated operating interface of the TV to display the corresponding prompt information. Finally, the second terminal device simulates the physical hardware of the speaker, and responds to the operating element information corresponding to the speaker through the simulated physical hardware of the speaker to display the corresponding prompt information.

[0283] In some embodiments, the second terminal device responds to the operation guidance information based on the operation interface and / or physical hardware. Specifically, the second terminal device can determine the position of the operation element information in the operation interface according to the operation interface layout in the operation element information, and display the prompt information drawn based on the operation element information at this position.

[0284] Exemplarily, when there is a display interface that needs to be operated in the operation element information corresponding to a certain operation guidance step in the operation guidance information, the second terminal device can extract the button element information in the corresponding operation element information and determine the position of the button element information in the operation interface, and display the prompt information drawn based on the operation element information in the simulated operation interface according to the position, for example, in the form of a highlight box or arrow guide in the simulated operation interface; in the case where there is text information, the second terminal device can use TTS to convert the text information into audio information for playback.

[0285] Among them, the second terminal device can detect the elements of the UI interface of the second electronic device operation interface simulated by the second terminal device through the system service, and locate the position, virtual key operation, screen sliding, mobile phone posture adjustment, physical key, etc. in the relevant application according to the operation element information (such as software package name, relative position of operation point, order of operation in the list, etc.), so as to give the user intuitive prompts. In addition, for the operation guidance information corresponding to the third electronic device, the second terminal device can send the operation guidance information to the corresponding third electronic device, and the fourth electronic device responds to the operation guidance information, or the second terminal device can also simulate the operation interface and / or physical hardware of the third electronic device, and respond to the operation guidance information corresponding to the third electronic device based on the operation interface and / or physical hardware. Based on this, the above method may include: when the second electronic device is an electronic device with an operation guidance information response unit, the second terminal device sends the operation guidance information to the second electronic device.

[0286] In some embodiments, the above-mentioned operation guidance information may include interface operation element information for the operation interface of the second electronic device, hardware operation element information for the physical hardware of the second electronic device, and text information converted from audio information used for operation guidance.

[0287] Among them, the interface operation element information is used to instruct the display unit of the second electronic device to display the interface operation element information, the hardware operation element information is used to instruct the audio playback unit of the second electronic device to play the hardware operation element information, and the text information is used to instruct the audio playback unit of the second electronic device to convert the text information into audio information for playback. For example, in the scenario where the user operates the speaker to adjust the volume, the speaker can generate a hardware operation guide element of "adjust the volume adjustment button of XX type speaker", and the speaker can play the audio information "please adjust the volume adjustment button" for the hardware operation guide element through the audio playback unit.

[0288] In some embodiments, for a second electronic device having an operation guidance information response unit, if the second terminal device cannot establish a communication connection with the second electronic device, the second terminal device simulates the operation interface and / or physical hardware of the electronic device and responds to the operation guidance information based on the operation interface and / or physical hardware.

[0289] Furthermore, the above method may also include: the second terminal device responds to the received operation instruction and sends the operation instruction to the corresponding second electronic device so that the second electronic device executes the operation instruction; wherein, the operation instruction may be an operation instruction based on the operation guidance information received by the second terminal device after the second terminal device responds to the operation guidance information based on the operation interface and / or physical hardware.

[0290] It can be understood that after the second terminal device responds to the operation guidance information based on the operation interface and / or physical hardware, the user can input operation instructions for operating the second electronic device on the operation interface and / or physical hardware simulated by the second terminal device, and the second terminal device sends the operation instructions to the corresponding second electronic device, so that the second electronic device can be operated through the second terminal device without the user having to perform operations on the second electronic device.

[0291] It should be noted that for GUI (Graphic User interaction) operations, the second terminal device can respond to operation element information, and the user can perform operations on the simulated operation interface and / or physical hardware. The second terminal device converts the user's operation information into operation instructions and sends them to the corresponding second electronic device, and the second electronic device executes the operation instructions; for physical hardware operations, such as buttons, sliders, knobs, posture adjustments, antenna adjustments, etc., which must be completed by operating physical hardware, the second terminal device can generate prompt information to remind the user to perform manual operations.

[0292] Exemplarily, the second terminal device can simulate the operating interface and / or physical hardware of the second electronic device in an AR / VR manner, and then combine technologies such as human motion recognition to obtain operating instructions and send the operating instructions to the corresponding second electronic device, so that the second electronic device executes the operating instructions.

[0293] In one application scenario, after the second terminal device obtains the operation guidance information, it can perform a signature verification on the operation guidance information and check the integrity of the operation guidance information. After the signature verification and integrity check on the operation guidance information pass, the second terminal device searches for a second electronic device corresponding to the electronic device type identifier, and displays the searched second electronic device in the form of a device list or device icon, etc., for the user to click.

[0294] If the number and type of the second electronic devices searched by the second terminal device are different from the number and type of the electronic device type identifiers in the operation guidance information, the second terminal device can generate a reminder message for reminding the user to check the working status of the corresponding second electronic device, or call a virtual device model of the second electronic device that has not been searched, and replay the operation guidance information of the second electronic device through the virtual device model. In addition, the second terminal device can save the operation guidance information locally or in a cloud space, so that the user can query the operation guidance information through relevant analysis software or smart assistants to execute the operation guidance information again.

[0295] The embodiment of the present application also provides a method for recording electronic device operation guidance information, which can be applied to a first electronic device. The method includes the following steps:

[0296] The first electronic device receives an operation element information acquisition request sent by the first terminal device;

[0297] The first electronic device receives the input operation information in response to the operation element information acquisition request, and acquires the operation element information generated based on the operation information;

[0298] The first electronic device sends the operation element information to the first terminal device.

[0299] After the first electronic device receives the operation element information acquisition request, the first electronic device may receive the operation information generated by the user operating on the operation interface and / or physical hardware of the first electronic device. The operation element information may include interface operation element information for the operation interface and / or hardware operation element information for the physical hardware.

[0300] In the above-mentioned method for recording operation element information of an electronic device, after the first electronic device receives the operation element information acquisition request sent by the terminal device, it responds to the operation element information acquisition request, receives the input operation information and obtains the operation element information generated based on the operation information, and sends the operation element information to the first terminal device, so that the first terminal device records the operation element information to generate operation guidance information corresponding to the electronic device. Since the operation element information does not involve content that may leak the user privacy of the electronic device compared to the operation interface image in the traditional technology, it can reduce the risk of leaking the user privacy in the process of guiding the operation of the electronic device; and the operation guidance element information does not involve the size of the operation interface, so it can be better suitable for electronic devices with different operation interface sizes compared to the operation interface image in the traditional technology; and the operation element information requires less space than the operation interface image, so the electronic device has lower requirements on the network in the process of transmitting the operation element information to the terminal device, which can reduce the data transmission delay.

[0301] In some embodiments, the method may further include: the first electronic device receives a time synchronization request sent by the first terminal device, and adjusts the time of the first electronic device to be consistent with the time of the first terminal device based on the time synchronization request;

[0302] Correspondingly, the first electronic device sends the operation element information to the first terminal device, which may specifically include: the first electronic device aggregates the operation element information generated by the first electronic device and sends the aggregated information to the first terminal device.

[0303] The embodiment of the present application also provides a method for responding to electronic device operation guidance information, which can be applied to a second electronic device. The method includes the following steps:

[0304] The second electronic device receives the operation guidance information sent by the second terminal device, where the operation guidance information includes multiple groups of operation element information, and each group of operation element information corresponds to a generation time;

[0305] The second electronic device responds to each set of operation element information in sequence based on the generation time of each set of operation element information.

[0306] In the above-mentioned method of responding to operation guidance information of an electronic device, after the second electronic device receives the operation guidance information sent by the second terminal device, it responds to each group of operation element information in the operation guidance information in sequence based on the generation time of each group of operation element information in the operation guidance information. Since the operation element information does not involve content that may leak the user privacy of the electronic device compared to the operation interface image in the traditional technology, it can reduce the risk of leaking user privacy during the operation guidance process of the electronic device; and the operation guidance element information does not involve the size of the operation interface, so it can be better suitable for electronic devices with different operation interface sizes compared to the operation interface image in the traditional technology.

[0307] In some embodiments, the operation element information includes interface operation element information for the operation interface of the second electronic device and / or hardware operation element information for the physical hardware of the second electronic device; the interface operation element information includes the operation interface layout and operation element information corresponding to the state change of the operation interface. The second electronic device displays the above interface operation element information through a display unit, plays the above hardware operation element information through an audio playback unit, and converts the above text information into audio information for playback through the audio playback unit.

[0308] The second electronic device responds to the operation guidance information, which may specifically be: the second electronic device determines the position of the operation element information in the operation interface according to the operation interface layout in the operation element information, and displays the prompt information drawn based on the operation element information at the position.

[0309] Exemplarily, when there is a display interface that needs to be operated in the operation element information corresponding to a certain operation guidance step in the operation guidance information, the second electronic device can extract the key element information in the corresponding operation element information and determine the position of the key element information in the operation interface, and display the prompt information drawn based on the operation element information in the simulated operation interface according to the position, for example, in the form of a highlight box or arrow guidance in the operation interface; in the case where there is text information, the second electronic device can use TTS to convert the text information into audio information for playback.

[0310] Corresponding to the electronic device operation guidance information recording method applied to the first terminal device in the above embodiment, Fig.16 A structural block diagram of an electronic device operation guidance information recording device 500 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0311] See also Fig.16 The electronic device operation guidance information recording device in the embodiment of the present application may include a first simulation unit 501, a first receiving unit 502, a first acquisition unit 503 and a recording unit 504.

[0312] A first simulation unit 501 is configured to simulate, in response to the received electronic device type identifier, an operation interface and / or physical hardware of a first electronic device corresponding to the electronic device type identifier;

[0313] A first receiving unit 502, configured to receive operation information based on an operation interface and / or physical hardware;

[0314] A first acquisition unit 503, used to acquire operation element information generated based on the operation information;

[0315] The recording unit 504 is used to record the operation element information and generate operation guide information.

[0316] In the above-mentioned electronic device operation guide information recording device, the first terminal device responds to the received electronic device type identification, simulates the operation interface and / or physical hardware of the first electronic device corresponding to the electronic device type identification, receives operation information based on the operation interface and / or physical hardware, and obtains operation element information generated based on the operation information. Since the first terminal device simulates the operation interface and / or physical hardware of the first electronic device, the first terminal device has the functions and attributes of the operation interface and / or physical hardware of the first electronic device, and the first terminal device and the first electronic device both produce the same result when executing the same operation information. Therefore, the operation element information generated by the first terminal device based on the operation information is the same as the operation element information generated by the first electronic device when executing the operation information, so that the first terminal device can obtain the operation element information of the first electronic device without going through the first electronic device.

[0317] Corresponding to the method for obtaining electronic device operation guidance information applied to the second terminal device in the above embodiment, Fig.17 A structural block diagram of an apparatus 600 for obtaining electronic device operation guidance information provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0318] See also Fig.17 The device for obtaining electronic device operation guidance information in the embodiment of the present application may include a second obtaining unit 601, a second simulation unit 602 and a first responding unit 603.

[0319] The second acquisition unit 601 is used to acquire operation guidance information for operating the second electronic device, where the operation guidance information includes an electronic device type identifier;

[0320] A second simulation unit 602, configured to simulate an operation interface and / or physical hardware of a second electronic device corresponding to the electronic device type identifier;

[0321] The first responding unit 603 is configured to respond to the operation guidance information based on the operation interface and / or the physical hardware.

[0322] The above-mentioned device for obtaining operation guidance information of an electronic device, after the second terminal device obtains the operation guidance information for operating the second electronic device, simulates the operation interface and / or physical hardware of the second electronic device corresponding to the electronic device type identifier, and responds to the above-mentioned operation guidance information based on the operation interface and / or physical hardware, and the second terminal device simulates the functions and properties of the operation interface and / or physical hardware of the second electronic device, and the second terminal device and the second electronic device both produce the same result when executing the same operation information, so the operation guidance effect of the second terminal device responding to the operation guidance information through the simulated operation interface and / or physical hardware is the same as the operation guidance effect of the second electronic device responding to the operation guidance information, so that the second terminal device can respond to the operation guidance information without going through the second electronic device, thereby being able to guide the user to operate the electronic device.

[0323] Corresponding to the electronic device operation guidance information recording method applied to the first electronic device in the above embodiment, the embodiment of the present application also provides an electronic device operation guidance information recording device. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0324] The electronic device operation guidance information recording device in the embodiment of the present application may include a second receiving unit, a second responding unit and a sending unit. The second receiving unit is used to receive an operation element information acquisition request sent by the first terminal device; the second responding unit is used to respond to the operation element information acquisition request, receive input operation information, and acquire operation element information generated based on the operation information; the sending unit is used to send the operation element information to the first terminal device.

[0325] The above-mentioned electronic device operation guidance information recording device, after the first electronic device receives the operation element information acquisition request sent by the first terminal device, responds to the operation element information acquisition request, receives the input operation information and obtains the operation element information generated based on the operation information, and sends the operation element information to the first terminal device, so that the first terminal device records the operation element information to generate operation guidance information corresponding to the first electronic device. Since the operation element information does not involve content that may leak the user privacy of the electronic device compared to the operation interface image in the traditional technology, it can reduce the risk of leaking the user privacy in the process of guiding the operation of the electronic device; and the operation guidance element information does not involve the size of the operation interface, so it can be better suitable for electronic devices with different operation interface sizes compared to the operation interface image in the traditional technology; and the operation element information requires less space than the operation interface image, so the electronic device has lower requirements on the network in the process of transmitting the operation element information to the terminal device, which can reduce the data transmission delay.

[0326] Corresponding to the method of responding to electronic device operation guidance information applied to a second electronic device in the above embodiment, the embodiment of the present application also provides a device for responding to electronic device operation guidance information. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0327] The device for responding to the operation guidance information of the electronic device in the embodiment of the present application may include a third receiving unit and a third responding unit. The third receiving unit is used to receive the operation guidance information sent by the second terminal device, the operation guidance information includes multiple groups of operation element information, each group of operation element information corresponds to a generation time; the third responding unit is used to respond to each group of operation element information in sequence based on the generation time of each group of operation element information.

[0328] The above-mentioned electronic device operation guidance information recording device, after the second electronic device receives the operation guidance information sent by the second terminal device, it responds to each group of operation element information in the operation guidance information in sequence based on the generation time of each group of operation element information in the operation guidance information. Since the operation element information does not involve content that may leak the user privacy of the electronic device compared to the operation interface image in the traditional technology, it can reduce the risk of leaking user privacy in the process of guiding the operation of the electronic device; and the operation guidance element information does not involve the size of the operation interface, so it can be better suitable for electronic devices with different operation interface sizes compared to the operation interface image in the traditional technology.

[0329] The present application also provides a terminal device, see Fig.18 The terminal device 700 may include: at least one processor 710, a memory 720, and a computer program stored in the memory 720 and executable on the at least one processor 710. When the processor 710 executes the computer program, the steps in any of the above-mentioned method embodiments are implemented, for example Figure 6 Steps S101 to S104 in the embodiment shown, or as Fig.12 Steps S401 to S403 in the illustrated embodiment. Alternatively, when the processor 710 executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Fig.16 The functions of modules 501 to 504 shown in FIG. Fig.17 Functions of modules 601 to 603 are shown.

[0330] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 720, and executed by the processor 710 to complete the present application. One or more modules / units may be a series of computer program segments capable of completing specific functions, and the program segments are used to describe the execution process of the computer program in the terminal device 700.

[0331] Those skilled in the art will understand that Fig.18 It is only an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, buses, etc.

[0332] The processor 710 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0333] The memory 720 may be an internal storage unit of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. The memory 720 is used to store computer programs and other programs and data required by the terminal device. The memory 720 may also be used to temporarily store data that has been output or is to be output.

[0334] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0335] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0336] The embodiment of the present application also provides a computer program product comprising instructions, and when the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0337] The embodiment of the present application also provides a chip system, which may include a memory and a processor, and the processor executes a computer program stored in the memory to implement one or more steps in any of the above methods. The chip system may be a single chip or a chip module composed of multiple chips.

[0338] The embodiment of the present application also provides a chip system, which may include a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement one or more steps in any of the above methods. The chip system may be a single chip or a chip module composed of multiple chips.

[0339] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0340] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0341] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for recording electronic equipment operation guidance information, characterized in that: include: The first terminal device simulates the operation interface and / or physical hardware of the first electronic device corresponding to the electronic device type identifier in response to the received electronic device type identifier; The first terminal device receives operation information based on the operation interface and / or the physical hardware; The first terminal device acquires operation element information generated according to the operation information; The first terminal device records the operation element information to generate operation guidance information, wherein the operation guidance information includes an electronic device type identifier; The first terminal device simulates, in response to the received electronic device type identifier, an operation interface and / or physical hardware of the first electronic device corresponding to the electronic device type identifier, including: The first terminal device requests to establish a communication connection with the first electronic device; In a case where the first terminal device fails to successfully establish a communication connection with the first electronic device, the first terminal device simulates the operation interface and / or the physical hardware of the first electronic device.

2. The method according to claim 1, characterized in that The first terminal device simulates the operation interface and / or the physical hardware of the first electronic device, including: The first terminal device calls the virtual device model of the first electronic device, and simulates the operation interface and / or the physical hardware of the first electronic device through the virtual device model.

3. The method according to claim 1, characterized in that: The method comprises: When the first terminal device successfully establishes a communication connection with the first electronic device, the first terminal device obtains the operation element information of the first electronic device from the first electronic device.

4. The method according to claim 3, characterized in that The first terminal device acquires the operation element information of the first electronic device from the first electronic device, including: The first terminal device sends an operation element information acquisition request to the first electronic device, where the operation element information acquisition request is used to request the first electronic device to obtain the operation element information generated by the first electronic device in response to the received operation information, and send the operation element information to the first terminal device.

5. The method according to claim 3, characterized in that: There are multiple electronic device type identifiers, each of which corresponds to a first electronic device, each of which corresponds to multiple groups of operation element information, and each group of operation element information corresponds to a generation time and an electronic device type identifier; The method further comprises: The first terminal device sends a time synchronization request to each first electronic device corresponding to the plurality of electronic device type identifiers, wherein the time synchronization request is used to request the first electronic device to adjust the time of the first electronic device to be consistent with the time of the first terminal device; The first terminal device records the operation element information to generate operation guidance information, including: The first terminal device determines, according to the electronic device type identifier corresponding to each group of the operation element information, the operation element information corresponding to each first electronic device; The first terminal device packages the operation element information corresponding to each first electronic device in the order of generation time of the operation element information, and generates operation guide information corresponding to each first electronic device.

6. The method according to claim 5, characterized in that The operation guidance information also includes text information, and the text information is text information obtained by converting the audio information obtained by the first electronic device for operation guidance together with the operation information within a time range corresponding to the acquisition of the operation information.

7. The method according to claim 1 or 3, characterized in that: The operation element information includes interface operation element information for an operation interface of the electronic device, and / or hardware operation element information for physical hardware of the electronic device; The interface operation element information includes the operation interface layout and the operation element information corresponding to the state change of the operation interface.

8. The method according to claim 1, characterized in that The method further comprises: Within a time range corresponding to when the first terminal device obtains the operation information, the first terminal device obtains audio information used for performing operation guidance together with the operation information; The first terminal device records the audio information in the operation guidance information.

9. The method according to claim 8, characterized in that The first terminal device records the audio information in the operation guidance information, including: The first terminal device converts the audio information into text information, and records the text information in the operation guide information.

10. A method for obtaining electronic equipment operation guidance information, characterized in that: include: The second terminal device acquires operation guidance information for operating the second electronic device, wherein the operation guidance information includes an electronic device type identifier and operation element information; The second terminal device simulates the operation interface and / or physical hardware of the second electronic device corresponding to the electronic device type identifier; The second terminal device responds to the operation element information based on the operation interface and / or the physical hardware to generate prompt information for operating the second electronic device.

11. The method according to claim 10, characterized in that The second terminal device simulates the operation interface and / or the physical hardware of the second electronic device corresponding to the electronic device type identifier, including: In the case where the second electronic device is an electronic device without an operation guidance information response unit, the second terminal device simulates the operation interface and / or the physical hardware of the second electronic device; Wherein, the operation guidance information response unit includes a display unit and / or an audio playing unit.

12. The method according to claim 11, characterized in that The second terminal device simulates the operation interface and / or the physical hardware of the second electronic device, including: The second terminal device calls the virtual device model of the second electronic device, and simulates the operation interface and / or the physical hardware of the second electronic device through the virtual device model.

13. The method according to claim 11, characterized in that There are multiple electronic device type identifiers, each of which corresponds to a second electronic device, each of which corresponds to an operation guide information, each of which includes multiple groups of operation element information, and each group of operation element information corresponds to a generation time; The responding to the operation guidance information based on the operation interface and / or the physical hardware includes: The second terminal device sequentially parses the operation guidance information corresponding to each of the second electronic devices to obtain multiple groups of operation element information corresponding to each of the second electronic devices; The terminal device responds to each set of operation element information in sequence through the operation interface and / or physical hardware of the second electronic device corresponding to each set of operation element information in accordance with the order of generation time corresponding to each set of operation element information.

14. The method according to claim 11, characterized in that The method comprises: In the case that the second electronic device is an electronic device having an operation guidance information response unit, the second terminal device sends the operation guidance information to the second electronic device.

15. The method according to claim 14, characterized in that The operation guidance information includes interface operation element information for the user interface of the second electronic device, hardware operation element information for the physical hardware of the second electronic device, and text information converted from audio information used for operation guidance; Among them, the interface operation element information is used to instruct the display unit of the second electronic device to display the interface operation element information, the hardware operation element information is used to instruct the audio playback unit of the second electronic device to play the hardware operation element information, and the text information is used to instruct the audio playback unit of the second electronic device to convert the text information into audio information for playback.

16. A terminal device, characterized in that: include: one or more processors, memory, and display screens; The memory and the display screen are coupled to the one or more processors, the memory is used to store computer program codes, and the computer program codes include computer instructions; When the one or more processors execute the computer instructions, the terminal device executes the method according to any one of claims 1 to 9, or the method according to any one of claims 10 to 15.

17. A chip system, characterized in that: The chip system includes a processor, which is coupled to a memory. The processor executes a computer program stored in the memory to implement the method as described in any one of claims 1 to 9 and / or the method as described in any one of claims 10 to 15.

Images