Display method and electronic device

Abstract

A display method, an electronic device, and a computer storage medium are provided, to simplify an interaction method, reduce user operations, and improve efficiency in a multi-device interconnection scenario.SOLUTION: A display method, applied to a first device connected to a second device, includes displaying a first interface including address information, receiving a first user operation, and in response to the first user operation, identifying the address information to determine intent information, where the intent information indicates that a location represented by the address information is a destination for navigation, and sending the intent information to the second device, where the intent information is used for the second device to display a second interface, and the second interface is used for providing a navigation service to a user, where the destination is the location represented by the address information.SELECTED DRAWING: Figure -1

Description

[Technical Field]

[0001] This application claims priority to Chinese Patent Application No. 202111493706.2 entitled "DISPLAY METHOD AND ELECTRONIC DEVICE", filed with the State Intellectual Property Office of China on December 8, 2021, and to Chinese Patent Application No. 202210093485.8 entitled "DISPLAY METHOD AND ELECTRONIC DEVICE", filed with the State Intellectual Property Office of China on January 26, 2022, which are incorporated herein by reference in their entireties.

[0002] This application relates to the field of computer technology, and in particular to display methods and electronic devices. [Background technology]

[0003] In a scenario where multiple devices are connected and communicate with each other (e.g., a distributed scenario), a user can use any one of the devices independently or multiple devices simultaneously (services on multiple devices may be related, e.g., a video on a smartphone is projected onto a smart TV for playback). However, electronic devices in this scenario lack simple and efficient interactions, making user operations complex. For example, in a scenario where a smartphone is connected to an on-board computer, if a user receives a communication message containing location information using the smartphone, the user needs to start a map application on the on-board computer, set a destination to the location indicated by the location information, and implement location-based navigation. This results in complex operations. If the user is driving, this affects driving safety and degrades the user experience. Summary of the Invention

[0004] The embodiments of this application disclose a display method and an electronic device to simplify the interaction method, reduce user operations, and improve efficiency in a multi-device interconnection scenario.

[0005] According to a first aspect, an embodiment of the present application provides a display method applied to a first device connected to a second device, the first device including first information related to a first service; receiving a first user operation; recognizing the first interface in response to the first user operation to determine intent information indicating execution of a first instruction, the first instruction being used to implement the first service; and transmitting the intent information to a second device, the intent information being used by the second device to execute the first instruction to generate second information, the second information being used by the second device to display the second interface.

[0006] In some embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0007] In some embodiments, the second information is used by the second device to display the second interface and play the first audio. In some other embodiments, the second information is used by the second device to play the first audio, and the second device does not display the second interface.

[0008] In the above method, when the first device receives a first user operation, it recognizes the user's intention based on the currently displayed first interface, and the second device executes a first command. The first command is used to implement a first service corresponding to the recognized intention information. In this way, the user does not need to manually operate the first device or the second device to trigger the implementation of the first service. This reduces user operations and makes the interaction method in a multi-device interconnection scenario more efficient and convenient.

[0009] In a possible implementation, the first interface further includes third information, the third information relating to the second device. Recognizing the first interface to determine the intent information includes recognizing the first information to determine fourth information and recognizing the third information to determine fifth information, where the fourth information indicates executing a first instruction and the fifth information indicates executing a second instruction, the second instruction being used to implement a second service, and determining from the fourth information and the fifth information that the intent information is the fourth information according to a first preset rule, where the first preset rule includes at least one of: a device type of the second device being a preset device type; services supported by the second device including a first service; and a priority of the first service being higher than a priority of the second service.

[0010] In some embodiments, the first information and the third information are instant messaging messages, and the first preset rule includes a receipt time of the first information being later than a receipt time of the third information.

[0011] In the above method, the first device may further determine, according to the first preset rule, intention information that is more suitable for the user's requirement in the current scenario, so that the interaction accuracy is further improved and the user experience is better.

[0012] In a possible implementation, the first information is location information, the first service is a navigation service, the second service is different from the first service, the first preset rule includes that the device type of the second device is the preset device type, and the preset device type is an in-vehicle device.

[0013] In a possible implementation, the first information is video information, the first service is a video playback service, the second service is different from the first service, the first preset rule includes that the device type of the second device is the preset device type, and the preset device type is a smart TV and a smart screen.

[0014] In a possible implementation, the first information is information indicating a first location, the first service is a navigation service, and the second information is display information generated by performing a navigation operation relative to the first location.

[0015] In some embodiments, the first device is a smartphone and the second device is an in-vehicle device.

[0016] In the above method, when a first user operation is received while the first device is displaying a first interface including location information, a navigation service for the location information may be implemented using the second device. In this way, the user does not need to manually input location information and manually trigger a navigation operation on the second device, thereby making the interaction method in a multi-device interconnection scenario more efficient and convenient.

[0017] In a possible implementation, the first information is information indicating a first video, the first service is a video playback service, and the second information is display information generated by playing the first video.

[0018] In some embodiments, the first device is a smartphone and the second device is a smart television.

[0019] In the above method, when a first user operation is received while a first device is displaying a first interface including video information, a service for playing the video information may be implemented using a second device. In this way, a user does not need to manually search for the video information on the second device and manually trigger the video playback service, so that the interaction method in a multi-device interconnection scenario is more efficient and convenient.

[0020] In a possible implementation, the first information is information indicating a first recipe, the first service is a cooking service, and the second information is display information generated to implement the cooking service corresponding to the first recipe.

[0021] In some embodiments, the first device is a smartphone and the second device is a smart food processor.

[0022] In the above method, when a first user operation is received while a first device is displaying a first interface including recipe information, a cooking service corresponding to the recipe information may be implemented using a second device. In this way, a user does not need to manually search for recipe information on the second device and manually trigger a cooking service on the second device, thereby making the interaction method in a multi-device interconnection scenario more efficient and convenient.

[0023] In a possible implementation, the first information is information indicating a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but does not include the answer to the first question.

[0024] In some embodiments, the first device is a smartphone and the second device is a tablet computer or a learning machine.

[0025] In the above method, when a first user operation is received while a first device is displaying a first interface including questions and answers, the second device may display the questions but not the answers. In this way, a child can practice the questions on the second device, and a parent does not need to manually search for questions on the second device or manually trigger a test paper generation service. Therefore, the interaction method is convenient and accurate, and can fully meet the needs of parents and children.

[0026] In a possible implementation, the first user operation is, for example, a shake operation, a swing operation, a tap operation with a knuckle of a finger, a slide operation with a knuckle of a finger, a multi-finger tap operation, a multi-finger slide operation, or the like.

[0027] In the above method, the first user operation is simple and convenient, and the user does not need to perform complex operations to trigger the implementation of the first service, thus lowering the interaction threshold and making the use more convenient for the user.

[0028] According to a second aspect, the present application provides another display method applicable to a first device connected to a second device, the first device including first information related to a first service; receiving a first user operation, wherein in response to the first user operation, recognizing the first interface to determine intent information, executing first instructions based on the intent information to generate second information, and transmitting the second information to the second device, wherein the second information is used by the second device to display the second interface.

[0029] In some embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0030] In some embodiments, the second information is used by the second device to display the second interface and play the first audio. In some other embodiments, the second information is used by the second device to play the first audio, and the second device does not display the second interface.

[0031] In the above method, upon receiving a first user operation, the first device may recognize the user's intention based on the currently displayed first interface and execute a first command indicated by the recognized intention information, and the second device may output multimedia data by executing the first command. A first service corresponding to the first command may be understood to be implemented by the second device. In this way, the user does not need to manually operate the first device or the second device to trigger the implementation of the first service. This reduces user operations and makes the interaction method in a multi-device interconnection scenario more efficient and convenient.

[0032] In a possible embodiment, the first interface further includes third information, the third information relating to a second service. Recognizing the first interface to determine the intent information includes recognizing the first information to determine fourth information and recognizing the third information to determine fifth information, where the fourth information indicates executing a first instruction and the fifth information indicates executing a second instruction, the second instruction being used to implement the second service; and determining from the fourth information and the fifth information that the intent information is the fourth information according to a first preset rule, where the first preset rule includes the device type of the second device being a preset device type and / or the priority of the first service being higher than the priority of the second service.

[0033] In some embodiments, the first information and the third information are instant messaging messages, and the first preset rule includes a receipt time of the first information being later than a receipt time of the third information.

[0034] In the above method, the first device may further determine, according to the first preset rule, intention information that is more suitable for the user's requirement in the current scenario, so that the interaction accuracy is further improved and the user experience is better.

[0035] In a possible implementation, the first information is location information, the first service is a navigation service, the second service is different from the first service, the first preset rule includes that the device type of the second device is the preset device type, and the preset device type is an in-vehicle device.

[0036] In a possible implementation, the first information is information indicating a first location, the first service is a navigation service, and the second information is display information generated by performing a navigation operation relative to the first location.

[0037] In a possible implementation, the first information is information indicating a first video, the first service is a video playback service, and the second information is display information generated by playing the first video.

[0038] In a possible implementation, the first information is information indicating a first recipe, the first service is a cooking service, and the second information is display information generated to implement the cooking service corresponding to the first recipe.

[0039] In a possible implementation, the first information is information indicating a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but does not include the answer to the first question.

[0040] In a possible implementation, the first user operation is, for example, a shake operation, a swing operation, a tap operation with a knuckle of a finger, a slide operation with a knuckle of a finger, a multi-finger tap operation, a multi-finger slide operation, or the like.

[0041] According to a third aspect, this application provides another display method applied to a second device connected to a first device. The second device includes: receiving intent information transmitted by the first device, the intent information being determined by recognizing a first interface displayed when the first device receives a first user operation, the first interface including first information, the first information being related to a first service; executing first instructions based on the intent information to generate second information, the first instructions being used to implement the first service; and displaying the second interface based on the second information.

[0042] In some other embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0043] In some embodiments, the second information is used by the second device to display the second interface and play the first audio. In some other embodiments, the second information is used by the second device to play the first audio, and the second device does not display the second interface.

[0044] In the above method, when the first device receives a first user operation, the first device may recognize the user's intention based on the currently displayed first interface and send the recognized intention information to the second device. The second device may execute the first instruction indicated by the intention information to implement the first service. In this way, the user does not need to manually operate the first device or the second device to trigger the implementation of the first service. This reduces user operations and makes the interaction method in a multi-device interconnection scenario more efficient and convenient.

[0045] In a possible implementation, the first information is information indicating a first location, the first service is a navigation service, and the second information is display information generated by performing a navigation operation relative to the first location.

[0046] In a possible implementation, the first information is information indicating a first video, the first service is a video playback service, and the second information is display information generated by playing the first video.

[0047] In a possible implementation, the first information is information indicating a first recipe, the first service is a cooking service, and the second information is display information generated to implement the cooking service corresponding to the first recipe.

[0048] In a possible implementation, the first information is information indicating a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but does not include the answer to the first question.

[0049] In a possible implementation, the first user operation is, for example, a shake operation, a swing operation, a tap operation with a knuckle of a finger, a slide operation with a knuckle of a finger, a multi-finger tap operation, a multi-finger slide operation, or the like.

[0050] According to a fourth aspect, the present application provides another display method applied to a second device connected to a first device. The method includes receiving first information transmitted by the first device, the first information being information generated by executing first instructions, the first instructions being used to implement a first service, the first instructions being instructions to be executed as indicated by intention information, the intention information being determined by recognizing a first interface displayed when the first device receives a first user operation, the first interface including second information, the second information being related to the first service; and displaying the second interface based on the first information.

[0051] In some other embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0052] In some embodiments, the second information is used by the second device to display the second interface and play the first audio. In some other embodiments, the second information is used by the second device to play the first audio, and the second device does not display the second interface.

[0053] In the above method, upon receiving a first user operation, the first device may recognize the user's intention based on the currently displayed first interface and execute a first command indicated by the recognized intention information, and the second device may output multimedia data by executing the first command. A first service corresponding to the first command may be understood to be implemented by the second device. In this way, the user does not need to manually operate the first device or the second device to trigger the implementation of the first service. This reduces user operations and makes the interaction method in a multi-device interconnection scenario more efficient and convenient.

[0054] In a possible implementation, the second information is information indicating a first location, the first service is a navigation service, and the first information is display information generated by performing a navigation operation on the first location.

[0055] In a possible implementation, the second information is information indicating the first video, the first service is a video playback service, and the first information is display information generated by playing the first video.

[0056] In a possible implementation, the second information is information indicating a first recipe, the first service is a cooking service, and the first information is display information generated to implement the cooking service corresponding to the first recipe.

[0057] In a possible implementation, the second information is information indicating a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but does not include the answer to the first question.

[0058] In a possible implementation, the first user operation is, for example, a shake operation, a swing operation, a tap operation with a knuckle of a finger, a slide operation with a knuckle of a finger, a multi-finger tap operation, a multi-finger slide operation, or the like.

[0059] According to a fifth aspect, an embodiment of the present application provides an electronic device including one or more processors and one or more memories, the one or more memories coupled to the one or more processors, the one or more memories configured to store computer program code, the computer program code including computer instructions, and when the one or more processors execute the computer instructions, the communication device is enabled to perform a display method according to any possible implementation of any one of the above aspects.

[0060] According to a sixth aspect, an embodiment of the present application provides a computer storage medium having a computer program stored thereon, the computer program, when executed by a processor, causing the display method according to any one of the above possible implementations to be performed.

[0061] According to a seventh aspect, an embodiment of the present application provides a computer program product, which, when running on an electronic device, enables the electronic device to perform a display method according to any one of the possible implementations of the above aspects.

[0062] According to an eighth aspect, an embodiment of this application provides an electronic device. The electronic device includes a method or apparatus for performing any embodiment of this application. For example, the electronic device is a chip.

[0063] It should be understood that a description of a technical feature, technical solution, beneficial effect, or similar terminology in this application does not imply that all features and advantages can be implemented in any single embodiment. Conversely, a description of a feature or beneficial effect may be understood to indicate that a particular technical feature, technical solution, or beneficial effect is included in at least one embodiment. Thus, a description of a technical feature, technical solution, or beneficial effect in this specification does not necessarily refer to the same embodiment. Furthermore, technical features, technical solutions, and beneficial effects described in embodiments may be combined in any suitable manner. Those skilled in the art may understand that an embodiment may be implemented in a particular embodiment without one or more particular technical features, technical solutions, or beneficial effects. In other embodiments, additional technical features and beneficial effects may be further recognized in certain embodiments, which may not reflect all embodiments. [Brief explanation of the drawings]

[0064] The accompanying drawings used in the embodiments of this application will be described below.

[0065] [Figure 1A] 1 is a schematic diagram of the architecture of a communication system 10 according to an embodiment of the present application.

[0066] [Figure 1B] 1 is a schematic diagram of the architecture of another communication system 10 according to an embodiment of the present application.

[0067] [Figure 1C] FIG. 2 is a schematic diagram of yet another communication system architecture according to an embodiment of the present application;

[0068] [Figure 2A] 1 is a schematic diagram of a hardware structure of an electronic device 100 according to an embodiment of the present application.

[0069] [Figure 2B] 2 is a schematic diagram of a hardware structure of an electronic device 200 according to an embodiment of the present application.

[0070] [Figure 2C] 3 is a schematic diagram of a hardware structure of a network device 300 according to an embodiment of the present application.

[0071] [Figure 2D] 1 is a schematic diagram of the software architecture of an electronic device 100 according to one embodiment of the present application.

[0072] [Figure 3A-1] 1A-1C are schematic diagrams of several user interface embodiments according to embodiments of the present application. [Figure 3A-2] 1A-1C are schematic diagrams of several user interface embodiments according to embodiments of the present application. [Figure 3B] 1A-1C are schematic diagrams of several user interface embodiments according to embodiments of the present application. [Figure 3C]1A-1C are schematic diagrams of several user interface embodiments according to embodiments of the present application.

[0073] [Figure 4A-1] 10A-10C are schematic diagrams of some further user interface embodiments according to embodiments of the present application. [Figure 4A-2] 10A-10C are schematic diagrams of some further user interface embodiments according to embodiments of the present application. [Figure 4B-1] 10A-10C are schematic diagrams of some further user interface embodiments according to embodiments of the present application. [Figure 4B-2] 10A-10C are schematic diagrams of some further user interface embodiments according to embodiments of the present application.

[0074] [Figure 4C-1] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application. [Figure 4C-2] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application.

[0075] [Figure 5A] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application. [Figure 5B] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application.

[0076] [Figure 6A] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application. [Figure 6B] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application.

[0077] [Figure 7A-1] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application. [Figure 7A-2] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application.

[0078] [Figure 7B-1] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application. [Figure 7B-2] FIG. 10 is a schematic diagram of another user interface embodiment, according to an embodiment of the present application.

[0079] [Figure 8] 1 is a schematic diagram of a user interface according to an embodiment of the present application.

[0080] [Figure 9] 1 is a schematic flowchart of a display method according to an embodiment of the present application.

[0081] [Figure 10] 1 is a schematic flowchart of another display method according to an embodiment of the present application;

[0082] [Figure 11] 1 is a schematic flowchart of yet another display method according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0083] The technical solutions according to the embodiments of this application are clearly and completely described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise specified, " / " indicates "or". For example, A / B may indicate A or B. The term "and / or" in this specification only describes the association relationship between associated objects, and indicates that three relationships may exist. For example, A and / or B may represent three cases: only A exists, both A and B exist, and only B exists. Additionally, in the description of the embodiments of this application, "plurality" means two or more.

[0084] The terms "first" and "second" referred to below are intended for descriptive purposes only and shall not be understood as an indication or implication of relative importance or an implicit indication of the number of technical features indicated. Thus, features qualified as "first" and "second" may explicitly or implicitly include one or more features. In the description of implementations of this application, unless otherwise specified, "plurality" means two or more.

[0085] Embodiments of the present invention may be applied to scenarios in which multiple devices are connected and communicate with each other, e.g., distributed scenarios. In this scenario, a user may use multiple devices simultaneously. In this case, the services of multiple devices may be associated, e.g., a video on a smartphone may be projected on a smart TV for playback. However, electronic devices in this scenario lack simple and efficient interaction, and user operation is complex. Specific examples are as follows:

[0086] Example 1: In a scenario where a smartphone is connected to an on-board computer, when a user receives a communication message containing location information using the smartphone, the user needs to start a map application on the on-board computer, set the destination to the location indicated by the location information, and implement location-based navigation. As a result, the operation is complicated. If the user is driving, this will affect driving safety and degrade the user experience.

[0087] Example 2: In a scenario where a smartphone is connected to a smart TV, if a user views information about a specific video on the smartphone (such as a synopsis or review) and wants to watch that video on the smart TV, the user must either search for the movie on the smart TV for playback, or must first launch a video application and movie playback interface on the smartphone, operate the projection controls, select the device to which the movie will be projected (i.e., the smart TV), and then project the movie onto the smart TV for viewing. This results in complex operations and poor interaction efficiency.

[0088] Example 3: In a scenario where a smartphone is connected to a smart food processor, if a user sees information about a specific recipe on the smartphone and wants to cook the corresponding dish using the smart food processor, the user needs to search for the recipe and cook it on the smart food processor, resulting in complicated operations and low interaction efficiency.

[0089] Example 4: In a scenario where a smartphone is connected to a tablet computer, a child may use the tablet computer or learning machine for study, and a parent may use the smartphone to search for related exercises. If a parent finds an exercise on the smartphone that they want their child to answer, the parent must search for the exercise again on the tablet computer or learning machine. This results in complicated operations and low interaction efficiency.

[0090] One embodiment of the present application provides a display method. In response to a user operation, a first device may recognize a currently displayed first interface, determine intention information, and implement a service indicated by the intention information via a second device. In this way, the user does not need to manually trigger the second device to implement the service indicated by the intention information, providing an efficient and convenient interaction method applicable to multi-device interconnection scenarios. This reduces user operations and improves the user experience.

[0091] For example, a smartphone (first device) may recognize a chat interface (first interface) including a location card (a message displaying a geographic location in card format) in response to a shake operation (user operation) and determine intent information. The intent information may indicate a navigation service to perform navigation to the location indicated by the location card, and the intent information may be obtained based on the location card. In this case, the smartphone may execute the navigation service based on the intent information and optionally indicate to set the location indicated by the location card as a destination in a map application for navigation.

[0092] A communication system 10 according to an embodiment of this application will be described below.

[0093] FIG. 1A illustrates an example of a schematic diagram of the architecture of a communication system 10 according to one embodiment of the present application.

[0094] As shown in FIG. 1A, communication system 10 may include electronic device 100, electronic device 200, and network device 300.

[0095] In some embodiments, electronic device 100 may be connected to at least one electronic device 200 via a wired and / or wireless connection. Wired connections include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB), a coaxial cable, or an optical fiber. Wireless connections include, for example, Bluetooth, wireless fidelity (Wi-Fi), near field communication (NFC) technology, or ultra-wideband (UWB). Electronic device 100 may communicate with electronic device 200 via a connection (e.g., Bluetooth or Wi-Fi). In this case, the information transmission speed between electronic device 100 and electronic device 200 is high, allowing for the transmission of large amounts of information.

[0096] In some other embodiments, the electronic device 100 may be connected to the network device 300 in a wired and / or wireless manner, and the network device 300 may be connected to at least one electronic device 200 in a wired and / or wireless manner. The electronic device 100 may communicate with the electronic device 200 via the network device 300. For example, the electronic device 100 is a smartphone, the electronic device 200 is a vehicle, and the network device 300 is a cloud server that provides a HUAWEI HiCar function. In this case, the connection and projection between the electronic device 100 and the electronic device 200 may be implemented using the HUAWEI HiCar function.

[0097] In some other embodiments, electronic device 100 may not be connected to electronic device 200, but may establish a connection to electronic device 200 and then communicate with electronic device 200. Electronic device 200 may also be understood to be an electronic device that is not connected to electronic device 100 but can communicate with electronic device 100. Optionally, electronic device 100 may store connection information (e.g., Bluetooth address and password, Wi-Fi name and password) of at least one electronic device 200 and use the connection information to connect to at least one electronic device 200 (e.g., send information including the password to the electronic device 200 corresponding to the Bluetooth address to request establishment of a connection). Optionally, the connection information of electronic device 200 may be obtained when electronic device 100 previously connected to electronic device 200. Optionally, the connection information of electronic device 200 may be obtained by electronic device 100 via network device 300. For example, after logging in to a specific account, the electronic device 100 may obtain connection information of the electronic device 200 that previously logged in to the account. The manner in which the electronic device 100 obtains the connection information of the electronic device 200 is not limited in this application.

[0098] The electronic devices and network devices shown in FIG. 1A are merely examples and are not limited to any particular device form.

[0099] In this application, the electronic device 100 may be a mobile terminal such as a mobile phone, a tablet computer, a handheld computer, or a personal digital assistant (PDA); a smart home device such as a smart TV, a smart camera, or a smart food processor; a wearable device such as a smart band, a smart watch, or smart glasses; or another device such as a desktop, laptop, notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a smart screen, or a learning machine. The description of the electronic device 200 is similar and will not be described in detail again. In the embodiments of this application, the specific types of the electronic device 100 and the electronic device 200 are not specifically limited.

[0100] In this application, network device 300 may include at least one server. In some embodiments, any server may be a hardware server. In some embodiments, any server may be a cloud server.

[0101] FIG. 1B shows an example of a schematic diagram of the architecture of another communication system 10 according to an embodiment of the present application.

[0102] As shown in FIG. 1B, the electronic device 100 in the communication system 10 may include an interface analysis module, an intent analysis module, and an intent trigger module, and the electronic device 200 in the communication system 10 may include an output module.

[0103] When the electronic device 100 detects a user action, for example, when detecting the user action via the sensor module 180 shown in FIG. 2A, the electronic device 100 may report an event corresponding to the user action (sometimes called a trigger event) to the interface analysis module.

[0104] Upon receiving the trigger event, the interface analysis module of the electronic device 100 may recognize the user interface displayed by the electronic device 100 and obtain an interface recognition result. In some embodiments, the interface analysis module may recognize and analyze the layer structure and text of the current interface through keyword extraction, natural language understanding (NLU), etc. The interface recognition result may include, for example, text information and structural information indicating the structure in the user interface. The interface recognition result may be, for example, data in XML format, data in JSON format, or data in another existing format. Note that the interface recognition result is not limited to these and may alternatively be data in a customized format. The interface analysis module may send the interface recognition result to the intent analysis module.

[0105] In some embodiments, the interface analysis module may recognize several pages in the displayed user interface and obtain interface recognition results. For example, the user interface displayed by electronic device 100 is a split interface. Assume that the split-screen interface includes a page of a first application and a page of a second application, and the application last operated by the user is the first application. The interface analysis module may recognize the page of the first application and obtain corresponding interface recognition results. This is not a limitation. The interface analysis module may also recognize the page of the application selected by the user. The manner in which information that needs to be recognized in the user interface is determined is not limited in this application.

[0106] The intention analysis module of the electronic device 100 may perform intent recognition based on the interface recognition result and obtain intent information. The intent information may be specific data obtained by performing interface recognition and intent recognition on a user interface displayed by the electronic device 100. The intent information may be, for example, data in XML format, JSON format, or another existing format. Note that the intent information is not limited to these and may alternatively be data in a customized format. In some embodiments, the intent information indicates a goal that needs to be achieved from the user's perspective. Optionally, the intent information indicates that the implemented service corresponds to some service information in the user interface displayed by the electronic device 100. In some embodiments, the interface recognition result includes first structural information and first text information. In this case, the intent analysis module may recognize the first structural information, determine an interface structure indicated by the first structural information, and then obtain intent information based on the first text information and the determined interface structure. For example, the intention analysis module may obtain an interface structure of a location card and an interface structure of a text box through recognition, determine based on the interface structure of the location card that the type of the text information "Beijing Railway Station" included in the location card is address information, and determine based on the interface structure of the text box that the type of the text information "Meet here" included in the text box is chat information, and obtain intent information indicating navigating to the geographical location "Beijing Railway Station" based on the address information "Beijing Railway Station" and the chat information "Meet here". The intent analysis module may then send the intent information to the intent trigger module.

[0107] In some embodiments, the intent analysis module may further determine whether the intent information is valid. The intent analysis module transmits the intent information to the intent trigger module only when it determines that the intent information is valid. For example, when the intent information indicates navigating to the geographic location “Beijing Railway Station,” the intent analysis module determines whether the address information “Beijing Railway Station” in the intent information corresponds to an actual valid geographic location on the map. The intent analysis module transmits the intent information to the intent trigger module only when it determines that the address information “Beijing Railway Station” in the intent information corresponds to an actual valid geographic location on the map. In another example, when the intent information indicates playing a movie named “Movie 1,” the intent analysis module determines whether the video information “Movie 1” in the intent information corresponds to an actual video that can be played. The intent analysis module transmits the intent information to the intent trigger module only when it determines that the video information “Movie 1” in the intent information corresponds to an actual video that can be played.

[0108] An intention trigger module of electronic device 100 may perform an intention operation based on the intention information. In some embodiments, the intention trigger module may analyze the intention information to obtain specific instructions and invoke the instructions to perform the intention operation. In some embodiments, from a user's perspective, the intention information may indicate a goal that needs to be achieved, and the intention operation may correspond to a user operation the user needs to perform to achieve the goal. In other words, the user can control electronic device 100 to perform the intention operation only after performing multiple user operations. In some embodiments, the intention trigger module may invoke a corresponding service module to perform the intention operation. For example, when the intention information indicates navigating to the geographic location "Beijing Railway Station," the intention trigger module may invoke a navigation module of a map application to perform the intention operation, i.e., set the destination to the geographic location "Beijing Railway Station" and perform navigation. After performing the intention operation, the intention trigger module may send corresponding multimedia data (e.g., audio and video streams corresponding to the navigation service) to an output module of electronic device 200.

[0109] After receiving the multimedia data sent by the intent trigger module of the electronic device 100, the output module of the electronic device 200 may output the multimedia data, for example, play an audio stream corresponding to the navigation service and display a video stream corresponding to the navigation service.

[0110] In some embodiments, the interface analysis module of electronic device 100 may include an interface analysis model. The interface analysis model is used to recognize the displayed user interface and obtain interface recognition results. Optionally, the interface analysis module may use content within the user interface displayed by electronic device 100 as input for the interface analysis model to obtain output interface recognition results. For example, the interface analysis module may use interface content including address information in text format as input to obtain output text structure and / or address information, or use interface content including address information in the form of a card (e.g., the location card described above) as input to obtain output card structure and / or address information.

[0111] In some embodiments, the intent analysis module of electronic device 100 may include an intent analysis model that is used to perform intent recognition via the intent analysis module. Optionally, the intent analysis module may use the interface recognition results as input for the intent analysis model to obtain output intent information.

[0112] The foregoing examples are not limiting. The interface analysis module and the intent analysis module of the electronic device 100 may be disposed within the same fusion module. The fusion module may include a fusion model, which is used to determine intent information based on the displayed user interface. Optionally, the fusion module may use the displayed interface content as input to the fusion model to obtain output intent information. For example, interface content including address information is used as input to the fusion model to obtain output intent information. The intent information indicates navigation to a location indicated by the address information.

[0113] In some embodiments, the electronic device 100 may train the interface analysis model and / or the intent analysis model, or the electronic device 100 may train the fusion model. In some other embodiments, the network device 300 in the communication system 10 may train the interface analysis module and / or the intent analysis model and transmit the trained interface analysis module and / or the trained intent analysis model to the electronic device 100, or the network device 300 may train the fusion model and transmit the trained fusion model to the electronic device 100. The manner in which the network device 300 transmits the interface analysis module and / or the intent analysis model or the fusion model to the electronic device 100 is not limited in this application. For example, the electronic device 100 may transmit a request message to the network device 300 requesting to obtain the aforementioned models after receiving a user operation. In another example, the network device 300 may transmit the aforementioned models to the electronic device 100 at preset time intervals, for example, once a week. In another example, when the version of a model is updated, the network device 300 may send the model with the updated version to the electronic device 100.

[0114] In some embodiments, electronic device 100 or network device 300 may train an interface analysis model by using the content within the user interface as input, using the structure and text contained in the user interface as input. Example inputs and outputs are similar to the previous examples in which an interface analysis model is used to recognize a displayed user interface. Details will not be repeated here.

[0115] In some embodiments, the electronic device 100 or the network device 300 may train an intent analysis model by using the interface recognition results as input and the corresponding intent operations and / or corresponding intent information as output.

[0116] In some embodiments, electronic device 100 or network device 300 may train a fusion model by using content in a user interface as input and corresponding intent actions and / or corresponding intent information as output. For example, the fusion model may be trained by using user interface content including address information as input and an intent action (setting a destination to a location indicated by the address information and performing navigation) as output. Alternatively, the fusion model may be trained by using user interface content without address information as input and corresponding user actions (e.g., actions performed by a user while electronic device 100 is displaying the user interface) as output. Alternatively, the fusion model may be trained by using user interface content without address information as input and information indicating a lack of navigation intent as output.

[0117] This is not limited to the example of FIG. 1B . In some other embodiments, at least one of the interface analysis module, the intention analysis module, and the intention trigger module may be a module included in the electronic device 200, rather than a module included in the electronic device 100. For example, the intention trigger module is a module included in the electronic device 200. See FIG. 1C for a specific example. As shown in FIG. 1C , after receiving the intention information sent by the intention analysis module of the electronic device 100, the intention trigger module of the electronic device 200 performs an intention operation based on the intention information and sends multimedia data corresponding to the intention operation to the output module, which outputs the multimedia data. Other details are similar to those of FIG. 1B , and will not be described again in detail.

[0118] The electronic device 100, the electronic device 200, and the network device 300 according to the embodiments of this application will be described below.

[0119] FIG. 2A shows an example of a schematic diagram of the hardware configuration of electronic device 100.

[0120] The embodiments will be described in detail below using electronic device 100 as an example. The electronic device 100 shown in Figure 2A is only an example and may have more or fewer components than those shown in Figure 2A, a combination of two or more components, or a different arrangement of components. The components shown in Figure 2A may be implemented using hardware, including one or more signal processing and / or application specific integrated circuits, software, or a combination of hardware and software.

[0121] As shown in FIG. 2A , the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identity module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, an optical proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0122] It may be understood that the structure shown in this embodiment of the present invention does not constitute a particular limitation on electronic device 100. In some other embodiments of this application, electronic device 100 may include more or fewer components than those shown in the figures, a combination of some components, a split of some components, or a different arrangement of components. The components shown in the figures may be implemented by hardware, software, or a combination of software and hardware.

[0123] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), etc. The different processing units may be separate components or may be integrated into one or more processors.

[0124] The controller may generate operation control signals according to the instruction operation code and the time sequence signal to complete the complete control of instruction reading and instruction execution.

[0125] The memory may be disposed within the processor 110 and configured to store instructions and data. In some embodiments, the memory within the processor 110 is cache memory. The memory may store instructions or data that are just used or periodically used by the processor 110. If the processor 110 needs to use the instructions or data again, the processor may retrieve the instructions or data directly from the memory. This avoids repeated accesses and reduces the latency of the processor 110, improving system efficiency.

[0126] In some embodiments, processor 110 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM), a universal serial bus (USB) interface, etc.

[0127] The I2C interface is a bidirectional synchronous serial bus and includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus groups. The processor 110 may be separately coupled to the touch sensor 180K, a charger, a flashlight, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, and the processor 110 communicates with the touch sensor 180K through the I2C bus interface to implement touch functionality for the electronic device 100.

[0128] A MIPI interface may be configured to connect the processor 110 to peripheral components such as a display 194 or a camera 193. MIPI interfaces include a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 via the CSI to implement photo-taking functionality of the mobile device 100. The processor 110 communicates with the display 194 via the DSI interface to implement display functionality of the electronic device 100.

[0129] It may be understood that the interface connection relationships between modules shown in this embodiment of the present invention are merely examples for explanation and do not constitute limitations on the structure of the electronic device 100. It should be noted that in other embodiments of the present invention, the electronic device 100 may use an interface connection method different from that of the above embodiment, or may use a combination of multiple interface connection methods.

[0130] The charging management module 140 is configured to receive a charging input from a charger.

[0131] The power management module 141 is configured to connect to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, etc.

[0132] The wireless communication functionality of the electronic device 100 may be implemented via antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, a baseband processor, and the like.

[0133] Antenna 1 and Antenna 2 are configured to transmit and receive electromagnetic signals. Each antenna in electronic device 100 may be configured to cover one or more communication frequency bands. Different antennas may be further multiplexed to improve antenna utilization. For example, Antenna 1 may be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, antennas may be used in combination with tuning switches.

[0134] Mobile communication module 150 can provide a solution for wireless communication, including 2G / 3G / 4G / 5G, applied to electronic device 100. Mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. Mobile communication module 150 may receive electromagnetic waves via antenna 1, perform processing such as filtering or amplification on the received electromagnetic waves, and send the processed electromagnetic waves to a modem processor for demodulation. Mobile communication module 150 may further amplify signals modulated by the modem processor and convert the amplified signals into electromagnetic waves for emission via antenna 1. In some embodiments, at least some functional modules in mobile communication module 150 may be disposed within processor 110. In some embodiments, at least some functional modules of mobile communication module 150 may be disposed in the same device as at least some modules of processor 110.

[0135] The modem processor may include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-frequency signal. The demodulator is configured to demodulate a received electromagnetic signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to a baseband processor for processing. The low-frequency baseband signal is processed by the baseband processor and then transmitted to an application processor. The application processor outputs a sound signal via an audio device (such as, but not limited to, speaker 170A, receiver 170B, etc.) or displays an image or video on display 194. In some embodiments, the modem processor may be a separate component. In some other embodiments, the modem processor may be separate from processor 110 and may be disposed within the same device as mobile communication module 150 or another functional module.

[0136] The wireless communication module 160 may provide solutions for wireless communication applied to the electronic device 100, including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC) technology, infrared (IR) technology, etc. The wireless communication module 160 may be one or more components integrating at least one communication processor module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 further receives signals to be transmitted from the processor 110, performs frequency modulation and amplification on the signals, and converts the processed signals into electromagnetic waves for emission via the antenna 2.

[0137] In some embodiments, in electronic device 100, antenna 1 and mobile communication module 150 are coupled, and antenna 2 and wireless communication module 160 are coupled, and electronic device 100 can communicate with networks and other devices by using wireless communication technologies, which may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technologies, etc. GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a BeiDou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite-based augmentation system (SBAS).

[0138] The electronic device 100 may implement display functionality via a GPU, a display 194, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric computations and render images. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0139] Display 194 is configured to display images, videos, etc. Display 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0140] The electronic device 100 may implement imaging functionality via an ISP, a camera 193, a video codec, a GPU, a display 194, an application processor, and the like.

[0141] The ISP is configured to process data fed back by the camera 193. For example, during photography, the shutter is pressed and light is transmitted through the lens to the camera's photoreceptor. The light signal is converted into an electrical signal, which is then transmitted to the ISP for processing, converting the electrical signal into a visible image. The ISP may also perform algorithmic optimization for image noise, brightness, etc. The ISP may also optimize parameters such as exposure and color temperature for the photography scenario. In some embodiments, the ISP may be disposed within the camera 193.

[0142] The camera 193 is configured to capture still images or video. An optical image of an object is generated through a lens and projected onto a photoreceptor. The photoreceptor may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photoreceptor converts the optical signal into an electrical signal, which is then transferred to an ISP for conversion to a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB, YUV, etc. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0143] The digital signal processor is configured to process digital signals, and may process other digital signals in addition to the digital image signal.

[0144] Video codecs are configured to compress or decompress digital video.

[0145] The NPU is a neural network (NN) computing processor that can rapidly process input information by referencing the structure of biological neural networks, for example, by referencing the transfer modes between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent perception of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, and text understanding.

[0146] The external memory interface 120 may be configured to connect to an external memory card, such as a microSD card, to expand the storage capabilities of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement data storage functions. For example, music, video, and other files may be stored on the external memory card.

[0147] The internal memory 121 may be configured to store computer-executable program code. The executable program code includes instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., audio playback function or image playback function), etc. The data storage area may store data created during use of the electronic device 100 (e.g., audio data, address book, etc.). Additionally, the internal memory 121 may include high-speed random access memory or non-volatile memory, such as at least one magnetic disk storage device, flash memory, or universal flash storage (UFS). The processor 110 executes instructions stored in the internal memory 121 and / or instructions stored in memory disposed within the processor to perform various functional applications and data processing of the electronic device 100.

[0148] The electronic device 100 may implement audio functionality, such as music playback and recording, via an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, an application processor, etc.

[0149] Audio module 170 is configured to convert digital audio information into analog audio signals for output, and is also configured to convert analog audio input into digital audio signals. Audio module 170 may be further configured to encode and decode audio signals.

[0150] Speaker 170A, also known as a "loudspeaker," is configured to convert audio electrical signals into sound signals. Electronic device 100 may use speaker 170A to listen to music or receive calls in hands-free mode.

[0151] Receiver 170B, also referred to as an "earpiece," is configured to convert electrical audio signals into sound signals. When receiving an incoming call or speech information via electronic device 100, receiver 170B may be placed near a person's ear to hear the sound.

[0152] The microphone 170C, also referred to as a "mike" or "mic," is configured to convert sound signals into electrical signals. When a user makes a call or sends a voice message, the user may emit a sound near the microphone 170C through the user's mouth to input the sound signal into the microphone 170C. At least one microphone 170C may be disposed within the electronic device.

[0153] The headset jack 170D is configured to connect to a wired headset.

[0154] The pressure sensor 180A is configured to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display 194. When a touch operation is performed on the display 194, the electronic device 100 detects the strength of the touch operation via the pressure sensor 180A. The electronic device 100 may also calculate a touch position based on the detection signal of the pressure sensor 180A.

[0155] Gyroscope sensor 180B may be configured to determine a motion attitude of electronic device 100. In some embodiments, angular velocity of electronic device 100 about three axes (i.e., axes x, y, and z) may be determined using gyroscope sensor 180B. Gyroscope sensor 180B may also be used in image stabilization scenarios, navigation scenarios, and physical gaming scenarios.

[0156] Barometric pressure sensor 180C is configured to measure barometric pressure. In some embodiments, electronic device 100 calculates altitude via barometric pressure values ​​measured by barometric pressure sensor 180C to assist in positioning and navigation.

[0157] The magnetic sensor 180D includes a Hall sensor. The electronic device may use the magnetic sensor 180D to detect whether the flip cover is open or closed.

[0158] The acceleration sensor 180E may detect acceleration in various directions (typically three axes) of the electronic device 100. The magnitude and direction of gravity may be detected when the electronic device 100 is stationary. The acceleration sensor 180E may be further configured to identify the orientation of the electronic device, which may be used in applications such as switching between landscape and portrait modes, a pedometer, etc.

[0159] The distance sensor 180F is configured to measure distance. The electronic device 100 may measure distance using an infrared method or a laser method.

[0160] Ambient light sensor 180L is configured to sense the brightness of ambient light.

[0161] Fingerprint sensor 180H is configured to collect fingerprints, and electronic device 100 may use collected fingerprint characteristics to implement fingerprint-based unlocking, application lock access, fingerprint-based photography, fingerprint-based call answering, and the like.

[0162] The temperature sensor 180J is configured to detect a temperature.

[0163] The touch sensor 180K may also be referred to as a "touch device." The touch sensor 180K may be disposed on the display 194, and the touch sensor 180K and the display 194 may comprise a touchscreen, also referred to as a "touch screen." The touch sensor 180K is configured to detect touches occurring at or near the touch sensor. The touch sensor may forward the detected touches to an application processor to determine the type of touch event. A visual output associated with the touch may be provided via the display 194. In some other embodiments, the touch sensor 180K may alternatively be disposed on the surface of the mobile device 100 at a location different from the display 194.

[0164] The bone conduction sensor 180M may acquire a vibration signal.

[0165] Buttons 190 include a power button, a volume button, etc. Buttons 190 may be mechanical buttons or touch buttons. Electronic device 100 may receive button input and generate button signal input related to user settings and function control of electronic device 100.

[0166] The motor 191 may generate a vibration prompt.

[0167] The indicator 192 may be an indicator light and may be configured to indicate charging status and power changes, or may be configured to indicate messages, missed calls, notifications, etc.

[0168] The SIM card interface 195 is configured to connect to a SIM card.

[0169] In some embodiments, electronic device 100 may detect a user operation via sensor module 180. In response to the user operation, processor 110 may perform intent recognition based on a user interface displayed by display 194. Based on the recognized intent information, electronic device 100 transmits instruction information to electronic device 200 via mobile communication module 150 and / or wireless communication module. After receiving the instruction information, electronic device 200 may output multimedia data corresponding to the intent information, for example, display a navigation interface corresponding to the navigation intent.

[0170] For example, the electronic device 100 detects a touch operation performed by a user on the electronic device 100, such as tapping the display 194 with the knuckle of a finger or sliding the knuckle, two fingers, or three fingers across the display 194, via the pressure sensor 180A and / or the touch sensor 180K. In another example, the electronic device 100 detects a user's shaking operation and hand swing operation via the gyroscope sensor 180B and / or the acceleration sensor 180E. In another example, the electronic device 100 detects a user's gesture operation via the camera 193. The module for detecting a user operation is not limited in this application.

[0171] FIG. 2B shows an example of a schematic diagram of the hardware configuration of electronic device 200.

[0172] The following describes the embodiments in detail using the electronic device 200 as an example. The electronic device 200 shown in Figure 2B is only an example, and may have more or fewer components than those shown in Figure 2B, a combination of two or more components, or a different arrangement of components.

[0173] 2B, the electronic device 200 may include a processor 201, a memory 202, a wireless communication module 203, an antenna 204, and a display 205. Optionally, the electronic device 200 may further include a wired communication module (not shown).

[0174] Specifically, the processor 201 may be configured to read and execute computer-readable instructions. In a particular implementation, the processor 201 may mainly include a controller, an arithmetic logic unit, and registers. The controller is mainly responsible for decoding instructions and sending control signals for operations corresponding to the instructions. The arithmetic logic unit is mainly responsible for storing register calculation amounts, intermediate calculation results, etc. that are temporarily stored during instruction execution. In a particular implementation, the hardware architecture of the processor 201 may be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, an NP architecture, etc. In some embodiments, the processor 201 may further be configured to generate a signal, such as a Bluetooth broadcast signal or a beacon signal, to be transmitted externally by the wireless communication module 203.

[0175] The memory 202 is coupled to the processor 201 and configured to store various software programs and / or instruction groups. In certain implementations, the memory 202 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash storage devices, or other non-volatile solid-state storage devices. The memory 202 may also store an operating system, such as an embedded operating system such as uCOS, VxWorks, or RTLinux. The memory 202 may further store a communication program. The communication program may be used to communicate with the electronic device 100 or another device.

[0176] The wireless communication module 203 may include one or more of a WLAN communication module 203A and a Bluetooth communication module 203B. Optionally, the Bluetooth communication module 203B may be integrated with another communication module (e.g., the WLAN communication module 203A).

[0177] In some embodiments, one or more of the WLAN communication module 203A and the Bluetooth communication module 203B may monitor signals transmitted by another device, such as measurement signals or scanning signals, and transmit response signals, such as measurement responses or scanning responses, to allow the other device to discover the electronic device 200, and may establish a wireless communication connection to the other device using one or more of Bluetooth and WLAN, or other short-range wireless communication technologies, for data transmission.

[0178] In some other embodiments, the WLAN communication module 203A may transmit a signal, for example, broadcast a discovery signal or a beacon signal, so that the router may discover the electronic device 200 and establish a wireless communication connection to the router using WLAN, thereby connecting the electronic device 100 and the network device 300.

[0179] The wired communication module (not shown) may be configured to establish a connection to a device such as a router via a network cable and to be connected to the electronic device 100 and the network device 300 via the router.

[0180] The antenna 204 is configured to transmit and receive electromagnetic signals. The antennas of different communication modules may be multiplexed or independent of each other to improve antenna utilization. For example, the antenna of the Bluetooth communication module 203B may be multiplexed with the antenna of the WLAN communication module 203A.

[0181] Display 205 may be configured to display images, videos, etc. Display 205 includes a display panel. The display panel may be a liquid crystal display, an organic light emitting diode, an active matrix organic light emitting diode, a flexible light emitting diode, a quantum dot light emitting diode, etc. In some embodiments, electronic device 200 may include one or N displays 205, where N is a positive integer greater than 1.

[0182] In some embodiments, computing device 200 may further include sensors. See sensor module 180 shown in FIG. 2A for a specific example, the details of which will not be described again here.

[0183] In some embodiments, electronic device 200 may receive the instruction information transmitted by electronic device 100 using wireless communication module 203 and / or a wired communication module (not shown). Processor 201 may use display 205 to display a user interface corresponding to the intent information based on the instruction information, for example, a navigation interface corresponding to the navigation intent.

[0184] FIG. 2C shows an example of a schematic diagram of the hardware configuration of the network device 300.

[0185] 2C , the network device 300 may include one or more processors 301, communication interfaces 302, and memories 303. The processors 301, communication interfaces 302, and memories 303 may be connected via a bus or in another manner. In the embodiment of this application, an example will be described in which the processors 301, communication interfaces 302, and memories 303 are connected via a bus 304.

[0186] Specifically, the processor 301 may include one or more general-purpose processors, such as a CPU, and may be configured to run program code related to a device control method.

[0187] The communication interface 302 may be a wired interface (e.g., an Ethernet interface) or a wireless interface (e.g., a cellular network interface or a wireless local area network interface) and is configured to communicate with another node. In this embodiment of the application, the communication interface 302 may be specifically configured to communicate with the electronic device 100 and the electronic device 200.

[0188] The memory 303 may include volatile memory, such as RAM. Alternatively, the memory may include non-volatile memory, such as ROM, flash memory, HDD, or solid-state drive (SSD). Alternatively, the memory 303 may include a combination of the aforementioned types of memory. The memory 303 may be configured to store a group of program codes such that the processor 201A invokes the program codes stored in the memory 203A to implement the methods implemented by the server in the embodiments of this application. In this embodiment of this application, the memory 303 may alternatively be a storage array or the like.

[0189] In some embodiments, the network device 300 may include multiple servers, such as a web server, a background server, a download server, etc. For the hardware structure of the multiple servers, see the hardware structure of the network device 300 shown in FIG. 2C.

[0190] It should be understood that the network device 300 shown in FIG. 2C is merely one implementation of an embodiment of the present application. In an actual application, the network device 300 may alternatively include more or fewer components. This is not a limitation herein.

[0191] The software system of the electronic device 100 may use a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. For example, the software system of the layered architecture may be an Android system, a Huawei mobile services (HMS) system, or another software system. In this embodiment of the present application, the Android system of the layered architecture is used as an example to describe the software structure of the electronic device 100.

[0192] FIG. 2D shows an example of a schematic diagram of the hardware configuration of electronic device 100.

[0193] In a layered architecture, software is divided into layers, each with a distinct role and task. These layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers from top to bottom: the application layer, the application framework layer, the Android runtime and system / libraries layer, and the kernel layer.

[0194] The application layer may include a series of application packages.

[0195] As shown in FIG. 2D, the application package may include applications such as a camera, a map, a HiCar, a music, a chat application, an entertainment application, a home application, and a learning application.

[0196] The application framework layer provides the application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0197] As shown in Figure 2D, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, an intent forwarding service, and the like.

[0198] A window manager is configured to manage window programs. It may also get the size of the display, determine if there is a status bar, perform screen locking, take screenshots, etc.

[0199] Content providers are configured to store and retrieve data and make it accessible by applications, which may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0200] A view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system may be configured to build applications. A display interface may include one or more views. For example, a display interface that includes an SMS message notification icon may include a text display view and an image display view.

[0201] The telephone manager is configured to provide communication functionality for the electronic device 100, such as managing call status (including incoming calls, blocking calls, etc.).

[0202] The resource manager provides various resources such as localized strings, icons, pictures, layout files, and video files for your application.

[0203] A notification manager may be configured to carry notification messages, allowing applications to display notification information in the status bar. A notification manager may disappear automatically after a short pause without user interaction.

[0204] The intent transfer service may perform intent recognition based on an application at the application layer. In some embodiments, the intent transfer service may perform intent recognition based on the user interface of an application displayed by electronic device 100. Electronic device 100 may implement the recognized intent via electronic device 200. In some cases, a service on electronic device 100 used to implement the intent may be transferred to electronic device 200. In other cases, electronic device 100 may send the recognized intent to electronic device 200, and electronic device 200 may implement the recognized intent.

[0205] In some embodiments, the intent transfer service may provide services to a system application at the application layer to perform intent recognition for a third-party application at the application layer, for example, the system application is a HiCar application and the third-party application is a map application, a chat application, an entertainment application, a home application, a learning application, etc.

[0206] This is not limiting. In some other embodiments, the intention transfer service may be an embedded service of an application in the application layer. For example, a server corresponding to an application (sometimes referred to as an application server for short) may provide the intention transfer service to the application. When receiving a user operation, the electronic device 100 may send the content of the currently displayed user interface to the application server. The application server performs intent recognition based on the interface content and sends the recognized intent information to the electronic device 100. The electronic device 100 implements the intent information via the electronic device 200.

[0207] In some embodiments, the intent transfer service may correspond to the intent analysis module, optional page analysis module, and optional intent trigger module shown in Figure 1B. See the description of Figure 1B for details. The details will not be repeated here.

[0208] In some embodiments, the application in the application layer may correspond to the intent trigger module shown in Figure 1B. In some embodiments, the application in the application layer may correspond to the display module shown in Figure 1B.

[0209] The Android runtime includes the kernel library and virtual machine, and is responsible for scheduling and managing the Android system.

[0210] The core library contains two parts: the functions that need to be called in the Java language and the Android kernel library.

[0211] The application layer and application framework layer run on a virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine is configured to implement functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0212] The system library may include multiple functional modules such as a surface manager, a media library, a 3D graphics processing library (e.g., OpenGL ES), and a 2D graphics engine (e.g., SGL).

[0213] The Surface Manager is configured to manage the display subsystem and provide a blend of 2D and 3D layers for multiple applications.

[0214] The media library supports playback and recording of multiple commonly used audio and video formats, as well as still image files. The media library may support multiple audio and video coding formats, including MPEG-4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0215] The 3D graphics processing library is configured to implement 3D graphics drawing, image rendering, compositing, layer processing, etc.

[0216] The 2D graphics engine is a drawing engine for 2D drawing.

[0217] The kernel layer is a layer between the hardware and the software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver. In some embodiments, the sensor driver may correspond to the detection module shown in FIG. 1B.

[0218] An example of software and hardware operation of the electronic device 100 will now be described with reference to a navigation scenario.

[0219] Assume that the display 194 displays the user interface of a chat application, and this user interface is used to display address information for location 1. When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including information such as the touch coordinates and the timestamp of the touch operation). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. For example, the touch operation is a touch-tap operation, and the control corresponding to the tap operation is a navigation control. The map application invokes an interface in the application framework layer to start the map application, and then invokes the kernel layer to start the display driver and display the navigation interface via the display 194. The destination of the navigation interface is location 1.

[0220] The software architecture of electronic device 200 is similar to the software architecture of electronic device 100. See Figure 2D for a specific example.

[0221] The interaction method in the embodiment of this application will be described below with reference to an application scenario.

[0222] Scenario 1: Electronic device 100 is a smartphone, and electronic device 200 is an on-board computer. When electronic device 100 receives a user operation (e.g., a shake operation) while displaying a user interface including address information, electronic device 100 may perform an intention operation, i.e., set the location indicated by the address information as a destination, perform navigation, and transmit audio and video streams (hereinafter referred to as audio and video streams for short) corresponding to the intention operation to electronic device 200 for output. In this way, the user does not need to manually enter address information into a map application on electronic device 200 or operate navigation controls. In other words, the user does not need to manually trigger the execution of the intention operation, which implements more efficient and convenient interaction.

[0223] 3A-1 through 3C show example embodiments of user interfaces in application scenarios (e.g., Scenario 1 above).

[0224] As shown in FIG. 3A-1 , the electronic device 100 may display a user interface 310 of a chat application. The user interface 310 may include a session name 311 and a chat window 312. Assume that the current session is a two-party session. The session name 311 may include, but is not limited to, the name of a chat participant, such as "Xiao Wang." If the current session is a multi-party session, the session name 311 may include a name of the current session, such as a group name. The chat window 312 may be configured to display a chat history of the current session, such as messages 3121 and 3122 sent by the chat participants. Message 3121 contains the text "Meet here", message 3122 contains the place name 3122A (containing the text "Beijing Railway Station") and the location information for "Beijing Railway Station" 3122B (containing the text "A13 Maojiawan Hutong, Dongcheng District, Beijing"), and message 3122 is a location card showing the geographic location "Beijing Railway Station".

[0225] 3A-1 , electronic device 100 may be connected to electronic device 200 using, for example, a HUAWEI HiCar feature. Electronic device 200 may display home screen 320. Home screen 320 may include one or more application icons, such as a map application icon, a phone application icon, a music application icon, a radio application icon, a dashboard camera application icon, and a settings application icon. Home screen 320 may further include a main menu control, which may be used to return to home screen 320.

[0226] As shown in FIG. 3A-1 , the electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and recognize the currently displayed user interface 310 in response to the user operation. In some embodiments, the electronic device 100 recognizes the message 3122 to obtain location information of the geographical location “Beijing Railway Station” and, based on the location information, determines intent information, i.e., to perform navigation to the geographical location “Beijing Railway Station.” Optionally, the electronic device 100 may alternatively refer to the message 3121 to determine that the user wants to go to the geographical location “Beijing Railway Station” and determine the intent information based on the user's intention. Optionally, the intent information may be understood to correspond to a navigation service, or the intent information may be understood to correspond to the message 3122 (location card). Based on the obtained intent information, the electronic device 100 may perform an intention operation corresponding to the intent information, where the intent operation is to set a destination to the location information of the geographical location “Beijing Railway Station” and perform navigation. The electronic device 100 may then transmit audio and video streams corresponding to the performed intention operation to the electronic device 200 for output. For details, see FIG. 3A-2. Optionally, the intention operation may be understood to be used to implement a navigation service, or the intention operation may be understood to correspond to the message 3122 (location card).

[0227] 3A-2, the electronic device 200 may display a map application user interface 330. The user interface 330 is used to display information related to navigation services. The user interface 330 may include a map window 331, a route window 332, and a prompt box 333.

[0228] Specifically, the map window 331 is used to display a schematic diagram of the selected navigation route on a map.

[0229] The route window 332 includes navigation information 332A, a route 332B, a route 332C, and navigation controls 332D. The navigation information 332A includes text indicating the location information of the navigation destination, such as "To A13 Maojiawan Hutong, Dongcheng...." The navigation information 332A indicates only a portion of the location information of the destination. The electronic device 200 may also display all of the location information of the destination in response to a touch operation (e.g., a tap operation) performed on the navigation information 332A. The route 332B and the route 332C may indicate two navigation routes. Compared to the route 332C, the route 332B is highlighted (e.g., the text of the route 332B is highlighted in bold, but the text of the route 332C is not), indicating that the currently selected navigation route is the navigation route indicated by the route 332B. In this case, the map window 331 is used to display a schematic diagram of the navigation route indicated by the route 332B on the map. In response to a touch operation (e.g., a tap operation) performed on the path 332C, the electronic device 200 may remove the highlighting from the path 332B and highlight the path 332C. In this case, the selected navigation path is the navigation path indicated by the path 332C, and the map window 331 displays a schematic diagram of the navigation path indicated by the path 332C on the map. The navigation control 332D may be configured to enable a navigation function. In response to a touch operation (e.g., a tap operation) performed on the navigation control 332D, the electronic device 200 may perform navigation based on the currently selected path (the navigation path indicated by the path 332B in the user interface 330).

[0230] The prompt box 333 is used to display information about the current navigation service. The prompt box 333 contains text such as "Navigating to A13 Maojiawan Hutong, Dongcheng District, Beijing in chat with Xiao Wang," which may indicate detailed location information of the destination of the navigation service. The navigation service is triggered by a chat session with the chat participant "Xiao Wang" in the chat application, and the detailed location information of the destination is obtained from the chat session.

[0231] In the example shown in Figures 3A-1 and 3A-2, the address information (i.e., message 3122) included in the user interface 310 displayed by the electronic device 100 is displayed in a card format. This is not limiting. In some other examples, the address information may alternatively be displayed in a text format. See Figure 3B for a specific example. This is not limiting in this application.

[0232] As shown in FIG. 3B , the electronic device 100 may display a user interface 340 of a chat application. The user interface 340 is similar to the user interface 310 shown in FIG. 3A-1 , except that the two user interfaces differ in the chat history of the current session. The user interface 340 may include a message 341 and a message 342. The message 341 includes the text “Where shall we meet,” and the message 342 includes the text “Meet at the Beijing Railway Station.” The electronic device 100 may be connected to the electronic device 200, which may display the home screen 320 shown in FIG. 3A-1 . The electronic device 100 may receive a user action (e.g., shaking the electronic device 100) and, in response to the user action, recognize the currently displayed user interface 340. In some embodiments, electronic device 100 recognizes message 342, acquires information indicating that the user wants to go to the geographic location "Beijing Railway Station," and, based on the acquired information, determines intent information, i.e., to perform navigation to the geographic location "Beijing Railway Station." Optionally, the intent information may be understood to correspond to message 342. Electronic device 100 may perform an intent operation corresponding to the intent information based on the acquired intent information, where the intent operation is to set a destination to the location information of the geographic location "Beijing Railway Station" and perform navigation. Next, electronic device 100 may transmit audio and video streams corresponding to the performed intent operation to electronic device 200 for output. For details, see FIG. 3A-2. Optionally, the intent operation may be understood to correspond to message 342.

[0233] In the foregoing examples, the address information displayed by electronic device 100 is displayed in the form of a chat message (i.e., message 3122 of user interface 340 or message 342 of user interface 310). This is not limiting. In some other examples, the address information may alternatively be displayed in a location description. See FIG. 3C for a specific example. This is not limiting in this application.

[0234] As shown in FIG. 3C , electronic device 100 may display a user interface 350 of an entertainment application. User interface 350 includes a place name 351 and a location control 352. Place name 351 includes text "Capital Museum," which is the name of the place displayed in user interface 350. Location control 352 includes text "16 Fuxingmen Outer Street, Xicheng District," which is location information of the place displayed in user interface 350, and may indicate address information for the place "Capital Museum." Electronic device 100 may be connected to electronic device 200, and electronic device 200 may display home screen 320 shown in FIG. 3A-1 . Electronic device 100 receives a user operation (e.g., shaking electronic device 100) and, in response to the user operation, recognizes the currently displayed user interface 350, obtains location information of the place named "Capital Museum," and determines, based on the location information, intent information, i.e., to navigate to the place "Capital Museum." Optionally, the intention information may be understood to correspond to the location control 352. The electronic device 100 may transmit instruction information to the electronic device 200 based on the acquired intention information, and the electronic device 200 may perform an intention operation corresponding to the intention information based on the instruction information, where the intention operation is to set a destination to the location information of the place "Capital Museum" and perform navigation. A specific example is similar to that of FIG. 3A-2 . There is a difference between the destination and the navigation route. Optionally, the intention operation may be understood to correspond to the location control 352.

[0235] In the embodiment shown in FIGS. 3A-1 to 3B, the electronic device 100 performs an intention operation corresponding to the intention information and then transmits audio and video streams corresponding to the intention operation to the electronic device 200 for output. It may be understood that content on the electronic device 100 is projected on the electronic device 200 for output, and a service on the electronic device 100 is actually triggered. In the embodiment shown in FIG. 3C, the electronic device 100 indicates to the electronic device 200 to perform an intention operation corresponding to the intention information, and a service on the electronic device 200 is actually triggered. This is not limited thereto. In certain embodiments, a service on the electronic device 200 may be alternatively triggered in the embodiments shown in FIGS. 3A-1 to 3B, and a service on the electronic device 100 may be alternatively triggered in the embodiment shown in FIG. 3C. In the following embodiment, an example in which a service on the electronic device 200 is triggered is used for explanation. However, this is not a limitation in a specific implementation.

[0236] In a possible implementation, the service type corresponding to the intent information determined by the electronic device 100 is related to the device type of the electronic device 200 connected to the electronic device 100 .

[0237] Scenario 2: When the electronic device 100 is a smartphone and displays a user interface including address information and video information, the electronic device 100 receives a user operation (e.g., a shake operation). If the electronic device 200 connected to the electronic device 100 is an on-board computer, the electronic device 100 may transmit instruction information to the electronic device 200 in response to the user operation. The electronic device 200 may perform an intended operation corresponding to a navigation service based on the instruction information. The intended operation is to set the location indicated by the address information as a destination and perform navigation. For specific examples, see FIGS. 4A-1 and 4A-2. If the electronic device 200 connected to the electronic device 100 is a smart TV, the electronic device 100 may transmit instruction information to the electronic device 200 in response to the user operation. The electronic device 200 may perform an intended operation corresponding to a video service based on the instruction information. The intended operation is to play the video indicated by the video information. For specific examples, see FIGS. 4B-1 and 4B-2. In this way, it can better meet user requirements in real application scenarios and further improve interaction accuracy.

[0238] 4A-1 to 4B-2 show example embodiments of user interfaces in application scenarios (e.g., scenario 2 described above).

[0239] As shown in FIG. 4A-1, the electronic device 100 may display a user interface 410 of a chat application. The user interface 410 is similar to the user interface 310 shown in FIG. 3A-1, and the two user interfaces differ in the chat history of the current session. The user interface 410 may include messages 411, 412, 413, and 414. The messages 411 and 412 correspond to the messages 3121 and 3122, respectively, of the user interface 310 shown in FIG. 3A-1. Details will not be described again here. The message 413 includes the text "Look at this," and the message 414 is a message for displaying videos in card format, including the text "My Day," which is the name of the video being displayed. The electronic device 100 may be connected to the electronic device 200 (on-board computer), which may display the home screen 320 shown in FIG. 3A-1. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and, in response to the user operation, recognize the currently displayed user interface 410. In some embodiments, the electronic device 100 recognizes the user interface 410 and obtains information indicating that the message 412 corresponds to a navigation service and the message 414 corresponds to a video service. The electronic device 100 may determine the corresponding navigation service based on the device type (on-board computer) of the connected device. For example, a correspondence between the on-board computer and the navigation service is preset. In this case, the electronic device 100 recognizes the message 412 and determines intent information corresponding to the navigation service, where the intent information indicates navigation to the geographic location "Beijing Railway Station."Based on the acquired intention information, the electronic device 100 may send instruction information to the electronic device 200 (on-board computer). The electronic device 200 (on-board computer) may then perform an intention operation corresponding to the intention information based on the instruction information, such as setting a destination to the location information of the geographic location "Beijing Railway Station" and performing navigation. For details, see FIG. 4A-2. The user interface displayed by the electronic device 200 in FIG. 4A-2 is consistent with the user interface displayed by the electronic device 200 in FIG. 3A-2.

[0240] As shown in FIG. 4B-1, electronic device 100 may display user interface 410 shown in FIG. 4A-1. Electronic device 100 may be connected to electronic device 200 (smart TV), which may display home screen 420. Home screen 420 may include one or more categories, such as a TV series category, a video category, an anime category, a children's category, and a game category. Electronic device 100 may receive a user operation (e.g., shaking electronic device 100) and, in response to the user operation, recognize the currently displayed user interface 410. In some embodiments, electronic device 100 recognizes user interface 410 and obtains information indicating that message 412 corresponds to a navigation service and message 414 corresponds to a video service. Electronic device 100 may determine the corresponding navigation service based on the device type (smart TV) of the connected device. For example, a correspondence between smart TVs and video services is preset. In this case, the electronic device 100 recognizes the message 414 and determines intent information corresponding to a video service, where the intent information indicates playing a video named "My Day." Based on the acquired intent information, the electronic device 100 may transmit instruction information to the electronic device 200 (smart TV). The electronic device 200 (smart TV) may then perform an intent operation corresponding to the intent information, where the intent operation is to play a video named "My Day." See FIG. 4B-2 for details. As shown in FIG. 4B-2, the electronic device 200 may display a user interface 430. The user interface 430 includes a title 431. The title 431 includes the text "My Day," which is the name of the currently playing video.

[0241] This is not limited to the above example. In some other example scenarios, the user may select the service information to be recognized, and the service type corresponding to the intent information is determined based on the user selection. For specific examples, see Figures 4C-1 and 4C-2.

[0242] As shown in Fig. 4C-1, the electronic device 100 may display the user interface 410 shown in Fig. 4A-1. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and, in response to the user operation, display the user interface 440 shown in Fig. 4C-2.

[0243] As shown in FIG. 4C-2, user interface 440 may include prompt information 441, prompt box 442, and prompt box 443. Prompt information 441 includes the text "Please select the service you need to transfer" which is used to prompt the user to select the service information to be recognized.

[0244] The prompt box 442 includes a service name 442A and service information 442B, where the service name 442A includes the text "Map Navigation," and the service information 442B is the message 412 in the user interface 410 shown in FIG. 4C-1. In response to a touch operation (e.g., a tap operation) performed on the prompt box 442, the electronic device 100 may determine that the recognized service information is the message 412 in the user interface 410, recognize the message 412, and obtain intent information corresponding to the navigation service, where the intent information indicates performing navigation to the geographic location "Beijing Railway Station." Based on the obtained intent information, the electronic device 100 may send instruction information to the connected electronic device 200, and the electronic device 200 may perform an intent operation corresponding to the intent information based on the instruction information. For example, the interfaces displayed before and after the electronic device 200 receives the instruction information refer to the user interface 320 shown in FIG. 4A-1 and the user interface 330 shown in FIG. 4A-2.

[0245] The prompt box 443 includes a service name 443A and service information 443B. The service name 443A includes the text "Play Video." The service information 443B is the message 414 in the user interface 410 shown in FIG. 4C-1. In response to a touch operation (e.g., a tap operation) performed on the prompt box 443, the electronic device 100 may determine that the recognized service information is the message 414 in the user interface 410 and recognize the message 414 to obtain intention information corresponding to the video service. The intention information indicates playing a video named "My Day." Based on the obtained intention information, the electronic device 100 may send instruction information to the connected electronic device 200. Based on the instruction information, the electronic device 200 may perform an intention operation corresponding to the intention information. For example, the interfaces displayed before and after the electronic device 200 receives the instruction information refer to the user interface 420 shown in FIG. 4B-1 and the user interface 430 shown in FIG. 4B-2.

[0246] Scenario 3: The electronic device 100 is a smartphone, and the electronic device 200 is a smart TV. When the electronic device 100 displays a user interface including video information, the electronic device 100 may transmit instruction information to the electronic device 200 upon receiving a user operation (e.g., a shake operation). The electronic device 200 may perform an intended operation, i.e., play the video indicated by the video information, based on the instruction information. In this way, the user does not need to manually trigger the execution of the intended operation, which allows for more efficient and convenient interaction.

[0247] 5A and 5B show example embodiments of a user interface in another application scenario (e.g., scenario 3 above).

[0248] As shown in FIG. 5A , electronic device 100 may display a user interface 510 of an entertainment application. User interface 510 includes a name 521. Name 521 includes the text "Movie 1," which is the name of the movie being displayed in user interface 510. User interface 510 is used to display details about "Movie 1," such as related videos, stills, and video reviews. Electronic device 100 may be connected to electronic device 200, which may display home screen 420 shown in FIG. 4B-1 . Home screen 420 further includes a search control 421. Search control 421 is configured to implement a search function. A user may input and play a desired video to be viewed based on the search function. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and, in response to the user operation, recognize the currently displayed user interface 510 to obtain information about the video named "Movie 1." Based on the information, the electronic device 100 may determine intention information, i.e., to play the video named "Movie 1." Based on the obtained intention information, the electronic device 100 may send instruction information to the electronic device 200. Based on the instruction information, the electronic device 200 may perform an intention operation corresponding to the intention information, where the intention operation is to play the video named "Movie 1." For details, see FIG. 5B.

[0249] As shown in FIG. 5B , electronic device 200 may display user interface 520. User interface 520 includes title 521. Title 521 includes the text "Movie 1," which is the name of the currently playing video. In some cases, electronic device 100 may retrieve a video stream for "Movie 1" from a video application and continuously transmit the video stream to electronic device 200 for playback. This may be understood as projecting the video on electronic device 100 to electronic device 200 for playback. In this way, the user does not need to start a video application and a video playback interface on electronic device 100 (smartphone), operate projection controls, or select a device (smart TV) to project the video to. In other cases, electronic device 200 searches for and plays the video after receiving instruction information. This may be understood as playing the video on electronic device 200. In this way, the user does not need to search for videos on the electronic device 200 (smart TV) (e.g., search for videos using the search control 421 in the user interface 420 shown in FIG. 5A), thus simplifying user operations and greatly improving interaction efficiency.

[0250] In the examples shown in Figures 5A and 5B, the video information displayed by the electronic device 100 is displayed in a video introduction, but is not limited thereto. In other examples, the video information may alternatively be displayed in the form of a chat message, such as message 414 of the user interface 410 shown in Figure 4B-1. See Figures 4B-1 and 4B-2 for specific example scenarios, which are not limiting in this application.

[0251] Scenario 4: Electronic device 100 is a smartphone, and electronic device 200 is a smart food processor. When electronic device 100 displays a user interface including recipe information, it may transmit instruction information to electronic device 200 upon receiving a user operation (e.g., a shake operation). Based on the instruction information, electronic device 200 may perform an intended operation, i.e., work based on the recipe information. In this way, the user does not need to search for a recipe on the smart food processor to cook. In other words, the user does not need to manually trigger the execution of the intended operation, which implements more efficient and convenient interaction.

[0252] 6A and 6B show example embodiments of a user interface in another application scenario (e.g., scenario 4 above).

[0253] As shown in FIG. 6A , electronic device 100 may display a user interface 610 of a home application. User interface 610 includes a title 611. Title 611 includes the text “Crispy Pork Belly,” which is the name of the recipe displayed in user interface 610. User interface 610 is used to display details about the recipe named “Crispy Pork Belly,” such as ingredient information 612 and cooking steps 613. Electronic device 100 may be connected to electronic device 200, and electronic device 200 may display a home page 620. Home page 620 may include one or more categories, such as a daily recipe category, a Chinese category, and a Western category. Electronic device 100 may receive a user operation (e.g., shaking electronic device 100) and, in response to the user operation, recognize the currently displayed user interface 610 to obtain information about the recipe named “Crispy Pork Belly.” Based on the information, electronic device 100 may determine intent information, i.e., to cook the dish corresponding to the recipe. The electronic device 100 may transmit instruction information to the electronic device 200 based on the acquired intention information, and the electronic device 200 may perform an intended operation corresponding to the intention information based on the instruction information, where the intended operation is to work based on a recipe. For details, see FIG. 6B.

[0254] As shown in FIG. 6B, electronic device 200 may display user interface 630. User interface 630 includes title 631 and step information 632. Title 631 includes the text "Crispy Pork Belly," which is the name of the currently used recipe. Step information 632 indicates a cooking step of the currently used recipe, which corresponds to cooking step 613 in user interface 610 shown in FIG. 6A. User interface 630 may indicate that electronic device 200 is currently working based on a recipe named "Crispy Pork Belly."

[0255] This is not limited to the above example. In some other examples, when recognizing the currently displayed user interface 610, the electronic device 100 may only recognize the name of the dish "crispy pork belly" on the recipe, and determine the intention information, i.e., to cook the area named "crispy pork belly," based on the dish name. After receiving the instruction information, the electronic device 200 may perform an intention operation corresponding to the intention information, where the intention operation is to search for the dish name to obtain the corresponding recipe and work based on the found recipe.

[0256] Scenario 5: Electronic device 100 is a smartphone used by a parent, and electronic device 200 is a tablet computer used by a child for learning. When electronic device 100 receives a user operation (e.g., a shake operation) while displaying a user interface containing learning information, it may send instruction information to electronic device 200. Based on the instruction information, electronic device 200 may perform an intended operation, i.e., display all or part of the learning information. In this way, the parent does not need to search the tablet computer for learning information for the child to use. In other words, the user does not need to manually trigger the execution of an intended operation, which implements more efficient and convenient interaction.

[0257] 7A-1 and 7A-2 show example embodiments of user interfaces in different application scenarios (e.g., scenario 5 above).

[0258] As shown in FIG. 7A-1 , the electronic device 100 may display a user interface 710 of a learning application. The user interface 710 includes a title 711. The title 711 includes text “English test paper,” indicating that the user interface 710 is used to display details about a test paper named “English test paper.” The user interface 710 further includes details about multiple practice questions, such as practice question 712 and practice question 713. The practice question 712 includes question 712A and answer 712B, and the practice question 713 includes question 713A and answer 713B. The user interface 710 further includes a test control 714. The test control 714 is configured to provide a function for taking a practice test on the current test paper. The electronic device 100 may be connected to the electronic device 200, and the electronic device 200 may display a home screen 720. The home screen 720 may include one or more application icons, such as a clock application icon, a calendar application icon, a gallery application icon, and a settings application icon. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and, in response to the user operation, recognize the currently displayed user interface 720 to acquire information about a test paper named "English test paper." Based on the information, the electronic device 100 may determine intent information, i.e., to take a practice test on the test paper. Based on the acquired intent information, the electronic device 100 may send instruction information to the electronic device 200. Based on the instruction information, the electronic device 200 may perform an intent operation corresponding to the intent information. The intent operation is to enable a function for taking a practice test on the test paper. For details, see FIG. 7A-2.

[0259] As shown in FIG. 7A-2 , the electronic device 200 may display a user interface 730. The user interface 730 includes a title 731, a submission control 732, question information 733, and a switching control 734. The title 731 includes the text "English test paper," which is the name of the test paper for the currently running practice test. The submission control 732 is configured to end the current practice test and display the results of the practice test. The question information 733 displays information about the currently viewed question, and the switching control 734 is configured to switch information about the currently viewed question. The user interface 730 may indicate that the ability to take a practice test for the test paper named "English test paper" is currently enabled.

[0260] In some embodiments, the electronic device 200 responds to a touch operation (e.g., a tap operation) made on the submission control 732 by displaying the results of the practice test and transmitting the results of the practice test to the electronic device 100, thereby allowing parents to efficiently and conveniently learn about their child's learning progress.

[0261] 7A-1 and 7A-2, the user interface 710 displayed by the electronic device 100 includes questions and answers, while the user interface 730 displayed after the electronic device 200 receives instruction information includes only questions and no answers. In this way, the parent does not need to search for corresponding practice questions on the electronic device 200 or manually enable the function of taking a practice test. This further reduces user operations and improves interaction efficiency.

[0262] This is not limited to the examples shown in FIG. 7A-1 and FIG. 7A-2. In some other examples, the electronic device 100 may recognize the exercises 712 and / or 713 in the currently displayed user interface 710 in response to a shaking operation of the electronic device 100, and determine, based on the exercises 712 and / or 713, the intention information, i.e., to practice the exercises 712 and / or 713. In this case, the electronic device 200 may perform the corresponding intention operation, i.e., displaying the questions 712A of the exercises 712 and / or the questions 713A of the exercises 713 for the child to use for practice, after receiving the instruction information transmitted by the electronic device 100 based on the intention information. A specific example is similar to that of the user interface 730 shown in FIG. 7A-2.

[0263] This is not limited to the above example. In some other example scenarios, the user may select the service information to be recognized, and the service content corresponding to the intent information is determined based on the user selection. For specific examples, see Figures 7B-1 and 7B-2.

[0264] As shown in Fig. 7B-1, the electronic device 100 may display the user interface 710 shown in Fig. 7A-1. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and, in response to the user operation, display the user interface 740 shown in Fig. 7B-2.

[0265] 7B-2, user interface 740 may include prompt information 741, prompt box 742, and prompt box 743. Prompt information 741 includes the text "Please select the content you need to transfer" used to prompt the user to select the service information to be recognized.

[0266] The prompt box 742 is question 712A of the exercise 712 in the user interface 740 shown in FIG. 7B-1 . Furthermore, in response to a touch operation (e.g., a tap operation) performed on the prompt box 742, the electronic device 100 may determine that the recognized service information is the exercise 712 in the user interface 740, recognize the exercise 712, and obtain intent information, i.e., practicing the exercise 712. In this case, the electronic device 200 may perform a corresponding intent operation, i.e., display question 712A of the exercise 712, after receiving instruction information transmitted by the electronic device 100 based on the intent information. A specific example is similar to that of the user interface 730 shown in FIG. 7A-2 .

[0267] The prompt box 743 is question 713A of the exercise 713 in the user interface 740 shown in FIG. 7B-1. In response to a touch operation (e.g., a tap operation) performed on the prompt box 743, the electronic device 100 may determine that the recognized service information is the exercise 713 in the user interface 740, recognize the exercise 713, and obtain intent information, i.e., to practice the exercise 713. In this case, the electronic device 200 may perform a corresponding intent operation, i.e., display question 713A of the exercise 713, after receiving instruction information transmitted by the electronic device 100 based on the intent information. A specific example is similar to that of the user interface 730 shown in FIG. 7A-2.

[0268] In the above scenario 5, the electronic device 200 may alternatively be a learning machine-like device.

[0269] In the above example, the user operation that triggers the intention transfer (referred to as a trigger operation for short) is a shake operation. In some other examples, the trigger operation may alternatively be a knuckle slide operation. See (A) of FIG. 8 for a specific example. In some other examples, the trigger operation may alternatively be a two-finger slide operation. See (B) of FIG. 8 for a specific example. In some other examples, the trigger operation may alternatively be a gesture operation. See (C) of FIG. 8 for a specific example. This is not a limitation. Alternatively, the trigger operation may be a knuckle tap operation, a hand swing operation, or another operation. The specific type of the trigger operation is not limited in this application.

[0270] The display method provided in the embodiment of this application will be described below.

[0271] FIG. 9 shows an example of a schematic flowchart of a display method according to an embodiment of the present application.

[0272] This display method may be applied to the above-mentioned communication system 10. The communication system 10 may include an electronic device 100, an electronic device 200, and a network device 300.

[0273] As shown in FIG. 9, the method includes, but is not limited to, the following steps.

[0274] S101: The electronic device 100 establishes a connection to the electronic device 200.

[0275] In some embodiments, electronic device 100 may be directly connected to electronic device 200 in a wired and / or wireless manner, for example, using Bluetooth or Wi-Fi. In some other embodiments, electronic device 100 may be connected to electronic device 200 via network device 300. For more information, see the description of the connection between electronic device 100 and electronic device 200 in FIG. 1A.

[0276] S102: The electronic device 100 displays a first interface including first service information.

[0277] In some embodiments, the first service information corresponds to a first service and the different service information corresponds to a different service. Specific examples are described below.

[0278] For example, the first service information may be address information corresponding to a navigation service, such as message 3122 in user interface 310 shown in FIG. 3A-1, message 342 in user interface 310 shown in FIG. 3B, or location control 352 in user interface 340 shown in FIG. 3C.

[0279] For example, the first service information is video information corresponding to a video service (e.g., playing a video). The first service information may be the message 414 in the user interface 410 shown in FIG. 4A-1 or the information (e.g., name 521) included in the user interface 510 shown in FIG. 5A.

[0280] For example, the first service information is recipe information corresponding to a cooking service (e.g., cooking based on a recipe). Information included in the user interface 610 shown in FIG. 6A (e.g., title 611) is the first service information.

[0281] For example, the first service information is learning information corresponding to a learning service (e.g., practice questions). Information included in the user interface 710 shown in FIG. 7A-1 (e.g., practice 712, practice 713) is the first service information.

[0282] S103: The electronic device 100 receives a first user operation.

[0283] The first user operation type includes, but is not limited to, a touch operation performed on a display, a voice, a movement or posture (e.g., a gesture), and an electroencephalogram. For example, the first user operation is an operation of shaking the electronic device 100. In another example, the first user operation is a sliding operation with a knuckle of a finger as shown in (A) of FIG. 8. In another example, the first user operation is a sliding operation with two fingers as shown in (B) of FIG. 8. In another example, the first user operation is a gesture operation as shown in (C) of FIG. 8. The specific type of the first user operation is not limited in this application.

[0284] In some embodiments, the electronic device 100 may detect the first user action via a detection module shown in FIG. 1B.

[0285] In some embodiments, the electronic device 100 may detect a first user action via the sensor module 180 shown in Figure 2A. For a specific example, see the description of how the electronic device 100 may detect a user action via the sensor module 180 in Figure 2A.

[0286] In some embodiments, electronic device 100 may train a fusion model that is used to recognize user intent, such as to perform S107.

[0287] In some other embodiments, network device 300 trains a fusion model. For a description of training a fusion model, see the description of training a fusion model in FIG. 1B and training an interface analysis model and / or an intent analysis model. The details will not be described again here.

[0288] If the electronic device 100 receives the fusion model transmitted by the network device 300 before S103, the display method may include, but is not limited to, the following three steps after S103:

[0289] S104: The electronic device 100 sends a first request message to the network device 300.

[0290] In some embodiments, the first request message is used to request obtaining setting information of the fusion model.

[0291] S105: The network device 300 sends a first configuration message to the electronic device 100.

[0292] In some embodiments, the first configuration message includes configuration information for the fusion model.

[0293] S106: The electronic device 100 updates the fusion model based on the first configuration message.

[0294] In another case where the electronic device 100 does not receive the fusion model transmitted by the network device 300 before the process of S103, the electronic device 100 may request to obtain the fusion model from the network device. The specific process is the same as the above-described steps S104 to S106. The details will not be described again here.

[0295] S107: The electronic device 100 recognizes the first interface based on the fusion model and determines intention information corresponding to the first service information.

[0296] In some embodiments, the electronic device 100 may use the interface content in the first interface as input for the fusion model to obtain an output, i.e., intent information. Some examples of intent information are provided below.

[0297] For example, the first interface is the user interface 310 shown in FIG. 3A-1 or the user interface 340 shown in FIG. 3B, and the message 3122 in the user interface 310 or the message 342 in the user interface 340 is the first service information, which is address information indicating a geographical location named "Beijing Railway Station." In this case, the intention information corresponding to the first service information is to navigate to the geographical location named "Beijing Railway Station."

[0298] For example, the first interface is user interface 350 shown in Figure 3C, and location control 352 in user interface 350 is first service information, which is address information indicating a place named "Capital Museum." In this case, the intent information corresponding to the first service information is to navigate to the place named "Capital Museum."

[0299] For example, the first interface may be the user interface 410 shown in FIG. 4A-1, and the message 414 in the user interface 410 may be first service information, which may indicate a video named "My Day." In this case, the intent information corresponding to the first service information may be to play the video named "My Day."

[0300] For example, the first interface may be user interface 510 shown in FIG. 5A, and information included in user interface 510 (e.g., name 521) may be first service information, which indicates a video named "Movie 1." In this case, the intention information corresponding to the first service information is to play the video named "Movie 1."

[0301] For example, the first interface may be the user interface 610 shown in FIG. 6A , and information included in the user interface 610 (e.g., title 611) may be first service information, which may indicate a recipe named “Crispy Pork Belly.” In this case, the intention information corresponding to the first service information is to cook the dish corresponding to the recipe.

[0302] For example, the first interface may be the user interface 710 shown in FIG. 7A-1, and information included in the user interface 710 (e.g., exercises 712 and 713) may be first service information, which may indicate one or more exercises (e.g., at least one exercise included in a test paper named "English test paper," e.g., exercises 712 and 713). In this case, the intention information corresponding to the first service information is to practice one or more exercises.

[0303] S108: The electronic device 100 transmits instruction information to the electronic device 200 based on the intention information.

[0304] In some embodiments, electronic device 100 may perform an intention operation based on the intention information and send multimedia data corresponding to the performed intention operation to electronic device 200. The instruction information may instruct electronic device 200 to output the multimedia data. For example, in FIG. 1B , the intention analysis module of electronic device 100 sends the intention information to the intention trigger module, which performs the intention operation based on the intention information and sends audio and video streams corresponding to the intention operation to a display module of electronic device 200 for output.

[0305] In some other embodiments, the instructional information sent by electronic device 100 to electronic device 200 may include intent information, and the instructional information may indicate to electronic device 200 to implement the intent information. For example, in FIG. 1C , the intent analysis module of electronic device 100 sends the intent information to the intent trigger module of electronic device 200.

[0306] S109: The electronic device 200 outputs the multimedia data.

[0307] In some embodiments, when electronic device 200 receives the multimedia data and instructional information transmitted by electronic device 100, it may output the multimedia data based on the instructional information, such as in the embodiment shown in FIG. 1B.

[0308] In some other embodiments, upon receiving the instruction information transmitted by electronic device 100, if the instruction information includes intention information, electronic device 200 may perform an intention operation based on the intention information and output multimedia data corresponding to the performed intention operation, such as the embodiment shown in FIG. 1C . In some embodiments, the intention operation corresponds to first service information in the first interface. In some embodiments, electronic device 200 outputting multimedia data corresponding to the performed intention operation may also be referred to as electronic device 200 outputting multimedia data corresponding to the first service information.

[0309] Below are some examples of intentional operations.

[0310] For example, the first interface may be the user interface 310 shown in FIGS. 3A-1 and 3A-2 or the user interface 340 shown in FIG. 3B. The message 3122 in the user interface 310 or the message 342 in the user interface 340 is first service information, which is address information indicating a geographical location named "Beijing Railway Station." In this case, the intended operation corresponding to the first service information is to set the destination to the location information of the geographical location "Beijing Railway Station" and perform navigation. For multimedia data corresponding to the intended operation and output by the electronic device 200, see the multimedia data of the user interface 330 shown in FIG. 3A-2. For a description of a specific scenario, see the description of FIG. 3A-1 and 3A-2 or FIG. 3B.

[0311] For example, the first interface is the user interface 350 shown in FIG. 3C , and the location control 352 in the user interface 350 is first service information, which is address information indicating a place named "Capital Museum." In this case, the intended operation corresponding to the first service information is to set a destination to the location information of the place "Capital Museum" and perform navigation. The multimedia data output by the electronic device 200 corresponding to the intended operation is similar to the user interface 330 shown in FIG. 3A-2 , except for the navigation destination. For a description of the specific scenario, see the description of FIG. 3C .

[0312] For example, the first interface may be the user interface 410 shown in FIG. 4A-1, and the message 414 in the user interface 410 may be first service information, which may indicate a video titled "My Day." In this case, the intended operation corresponding to the first service information is to play the video titled "My Day." For multimedia data corresponding to the intended operation and output by the electronic device 200, see the multimedia data of the user interface 430 shown in FIG. 4B-2. For a description of a specific scenario, see the descriptions of FIGS. 4B-1 and 4B-2.

[0313] For example, the first interface may be the user interface 510 shown in FIG. 5A , and the information (e.g., name 521) included in the user interface 510 may be first service information, which may indicate a video named "Movie 1." In this case, the intended operation corresponding to the first service information is to play the video named "Movie 1." For multimedia data corresponding to the intended operation and output by the electronic device 200, see the multimedia data of the user interface 520 shown in FIG. 5B . For a description of a specific scenario, see the description of FIGS. 5A and 5B .

[0314] For example, the first interface may be the user interface 610 shown in FIG. 6A , and the information (e.g., title 611) included in the user interface 610 may be first service information, which may indicate a recipe named "Crispy Pork Belly." In this case, the intended operation corresponding to the first service information is to work based on the recipe. For multimedia data corresponding to the intended operation and output by the electronic device 200, see the multimedia data of the user interface 630 shown in FIG. 6B . For a description of a specific scenario, see the description of FIGS. 6A and 6B .

[0315] For example, the first interface may be the user interface 710 shown in FIG. 7A-1, and the information included in the user interface 710 (e.g., exercises 712 and 713) may be first service information, which may indicate one or more exercises (e.g., at least one exercise included in a test paper named "English test paper," e.g., exercises 712 and 713). In this case, the intended operation corresponding to the first service information is to display questions (without displaying answers) in one or more exercises. For multimedia data corresponding to the intended operation and output by the electronic device 200, see the multimedia data of the user interface 730 shown in FIG. 7A-2. For a description of a specific scenario, see the descriptions of FIGS. 7A-1 and 7A-2.

[0316] This is not limited to the above example. In another embodiment, if the first interface does not include the first service information, the electronic device 100 does not recognize the intention information corresponding to the first service information, so it does not send instruction information to the electronic device 200, and the electronic device 200 does not perform the intention operation corresponding to the first service information. For example, the electronic device 100 and the electronic device 200 may continue to display the current interface without changing it. This is not limited to this example. Alternatively, the electronic device 100 may display prompt information indicating that there are currently no services available for transfer. For example, the user interface 410 (first interface) shown in FIG. 4A-1 includes only message 411 and message 413, but does not include message 412 (address information) and message 414 (video information). In this case, the electronic device 100 and the electronic device 200 may continue to display the current interface without changing it.

[0317] For examples of the display method shown in FIG. 9, see FIGS. 3A-1 to 3C, 4A-1 to 4C-2, 5A, 5B, 6A, 6B, and 7A-1 to 7B-2.

[0318] 9, when the electronic device 100 receives a first user operation, the electronic device 100 may perform intent recognition based on the currently displayed user interface, and the electronic device 200 implements the recognized intent. In this way, the user does not need to manually trigger the implementation of the intent. This reduces user operations and implements more efficient and convenient interactions.

[0319] This is not limited to the example of FIG. 9 . In other embodiments, in S107, the electronic device 100 may recognize a first interface to obtain an interface recognition result. Optionally, the electronic device 100 may obtain the interface recognition result based on an interface analysis model. For details, see the description of the interface analysis module in FIG. 1B . Optionally, the manner in which the electronic device 100 obtains the interface analysis model is similar to the manner in which the fusion model is obtained as shown in FIG. 9 . The electronic device 100 may perform intention recognition based on the interface recognition result to obtain intention information. Optionally, the electronic device 100 may obtain the intention information based on the intention analysis model. For details, see the description of the intention analysis module in FIG. 1B . Optionally, the manner in which the electronic device 100 obtains the intention analysis model is similar to the manner in which the fusion model is obtained as shown in FIG. 9 .

[0320] This is not limited to the example of FIG. 9 . In some other embodiments, the electronic device 100 may not perform S107 and S108. After receiving a first user operation, the electronic device 100 may recognize a first interface to obtain an interface recognition result, and send the interface recognition result and instruction information to the electronic device 200. The instruction information may instruct the electronic device 200 to implement intention information corresponding to the interface recognition result. The electronic device 200 may perform intention recognition based on the interface recognition result, obtain the intention information, perform an intention operation based on the intention information, and output multimedia data corresponding to the performed intention operation. For example, the electronic device 100 includes the detection module and the interface analysis module shown in FIG. 1B , and the electronic device 200 includes the intention analysis module and the intention trigger module shown in FIG. 1B . Optionally, the electronic device 200 may obtain the intention information based on an intention analysis model. Optionally, the manner in which the electronic device 200 obtains the intention analysis model is similar to the manner in which the electronic device 100 obtains the fusion model shown in FIG. 9 .

[0321] This is not limited to the example of FIG. 9 . In some other embodiments, the electronic device 100 may not perform S107 and S108. After receiving the first user operation, the electronic device 100 may send interface content displayed by the electronic device 100 and instruction information to the electronic device 200. The instruction information may instruct the electronic device 200 to implement intention information corresponding to the interface content. The electronic device 200 may perform S107 of FIG. 9 to obtain the intention information, perform an intention operation based on the intention information, and output multimedia data corresponding to the performed intention operation. For example, the electronic device 100 includes the detection module shown in FIG. 1B , and the electronic device 200 includes the interface analysis module, the intention analysis module, and the intention trigger module shown in FIG. 1B . Optionally, the electronic device 200 may obtain interface recognition results based on an interface analysis model. Optionally, the electronic device 200 may obtain intention information based on the intention analysis model. Optionally, the manner in which the electronic device 200 obtains the interface analysis model and / or the intention analysis model is similar to the manner in which the electronic device 100 obtains the fusion model shown in Fig. 9. Optionally, the electronic device 200 may obtain intention information based on the fusion model and the interface content. Optionally, the manner in which the electronic device 200 obtains the fusion model is similar to the manner in which the electronic device 100 obtains the fusion model shown in Fig. 9.

[0322] 10 is a schematic flowchart of another display method of the present application. The first device in this method may be the aforementioned electronic device 100, and the second device in this method may be the aforementioned electronic device 200. The method may include, but is not limited to, the following steps:

[0323] S201: The electronic device displays a first interface.

[0324] In some embodiments, the first interface includes first information, and the first information is related to the first service. For examples of the first information, see the example of first service information in S102 of FIG. 9.

[0325] S202: The first device receives a first user operation.

[0326] In some embodiments, S202 is similar to S103 in Figure 9. For details, see the description of S103 in Figure 9.

[0327] S203: In response to a first user operation, the first device recognizes a first interface and determines intention information.

[0328] In some embodiments, the intent information indicates to execute a first instruction, the first instruction being used to implement a first service.

[0329] In some other embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0330] In some embodiments, the intention information includes first information. For example, the first information is information indicating a first location, and the intention information indicates navigation to the first location. In some embodiments, the intention information includes information related to the first information. For example, the first information is information indicating a first video. A manner of playing the first video (e.g., a playback source of the first video) may be obtained based on the first information, and the intention information indicates playing the first video in the obtained manner of playing the first video.

[0331] In some embodiments, see the description of S107 in FIG. 9 for a description of recognizing the first information by the first device to determine the intent information.

[0332] S204: The first device sends the intention information to the second device.

[0333] S205: The second device executes the first instruction based on the intention information to generate second information.

[0334] In some embodiments, executing the first instruction by the second device may correspond to performing the intention operation described above. For examples of intention operations, see the intention operations illustrated in FIG. 9.

[0335] In some embodiments, the second information is multimedia data, such as audio data, video data, or image data, generated by executing the first instructions.

[0336] S206: The second device displays a second interface based on the second information.

[0337] In some embodiments, the second device may output the second information, for example, play audio data included in the second information, display image data included in the second information, or play video data included in the second information.

[0338] In some embodiments, for an example of the second device displaying the second interface, see the example of the electronic device 200 outputting multimedia data corresponding to the intentional operation in the description of the intentional operation shown in FIG. 9.

[0339] In some embodiments, the first information is information indicating a first location. For example, the first information is message 3122 in user interface 310 shown in FIG. 3A-1. In this case, the first location indicated by message 3122 is the geographic location "Beijing Railway Station." In another example, the first information is message 342 in user interface 340 shown in FIG. 3B or location control 352 in user interface 350 shown in FIG. 3C. In this case, the first location indicated by location control 352 is a place called "Capital Museum." The first service is a navigation service. The second information is display information generated by performing a navigation operation for the first location. For example, the second information is multimedia data generated by performing navigation with a destination set to the location information of the geographic location "Beijing Railway Station." In this case, the second interface displayed by the second device based on the second information is user interface 330 shown in FIG. 3A-2. In another example, the second information is multimedia data generated by navigating with a destination set to location information for a location such as "Capital Museum." For a description of a specific scenario, see the description of Figures 3A-1 and 3A-2, 3B, or 3C.

[0340] In some other embodiments, the first information is information indicating a first video. For example, the first information is message 414 in user interface 410 shown in FIG. 4A-1. In this case, the name of the first video indicated by message 414 is "My Day." In another example, the first information is information (e.g., name 521) included in user interface 510 shown in FIG. 5A. In this case, the name of the first video indicated by the information is "Movie 1." The first service is a navigation service. The second information is display information generated by playing the first video. For example, the second information is multimedia data generated by playing the video "My Day." In this case, the second interface displayed by the second device based on the second information is user interface 430 shown in FIG. 4B-2. For example, the second information is multimedia data generated by playing the video "Movie 1." In this case, the second interface displayed by the second device based on the second information is user interface 520 shown in Figure 5B. For a description of the specific scenario, see the description of Figures 4B-1 and 4B-2 or 5A and 5B.

[0341] In some other embodiments, the first information is information indicative of a first recipe, such as information (e.g., title 611) included in user interface 610 shown in FIG. 6A. In this case, the name of the first recipe indicated by the information is "Crispy Pork Belly." The first service is a cooking service. The second information is display information generated by implementing the cooking service corresponding to the first recipe, such as multimedia data generated by working on the recipe "Crispy Pork Belly." In this case, the second interface displayed by the second device based on the second information is user interface 630 shown in FIG. 6B. See the description of FIGS. 6A and 6B for a description of the specific scenario.

[0342] In some other embodiments, the first information is information indicating a first question and an answer to the first question, such as exercise 712 in user interface 710 shown in FIG. 7A-1, where exercise 712 includes question 712A and answer 712B. In this case, the first service is a test paper generation service. This application will be described using an example in which the test paper includes at least one question but does not include an answer. The second interface includes the first question but does not include an answer to the first question. For example, the second interface is user interface 730 shown in FIG. 7A-2. User interface 730 includes question 712A (question information 733 in user interface 730) but does not include answer 712B. For a description of a specific scenario, see the descriptions of FIGS. 7A-1 and 7A-2.

[0343] In some embodiments, the first interface further includes third information, and the third information is related to a second service. The third information and a description of the second service are similar to the first information and the description of the first service. Specifically, in S203, the first device recognizes the first information to determine fourth information, recognizes the third information to determine fifth information, and determines that the intention information is the fourth information from the fourth information and the fifth information according to a first preset rule. The fourth information indicates that a first instruction is to be executed, and the fifth information indicates that a second instruction is to be executed, and the second instruction is used to implement the second service. The description of the second instruction is similar to the description of the first instruction.

[0344] Optionally, the first preset rule may include that the device type of the second device is the preset device type. This may be understood as meaning that the first device may determine the intent information to implement based on the device type of the connected second device. For example, in the above-described Scenario 2, the first interface is a chat interface, the first information and the third information are messages 412 and 414, respectively, in the user interface 410 shown in FIG. 4A-1 , the first information is location information, and the third information is video information. In this case, the first service corresponding to the first information is a navigation service, the fourth information indicates navigation to the geographic location "Beijing Railway Station," the second service corresponding to the third information is a video playback service, and the fifth information indicates playing a video titled "My Day." The first device is the electronic device 100 (smartphone), and the second device is the electronic device 200. If the second device is an on-board computer, the first device may determine that the intent information is the fourth information. For example scenarios, see Figures 4A-1 and 4A-2. If the second device is a smart TV, the first device may determine that the intent information is the fifth information. For example scenarios, see Figures 4B-1 and 4B-2.

[0345] Optionally, the first preset rule may include that the services supported by the second device include a first service. For example, the first service is a navigation service. If the second device is a device on which a map application is installed and capable of performing a navigation service based on the map application, the first device may determine that the intent information is the first information.

[0346] Optionally, the first preset rule is that the priority of the first service is higher than the priority of the second service.

[0347] Optionally, the first information and the third information are instant messaging messages, and the first preset rule includes that a receiving time of the first information is later than a receiving time of the third information. For example, in the above-described scenario 2, the first interface is a chat interface, and the first information and the third information are message 412 and message 414, respectively, in user interface 410 shown in FIG. 4A-1. Because message 414 is received later, the first device may determine that the intent information is fifth information corresponding to message 414, and that the fifth information indicates to play a video named "My Day." See FIGS. 4B-1 and 4B-2 for example scenarios.

[0348] The method shown in Figure 10 is applied, for example, to the communication system 10 shown in Figure 1C, where the first device is the electronic device 100 and the second device is the electronic device 200. For details, see the description of Figure 1C.

[0349] 11 is a schematic flowchart of yet another display method according to an embodiment of the present application. The first device in this method may be the aforementioned electronic device 100, and the second device in this method may be the aforementioned electronic device 200. The method may include, but is not limited to, the following steps:

[0350] S301: The electronic device displays a first interface.

[0351] S302: The first device receives a first user operation.

[0352] S303: In response to a first user operation, the first device recognizes a first interface and determines intention information.

[0353] S301 to S303 correspond to S201 to S203 in Fig. 10. For details, see the description of S201 to S203 in Fig. 10.

[0354] S304: The first device executes a first instruction based on the intention information to generate second information.

[0355] In some embodiments, S304 is similar to S205 in Figure 10. The difference is that S304 is executed by the first device instead of the second device.

[0356] S305: The first device transmits the second information to the second device.

[0357] S306: The second device displays a second interface based on the second information.

[0358] S306 corresponds to S206 in Figure 10. For details, see the description of S206 in Figure 10.

[0359] The example of Figure 11 is similar to the example of Figure 10, except that in Figure 11, the device that executes the first instruction to generate the second information is the first device, rather than the second device. For details, see the example of Figure 10.

[0360] The method shown in Figure 11 is applied, for example, to the communication system 10 shown in Figure 1B, where the first device is the electronic device 100 and the second device is the electronic device 200. For details, see the description of Figure 1B.

[0361] This is not limited to the cases of FIGS. 10 and 11. In some other embodiments, the device that recognizes the first interface and determines the intention information may be a second device instead of the first device. For example, in response to a first user operation, the first device transmits multimedia data (e.g., image data) related to the first interface to the second device. The second device performs intention recognition based on the received data. The specific processing is similar to the above-described processing in which the first device recognizes the first interface and determines the intention information. The details will not be described again here.

[0362] When any of the above modules or units is implemented in software, the software exists in the form of computer program instructions and is stored in a memory. A processor may be configured to execute the program instructions to implement the above method steps. A processor may be a computing device that runs software, i.e., at least one of various computing devices that run software, such as, but not limited to, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor. Each computing device may include one or more cores used to execute software instructions to perform operations or processes. A processor may be an independent semiconductor chip or may be integrated with other circuits on a semiconductor chip. For example, a processor may form a system-on-a-chip (SoC) with other circuits (e.g., codec circuits, hardware acceleration circuits, or various bus and interface circuits). Alternatively, a processor may be integrated into an ASIC as an embedded processor of the ASIC. An ASIC integrated with a processor may be packaged separately or with other circuits. In addition to a core for executing software instructions to perform an operation or process, a processor may further include necessary hardware accelerators, such as a field programmable gate array (FPGA), a programmable logic device (PLD), or logic circuitry for implementing dedicated logic operations.

[0363] When the above modules or units are implemented by hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, dedicated digital circuit, hardware accelerator, or non-integrated discrete device. The hardware may run necessary software or may perform the steps of the above method without software.

[0364] Those skilled in the art may understand that all or part of the steps of the methods in the above embodiments may be implemented by a computer program that instructs relevant hardware. The computer program may be stored in a computer-readable storage medium. When the computer program is executed, the steps in the above method embodiments are performed. The above storage medium includes any medium that can store computer program code, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. 1. A display method applied to a first device, the first device being connected to a second device, displaying a first interface, the first interface including address information; Receiving a first user operation; identifying the address information and determining intent information in response to the first user operation, the intent information indicating that a location represented by the address information is a destination for navigation; and sending the intent information to the second device, wherein the intent information is used by the second device to display a second interface, and the second interface is used to provide a navigation service to a user, the destination of which is the location represented by the address information.

2. The method of claim 1 , wherein the first user operation includes performing a shake operation, a knuckle tap operation, a knuckle slide operation, a multi-finger tap operation, or a multi-finger slide operation on the first device.

3. The method of claim 1 or 2, wherein the first device is a mobile phone and the second device is a car.

4. The method of claim 1 or 2, wherein the first interface is a user interface of a chat application.

5. The method of claim 1 or 2, wherein the address information contained in the first interface is displayed by the first device in card format or text format.

6. 1. A display method applied to a first device, the first device being connected to a second device, displaying a first interface, the first interface including first information, the first information relating to a first service; Receiving a first user operation; recognizing the first interface and determining intent information in response to the first user operation, the intent information indicating execution of a first instruction, the first instruction being used to implement the first service; and sending the intent information to the second device, wherein the intent information is used by the second device to execute the first instructions to generate second information, and the second information is used by the second device to display a second interface.

7. the first interface further includes third information, the third information relating to a second service; Recognizing the first interface to determine intent information includes: determining fourth information from recognizing the first information and determining fifth information from recognizing the third information, wherein the fourth information indicates executing the first instructions and the fifth information indicates executing second instructions, the second instructions being used to implement the second service; 7. The method of claim 6, wherein determining that the intent information is the fourth information from the fourth information and the fifth information according to a first preset rule, the first preset rule including at least one of: a device type of the second device being a preset device type; services supported by the second device including the first service; and a priority of the first service being higher than a priority of the second service.

8. 8. The method of claim 7, wherein the first information is location information, the first service is a navigation service, the second service is different from the first service, the first preset rule includes that the device type of the second device is the preset device type, and the preset device type is an in-vehicle device.

9. The method according to any one of claims 6 to 8, wherein the first information is information indicating a first location, the first service is a navigation service, and the second information is display information generated by performing a navigation operation on the first location.

10. 8. The method of claim 6 or 7, wherein the first information is information indicating a first video, the first service is a video playback service, and the second information is display information generated by playing the first video.

11. 8. The method according to claim 6 or 7, wherein the first information is information indicating a first recipe, the first service is a cooking service, and the second information is display information generated to implement the cooking service corresponding to the first recipe.

12. 8. The method according to claim 6 or 7, wherein the first information is information indicating a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but does not include the answer to the first question.

13. 1. A display method applied to a first device, the first device being connected to a second device, displaying a first interface, the first interface including first information, the first information relating to a first service; Receiving a first user operation; recognizing the first interface and determining intention information in response to the first user operation; executing first instructions based on the intent information to generate second information, the first instructions being used to implement the first service; and transmitting the second information to the second device, the second information being used by the second device to display a second interface.

14. the first interface further includes third information, the third information relating to a second service; Recognizing the first interface to determine intent information includes: determining fourth information from recognizing the first information and determining fifth information from recognizing the third information, wherein the fourth information indicates executing the first instructions and the fifth information indicates executing second instructions, the second instructions being used to implement a second service; 14. The method of claim 13, comprising: determining, from the fourth information and the fifth information, that the intention information is the fourth information according to a first preset rule, wherein the first preset rule comprises: a device type of the second device being a preset device type; and / or a priority of the first service being higher than a priority of the second service.

15. A display method applied to a second device, the second device being connected to a first device, receiving intent information transmitted by the first device, the intent information being determined by recognizing a first interface displayed when the first device receives a first user operation, the first interface including first information, the first information relating to a first service; executing first instructions based on the intent information to generate second information, the first instructions being used to implement the first service; and and displaying a second interface based on the second information.

16. A display method applied to a second device, the second device being connected to a first device, receiving first information transmitted by the first device, the first information being information generated by executing first instructions, the first instructions being used to implement a first service, the first instructions being instructions to be executed as indicated by intent information, the intent information being determined by recognizing a first interface being displayed when the first device receives a first user operation, the first interface including second information, the second information being related to the first service; and displaying a second interface based on the first information.

17. An electronic device comprising a transceiver, a processor, and a memory, wherein the memory is configured to store a computer program, and wherein the processor invokes the computer program to perform the method of any one of claims 1 to 2, 6 to 8, and 13 to 15.

18. A computer storage medium storing a computer program, the computer program implementing the method according to any one of claims 1 to 2, 6 to 8, and 13 to 15 when executed by a processor.

Images