Display method and electronic device

Abstract

An implementation of this application provides a display method and an electronic device, the method including: a first device displays a first interface, the first interface includes first information, the first information is related to a first service; the first device receives a first user operation, and in response to the first user operation, recognizes the first interface and determines intent information, the intent information indicating executing a first instruction, the first instruction is used to implement the first service; the first device transmits the intent information to a second device, the intent information is used by the second device to execute the first instruction to generate second information, the second information is used by the second device to display the second interface, or the first device transmits the second information to the second device, the second information is generated by the first device by executing the first instruction, the second information is used by the second device to display the second interface. In the embodiments of this application, the interaction method in a multi-device interconnection scenario can be simplified, reducing user operations and improving efficiency.

Description

[Technical field]

[0001] This application claims priority to Chinese Patent Application No. 202111493706.2, entitled "DISPLAY METHOD AND ELECTRONIC DEVICE", filed with the State Intellectual Property Office of China on December 8, 2021, and claims priority to Chinese Patent Application No. 202210093485.8, entitled "DISPLAY METHOD AND ELECTRONIC DEVICE", filed with the State Intellectual Property Office of China on January 26, 2022, which are incorporated by reference in their entireties herein.

[0002] This application relates to the field of computer technology, and in particular to display methods and electronic devices. [Background technology]

[0003] In a scenario where multiple devices are connected and communicate with each other (e.g., distributed scenario), a user can use any one of the devices independently and can use multiple devices simultaneously (services of multiple devices may be related, e.g., a video on a smartphone is projected on a smart TV for playback). However, electronic devices in this scenario lack simple and efficient interaction, and user operations are complicated. For example, in a scenario where a smartphone is connected to an on-board computer, if a user receives a communication message containing location information using the smartphone, the user needs to start a map application on the on-board computer and set the destination to the location indicated by the location information to implement location-based navigation. As a result, operations are complicated. If the user is driving, it will affect the driving safety and reduce the user experience. Summary of the Invention

[0004] SUMMARY OF THE PRESENT APPLICATION The embodiments of this application disclose a display method and an electronic device for simplifying interaction methods, reducing user operations, and improving efficiency in a multi-device interconnection scenario.

[0005] According to a first aspect, an embodiment of the present application provides a display method applied to a first device, the first device being connected to a second device, the method including: displaying a first interface, the first interface including first information, the first information being related to a first service; receiving a first user operation, and recognizing the first interface in response to the first user operation to determine intent information, the intent information indicating executing a first instruction, the first instruction being used to implement the first service; and transmitting the intent information to a second device, the intent information being used by the second device to execute the first instruction to generate second information, the second information being used by the second device to display the second interface.

[0006] In some embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0007] In some embodiments, the second information is used by the second device to display the second interface and play the first audio, in some other embodiments, the second information is used by the second device to play the first audio, and the second device does not display the second interface.

[0008] In the above method, when the first device receives a first user operation, the first device recognizes a user's intention based on the currently displayed first interface, and the second device executes a first command. The first command is used to implement a first service corresponding to the recognized intention information. In this way, the user does not need to manually operate the first device or the second device to trigger the implementation of the first service. This reduces the user operation, and the interaction method in a multi-device interconnection scenario is more efficient and convenient.

[0009] In a possible implementation, the first interface further includes third information, the third information relating to the second device. Recognizing the first interface to determine the intent information includes: recognizing the first information to determine fourth information and recognizing the third information to determine fifth information, the fourth information indicating to execute a first instruction, the fifth information indicating to execute a second instruction, the second instruction being used to implement a second service; and determining from the fourth information and the fifth information that the intent information is the fourth information according to a first preset rule, the first preset rule including at least one of: a device type of the second device is a preset device type, services supported by the second device include a first service, and a priority of the first service is higher than a priority of the second service.

[0010] In some embodiments, the first information and the third information are instant messaging messages, and the first preset rule includes a receipt time of the first information being later than a receipt time of the third information.

[0011] In the above method, the first device may further determine, according to the first preset rule, the intention information which is more suitable for the user's requirement in the current scenario, so that the interaction accuracy is further improved and the user experience is better.

[0012] In a possible implementation, the first information is location information, the first service is a navigation service, the second service is different from the first service, and the first preset rule includes a device type of the second device being a preset device type, and the preset device type is an in-vehicle device.

[0013] In a possible implementation, the first information is video information, the first service is a video playback service, the second service is different from the first service, and the first preset rule includes that a device type of the second device is a preset device type, and the preset device type is a smart TV and a smart screen.

[0014] In a possible implementation, the first information is information indicating a first location, the first service is a navigation service, and the second information is display information generated by performing a navigation operation relative to the first location.

[0015] In some embodiments, the first device is a smartphone and the second device is an in-vehicle device.

[0016] In the above method, when the first device displays a first interface including location information, if a first user operation is received, a navigation service for the location information may be implemented using the second device. In this way, the user does not need to manually input location information and manually trigger a navigation operation on the second device, so that the interaction manner in a multi-device interconnection scenario is more efficient and convenient.

[0017] In a possible implementation, the first information is information indicating a first video, the first service is a video playback service, and the second information is display information generated by playing the first video.

[0018] In some embodiments, the first device is a smartphone and the second device is a smart television.

[0019] In the above method, when the first device displays a first interface including video information, if a first user operation is received, a service for playing video information may be implemented using the second device. In this way, the user does not need to manually search for video information on the second device and manually trigger the video playing service, so that the interaction method in a multi-device interconnection scenario is more efficient and convenient.

[0020] In a possible implementation, the first information is information indicating a first recipe, the first service is a cooking service, and the second information is display information generated to implement the cooking service corresponding to the first recipe.

[0021] In some embodiments, the first device is a smartphone and the second device is a smart food processor.

[0022] In the above method, when the first device displays a first interface including recipe information, if a first user operation is received, a cooking service corresponding to the recipe information may be implemented using the second device. In this way, the user does not need to manually search for the recipe information on the second device and manually trigger the cooking service, so that the interaction method in a multi-device interconnection scenario is more efficient and convenient.

[0023] In a possible implementation, the first information is information indicative of a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but not the answer to the first question.

[0024] In some embodiments, the first device is a smartphone and the second device is a tablet computer or a learning machine.

[0025] In the above method, when the first device displays a first interface including questions and answers, if a first user operation is received, the second device may display the questions but not the answers. In this way, the child can practice the questions on the second device, and the parent does not need to manually search for the questions on the second device or manually trigger the test paper generation service, so that the interaction manner is convenient and accurate, and can fully meet the requirements of the parent and the child.

[0026] In a possible implementation, the first user operation is, for example, a shake operation, a swing operation, a tap operation with a knuckle of a finger, a slide operation with a knuckle of a finger, a multi-finger tap operation, a multi-finger slide operation, or the like.

[0027] In the above method, the first user operation is simple and convenient, and the user does not need to perform complicated operations to trigger the implementation of the first service. Thus, the interaction threshold is lowered, and the user's use is more convenient.

[0028] According to a second aspect, the application provides another display method applied to a first device, the first device being connected to a second device, the method including: displaying a first interface, the first interface including first information, the first information being related to a first service; receiving a first user operation, in response to the first user operation, recognizing the first interface to determine intent information, executing a first instruction based on the intent information to generate second information, and transmitting the second information to the second device, the second information being used by the second device to display the second interface.

[0029] In some embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0030] In some embodiments, the second information is used by the second device to display the second interface and play the first audio, in some other embodiments, the second information is used by the second device to play the first audio, and the second device does not display the second interface.

[0031] In the above method, the first device, upon receiving a first user operation, may recognize a user's intention based on a currently displayed first interface and execute a first command indicated by the recognized intention information, and the second device outputs multimedia data by executing the first command. A first service corresponding to the first command may be understood to be implemented by the second device. In this way, the user does not need to manually operate the first device or the second device to trigger the implementation of the first service. This reduces user operations, and the interaction method in a multiple device interconnection scenario is more efficient and convenient.

[0032] In a possible embodiment, the first interface further includes third information, the third information relating to the second service. Recognizing the first interface to determine the intent information includes: recognizing the first information to determine fourth information, and recognizing the third information to determine fifth information, the fourth information indicating to execute a first instruction, the fifth information indicating to execute a second instruction, the second instruction being used to implement the second service; and determining from the fourth information and the fifth information that the intent information is the fourth information according to a first preset rule, the first preset rule including that a device type of the second device is a preset device type and / or that a priority of the first service is higher than a priority of the second service.

[0033] In some embodiments, the first information and the third information are instant messaging messages, and the first preset rule includes a receipt time of the first information being later than a receipt time of the third information.

[0034] In the above method, the first device may further determine, according to the first preset rule, the intention information which is more suitable for the user's requirement in the current scenario, so that the interaction accuracy is further improved and the user experience is better.

[0035] In a possible implementation, the first information is location information, the first service is a navigation service, the second service is different from the first service, and the first preset rule includes a device type of the second device being a preset device type, and the preset device type is an in-vehicle device.

[0036] In a possible implementation, the first information is information indicating a first location, the first service is a navigation service, and the second information is display information generated by performing a navigation operation relative to the first location.

[0037] In a possible implementation, the first information is information indicating a first video, the first service is a video playback service, and the second information is display information generated by playing the first video.

[0038] In a possible implementation, the first information is information indicating a first recipe, the first service is a cooking service, and the second information is display information generated to implement the cooking service corresponding to the first recipe.

[0039] In a possible implementation, the first information is information indicative of a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but not the answer to the first question.

[0040] In a possible implementation, the first user operation is, for example, a shake operation, a swing operation, a tap operation with a knuckle of a finger, a slide operation with a knuckle of a finger, a multi-finger tap operation, a multi-finger slide operation, or the like.

[0041] According to a third aspect, the present application provides another display method applied to a second device, the second device being connected to the first device. The method includes: receiving intent information transmitted by the first device, the intent information being determined by recognizing a first interface being displayed when the first device receives a first user operation, the first interface including first information, the first information being related to a first service; executing first instructions based on the intent information to generate second information, the first instructions being used to implement the first service; and displaying the second interface based on the second information.

[0042] In some other embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0043] In some embodiments, the second information is used by the second device to display the second interface and play the first audio, in some other embodiments, the second information is used by the second device to play the first audio, and the second device does not display the second interface.

[0044] In the above method, when the first device receives the first user operation, the first device may recognize the user's intention based on the currently displayed first interface, and send the recognized intention information to the second device. The second device may execute the first instruction indicated by the intention information to implement the first service. In this way, the user does not need to manually operate the first device or the second device to trigger the implementation of the first service. This reduces the user operation, and the interaction method in the multiple device interconnection scenario is more efficient and convenient.

[0045] In a possible implementation, the first information is information indicating a first location, the first service is a navigation service, and the second information is display information generated by performing a navigation operation relative to the first location.

[0046] In a possible implementation, the first information is information indicating a first video, the first service is a video playback service, and the second information is display information generated by playing the first video.

[0047] In a possible implementation, the first information is information indicating a first recipe, the first service is a cooking service, and the second information is display information generated to implement the cooking service corresponding to the first recipe.

[0048] In a possible implementation, the first information is information indicative of a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but not the answer to the first question.

[0049] In a possible implementation, the first user operation is, for example, a shake operation, a swing operation, a tap operation with a knuckle of a finger, a slide operation with a knuckle of a finger, a multi-finger tap operation, a multi-finger slide operation, or the like.

[0050] According to a fourth aspect, the application provides another display method applied to a second device, the second device being connected to the first device. The method includes: receiving first information transmitted by the first device, the first information being information generated by executing first instructions, the first instructions being used to implement a first service, the first instructions being instructions to be executed as indicated by intent information, the intent information being determined by recognizing a first interface being displayed when the first device receives a first user operation, the first interface including second information, the second information being related to the first service; and displaying the second interface based on the first information.

[0051] In some other embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0052] In some embodiments, the second information is used by the second device to display the second interface and play the first audio, in some other embodiments, the second information is used by the second device to play the first audio, and the second device does not display the second interface.

[0053] In the above method, the first device, upon receiving a first user operation, may recognize a user's intention based on a currently displayed first interface and execute a first command indicated by the recognized intention information, and the second device outputs multimedia data by executing the first command. A first service corresponding to the first command may be understood to be implemented by the second device. In this way, the user does not need to manually operate the first device or the second device to trigger the implementation of the first service. This reduces user operations, and the interaction method in a multiple device interconnection scenario is more efficient and convenient.

[0054] In a possible implementation, the second information is information indicating a first location, the first service is a navigation service, and the first information is display information generated by performing a navigation operation relative to the first location.

[0055] In a possible implementation, the second information is information indicating the first video, the first service is a video playback service, and the first information is display information generated by playing the first video.

[0056] In a possible implementation, the second information is information indicating a first recipe, the first service is a cooking service, and the first information is display information generated to implement the cooking service corresponding to the first recipe.

[0057] In a possible implementation, the second information is information indicative of a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but not the answer to the first question.

[0058] In a possible implementation, the first user operation is, for example, a shake operation, a swing operation, a tap operation with a knuckle of a finger, a slide operation with a knuckle of a finger, a multi-finger tap operation, a multi-finger slide operation, or the like.

[0059] According to a fifth aspect, an embodiment of the present application provides an electronic device including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors. The one or more memories are configured to store computer program code, the computer program code including computer instructions. When the one or more processors execute the computer instructions, the communication device is enabled to perform a display method according to any possible implementation of any one of the above aspects.

[0060] According to a sixth aspect, an embodiment of the present application provides a computer storage medium, the computer storage medium storing a computer program product, the computer program product, when executed by a processor, causing a display method according to any one of the above possible implementations to be performed.

[0061] According to a seventh aspect, an embodiment of the present application provides a computer program product, which when running on an electronic device, enables the electronic device to perform a display method according to any one of the possible implementations of the above aspects.

[0062] According to an eighth aspect, an embodiment of the present application provides an electronic device. The electronic device includes a method or an apparatus for performing any of the embodiments of the present application. For example, the electronic device is a chip.

[0063] It should be understood that the description of technical features, technical solutions, beneficial effects, or similar language in this application does not imply that all features and advantages can be implemented in any single embodiment. Conversely, the description of a feature or beneficial effect may be understood to indicate that a particular technical feature, technical solution, or beneficial effect is included in at least one embodiment. Thus, the description of a technical feature, technical solution, or beneficial effect in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in the embodiments may be combined in any suitable manner. Those skilled in the art may understand that an embodiment may be implemented in a particular embodiment without one or more particular technical features, technical solutions, or beneficial effects. In other embodiments, additional technical features and beneficial effects may be further recognized in a particular embodiment that does not reflect all embodiments. [Brief description of the drawings]

[0064] The accompanying drawings used in the embodiments of this application will be described below.

[0065] [Figure 1A] 1 is a schematic diagram of the architecture of a communication system 10 according to an embodiment of the present application.

[0066] [Figure 1B] 1 is a schematic diagram of the architecture of another communication system 10 according to an embodiment of the present application.

[0067] [Figure 1C] FIG. 2 is a schematic diagram of the architecture of yet another communication system according to an embodiment of the present application;

[0068] [Figure 2A] 1 is a schematic diagram of a hardware structure of an electronic device 100 according to an embodiment of the present application.

[0069] [Figure 2B] 2 is a schematic diagram of a hardware structure of an electronic device 200 according to an embodiment of the present application.

[0070] [Figure 2C] 3 is a schematic diagram of a hardware structure of a network device 300 according to an embodiment of the present application.

[0071] [Figure 2D] 1 is a schematic diagram of the software architecture of an electronic device 100 according to one embodiment of the present application.

[0072] [Figure 3A-1] 1 is a schematic diagram of several user interface embodiments according to embodiments of the present application. [Figure 3A-2] 1 is a schematic diagram of several user interface embodiments according to embodiments of the present application. [Figure 3B] 1 is a schematic diagram of several user interface embodiments according to embodiments of the present application. [Figure 3C]1 is a schematic diagram of several user interface embodiments according to embodiments of the present application.

[0073] [Figure 4A-1] 11A-11C are schematic diagrams of several further user interface embodiments according to embodiments of the present application. [Figure 4A-2] 11A-11C are schematic diagrams of several further user interface embodiments according to embodiments of the present application. [Figure 4B-1] 11A-11C are schematic diagrams of several further user interface embodiments according to embodiments of the present application. [Figure 4B-2] 11A-11C are schematic diagrams of several further user interface embodiments according to embodiments of the present application.

[0074] [Figure 4C-1] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application. [Figure 4C-2] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application.

[0075] [Figure 5A] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application. [Figure 5B] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application.

[0076] [Figure 6A] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application. [Figure 6B] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application.

[0077] [Figure 7A-1] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application. [Figure 7A-2] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application.

[0078] [Figure 7B-1] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application. [Figure 7B-2] FIG. 11 is a schematic diagram of another user interface embodiment according to an embodiment of the present application.

[0079] [Figure 8] 1 is a schematic diagram of a user interface according to an embodiment of the present application.

[0080] [Figure 9] 1 is a schematic flowchart of a display method according to an embodiment of the present application.

[0081] [Figure 10] 4 is a schematic flowchart of another display method according to an embodiment of the present application.

[0082] [Figure 11] 1 is a schematic flowchart of yet another display method according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0083] The technical solutions according to the embodiments of this application are described below clearly and completely with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise specified, " / " indicates "or". For example, A / B may indicate A or B. The term "and / or" in this specification only describes the association relationship between associated objects, and indicates that three relationships may exist. For example, A and / or B may represent three cases: only A exists, both A and B exist, and only B exists. Additionally, in the description of the embodiments of this application, "multiple" means two or more.

[0084] The terms "first" and "second" referred to below are intended for descriptive purposes only and shall not be understood as an indication or implication of relative importance or an implied indication of the number of technical features depicted. Thus, features qualified as "first" and "second" may explicitly or implicitly include one or more features. In the description of the implementation of this application, unless otherwise specified, "plurality" means two or more.

[0085] The embodiments of the present invention may be applied in scenarios where multiple devices are connected and communicate with each other, e.g., distributed scenarios. In this scenario, a user may use multiple devices simultaneously. In this case, the services of multiple devices may be associated, e.g., a video on a smartphone is projected on a smart TV for playback. However, electronic devices in this scenario lack simple and efficient interaction and user operations are complex. Specific examples are as follows:

[0086] Example 1: In a scenario where a smartphone is connected to an on-board computer, if a user uses the smartphone to receive a communication message containing location information, the user needs to start a map application on the on-board computer and set the destination to the location indicated by the location information to implement location-based navigation. As a result, the operation is complicated. If the user is driving, it will affect the driving safety and degrade the user experience.

[0087] Example 2: In a scenario where a smartphone is connected to a smart TV, if a user sees information (such as synopsis, video reviews, etc.) of a certain video on the smartphone and wants to watch that video on the smart TV, the user needs to search for the movie on the smart TV for playback, or the user needs to first start a video application and movie playback interface on the smartphone, operate the projection control, select the device where the movie will be projected (i.e., the smart TV), and project the movie on the smart TV for viewing. This results in complicated operations and low interaction efficiency.

[0088] Example 3: In a scenario where a smartphone is connected to a smart food processor, if a user sees information about a certain recipe on the smartphone and wants to make the corresponding dish using the smart food processor, the user needs to search for the recipe and make the dish on the smart food processor, resulting in complicated operations and low interaction efficiency.

[0089] Example 4: In a scenario where a smartphone is connected to a tablet computer, a child may use the tablet computer or learning machine for learning, and a parent may use the smartphone to search for related exercises. If a parent finds an exercise on the smartphone that the child wants to answer, the parent must search for the exercise again on the tablet computer or learning machine. This results in complicated operations and low interaction efficiency.

[0090] An embodiment of this application provides a display method. In response to a user operation, a first device may recognize a currently displayed first interface, determine intent information, and implement a service indicated by the intent information via a second device. In this way, a user does not need to manually trigger a second device to implement a service indicated by the intent information, and an efficient and convenient interaction method is provided that is applied to a multi-device interconnection scenario. This reduces the user's operations and improves the user's experience.

[0091] For example, a smartphone (first device) may recognize a chat interface (first interface) including a location card (a message displaying a geographic location in a card format) in response to a shake operation (user operation) and determine intent information. The intent information may indicate a navigation service to navigate to a location indicated by the location card, and the intent information may be obtained based on the location card. In this case, the smartphone may execute the navigation service based on the intent information and optionally indicate to set the location indicated by the location card as a destination in a map application for navigation.

[0092] A communication system 10 according to an embodiment of the present application will be described below.

[0093] FIG. 1A illustrates an example of a schematic diagram of the architecture of a communication system 10 according to one embodiment of the present application.

[0094] As shown in FIG. 1A , the communication system 10 may include an electronic device 100 , an electronic device 200 , and a network device 300 .

[0095] In some embodiments, the electronic device 100 may be connected to at least one electronic device 200 in a wired and / or wireless manner. Wired methods include, for example, high-definition multimedia interface (HDMI), universal serial bus (USB), coaxial cable, or optical fiber. Wireless methods include, for example, Bluetooth, wireless fidelity (Wi-Fi), near field communication (NFC) technology, or ultra-wideband (UWB). The electronic device 100 may communicate with the electronic device 200 via a connection line (e.g., Bluetooth or Wi-Fi). In this case, the information transmission speed between the electronic device 100 and the electronic device 200 is high, and a large amount of information can be transmitted.

[0096] In some other embodiments, the electronic device 100 may be connected to the network device 300 in a wired and / or wireless manner, and the network device 300 may be connected to at least one electronic device 200 in a wired and / or wireless manner. The electronic device 100 may communicate with the electronic device 200 through the network device 300. For example, the electronic device 100 is a smartphone, the electronic device 200 is a vehicle, and the network device 300 is a cloud server providing a HUAWEI HiCar function. In this case, the connection and projection between the electronic device 100 and the electronic device 200 may be implemented using the HUAWEI HiCar function.

[0097] In some other embodiments, the electronic device 100 may not be connected to the electronic device 200, but the electronic device 100 may establish a connection to the electronic device 200 and then communicate with the electronic device 200. The electronic device 200 may be understood to be an electronic device that is not connected to the electronic device 100 but can communicate with the electronic device 100. Optionally, the electronic device 100 may store connection information (e.g., Bluetooth address and password, Wi-Fi name and password) of at least one electronic device 200 and use the connection information to connect to at least one electronic device 200 (e.g., send information including the password to the electronic device 200 corresponding to the Bluetooth address to request to establish a connection). Optionally, the connection information of the electronic device 200 may be obtained when the electronic device 100 was previously connected to the electronic device 200. Optionally, the connection information of the electronic device 200 may be obtained by the electronic device 100 via the network device 300. For example, after logging in to a specific account, the electronic device 100 may obtain connection information of the electronic device 200 that previously logged in to the account. The manner in which the electronic device 100 obtains the connection information of the electronic device 200 is not limited in this application.

[0098] The electronic devices and network devices illustrated in FIG. 1A are merely examples and are not limited to any particular device form.

[0099] In this application, the electronic device 100 may be a mobile terminal such as a mobile phone, a tablet computer, a handheld computer, a personal digital assistant (PDA), a smart home device such as a smart TV, a smart camera, a smart food processor, a wearable device such as a smart band, a smart watch, a smart glasses, or another device such as a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a smart screen, or a learning machine. The description of the electronic device 200 is similar and will not be described in detail again. In the embodiment of this application, the specific type of the electronic device 100 and the electronic device 200 is not specifically limited.

[0100] In this application, network device 300 may include at least one server. In some embodiments, any server may be a hardware server. In some embodiments, any server may be a cloud server.

[0101] FIG. 1B illustrates an example schematic diagram of the architecture of another communication system 10 according to an embodiment of the present application.

[0102] As shown in FIG. 1B, the electronic device 100 in the communication system 10 may include an interface analysis module, an intent analysis module, and an intent trigger module, and the electronic device 200 in the communication system 10 may include an output module.

[0103] When the electronic device 100 detects a user action, for example via the sensor module 180 shown in FIG. 2A, the electronic device 100 may report an event corresponding to the user action (sometimes referred to as a trigger event) to the interface analysis module.

[0104] The interface analysis module of the electronic device 100 may recognize the user interface displayed by the electronic device 100 when receiving the trigger event and obtain an interface recognition result. In some embodiments, the interface analysis module may recognize and analyze the layer structure and text of the current interface through keyword extraction, natural language understanding (NLU), etc. The interface recognition result includes, for example, text information and structure information indicating the structure in the user interface. The interface recognition result is, for example, data in XML format, data in JSON format, or data in other existing formats. Note that the interface recognition result is not limited to these and may alternatively be data in a customized format. The interface analysis module may send the interface recognition result to the intent analysis module.

[0105] In some embodiments, the interface analysis module may recognize some pages in the displayed user interface and obtain the interface recognition result. For example, the user interface displayed by the electronic device 100 is a split interface. Suppose the split screen interface includes a page of a first application and a page of a second application, and the application last operated by the user is the first application. The interface analysis module may recognize the page of the first application and obtain the corresponding interface recognition result. Not limited thereto. The interface analysis module may recognize the page of the application selected by the user. The manner of determining the information that needs to be recognized in the user interface is not limited in this application.

[0106] The intention analysis module of the electronic device 100 may perform intention recognition based on the interface recognition result and obtain the intention information. The intention information may be specific data obtained by performing interface recognition and intention recognition in a user interface displayed by the electronic device 100. The intention information may be, for example, data in an XML format, data in a JSON format, or data in another existing format. Note that the intention information is not limited thereto, and may alternatively be data in a customized format. In some embodiments, from the user's perspective, the intention information indicates an objective that needs to be achieved. Optionally, the intention information indicates that the implemented service corresponds to some service information in the user interface displayed by the electronic device 100. In some embodiments, the interface recognition result includes first structural information and first text information. In this case, the intention analysis module may recognize the first structural information, determine an interface structure indicated by the first structural information, and then obtain the intention information based on the first text information and the determined interface structure. For example, the intention analysis module may obtain an interface structure of a location card and an interface structure of a text box through recognition, determine that the type of the text information “Beijing Railway Station” included in the location card is address information based on the interface structure of the location card, determine that the type of the text information “Meet here” included in the text box is chat information based on the interface structure of the text box, and obtain intent information indicating navigating to the geographic location “Beijing Railway Station” based on the address information “Beijing Railway Station” and the chat information “Meet here”. The intent analysis module may then send the intent information to the intent trigger module.

[0107] In some embodiments, the intent analysis module may further determine whether the intent information is valid. The intent analysis module transmits the intent information to the intent trigger module only when it determines that the intent information is valid. For example, when the intent information indicates to navigate to a geographic location "Beijing Railway Station", the intent analysis module determines whether the address information "Beijing Railway Station" in the intent information corresponds to an actual valid geographic location on the map. The intent analysis module transmits the intent information to the intent trigger module only when it determines that the address information "Beijing Railway Station" in the intent information corresponds to an actual valid geographic location on the map. In another example, when the intent information indicates to play a movie named "Movie 1", the intent analysis module determines whether the video information "Movie 1" in the intent information corresponds to an actual video that can be played. The intent analysis module transmits the intent information to the intent trigger module only when it determines that the video information "Movie 1" in the intent information corresponds to an actual video that can be played.

[0108] The intention trigger module of the electronic device 100 may perform an intention operation based on the intention information. In some embodiments, the intention trigger module may analyze the intention information to obtain a specific instruction and invoke the instruction to perform the intention operation. In some embodiments, from the user's perspective, the intention information may indicate a goal that needs to be achieved, and the intention operation may correspond to a user operation that the user needs to perform to achieve the goal. In other words, the user can control the electronic device 100 to perform the intention operation only after performing multiple user operations. In some embodiments, the intention trigger module may invoke a corresponding service module to perform the intention operation. For example, when the intention information indicates to navigate to the geographic location "Beijing Railway Station", the intention trigger module may invoke a navigation module of a map application to perform the intention operation, i.e., set the destination to the geographic location "Beijing Railway Station" and perform navigation. After performing the intention operation, the intention trigger module may send corresponding multimedia data (e.g., audio streams and video streams corresponding to the navigation service) to an output module of the electronic device 200.

[0109] After receiving the multimedia data sent by the intent trigger module of the electronic device 100, the output module of the electronic device 200 may output the multimedia data, for example, play an audio stream corresponding to the navigation service and display a video stream corresponding to the navigation service.

[0110] In some embodiments, the interface analysis module of the electronic device 100 may include an interface analysis model. The interface analysis model is used to recognize the displayed user interface and obtain an interface recognition result. Optionally, the interface analysis module may use content in the user interface displayed by the electronic device 100 as input for the interface analysis model to obtain an output interface recognition result. For example, the interface analysis module may use interface content including address information in text format as input to obtain output text structure and / or address information, or use interface content including address information in the form of a card (e.g., the location card described above) as input to obtain output card structure and / or address information.

[0111] In some embodiments, the intent analysis module of the electronic device 100 may include an intent analysis model that is used to perform intent recognition via the intent analysis module. Optionally, the intent analysis module may use the interface recognition results as input for the intent analysis model to obtain output intent information.

[0112] The above examples are not limiting. The interface analysis module and the intent analysis module of the electronic device 100 may be disposed in the same fusion module. The fusion module may include a fusion model, which is used to determine intent information based on the displayed user interface. Optionally, the fusion module may use the displayed interface content as an input of the fusion model to obtain output intent information. For example, the interface content including address information is used as an input of the fusion model to obtain output intent information. The intent information indicates to perform navigation to a location indicated by the address information.

[0113] In some embodiments, the electronic device 100 may train the interface analysis model and / or the intent analysis model, or the electronic device 100 may train the fusion model. In some other embodiments, the network device 300 in the communication system 10 may train the interface analysis module and / or the intent analysis model and send the trained interface analysis module and / or the trained intent analysis model to the electronic device 100, or the network device 300 may train the fusion model and send the trained fusion model to the electronic device 100. The manner in which the network device 300 sends the interface analysis module and / or the intent analysis model or the fusion model to the electronic device 100 is not limited in this application. For example, the electronic device 100 may send a request message to the network device 300 to request to obtain the aforementioned models after receiving a user operation. In another example, the network device 300 may send the aforementioned models to the electronic device 100 at a preset time interval, for example, once a week. In another example, when the version of the model is updated, the network device 300 may send the model with the updated version to the electronic device 100.

[0114] In some embodiments, electronic device 100 or network device 300 may train an interface analysis model by using the content in the user interface as input, and using the structure and text contained in the user interface as input. Example inputs and outputs are similar to the previous examples where an interface analysis model is used to recognize a displayed user interface. Details will not be described again here.

[0115] In some embodiments, the electronic device 100 or the network device 300 may train an intent analysis model by using the interface recognition results as input and the corresponding intent operations and / or corresponding intent information as output.

[0116] In some embodiments, electronic device 100 or network device 300 may train a fusion model by using content in a user interface as input and corresponding intent actions and / or corresponding intent information as output. For example, the fusion model may be trained by using content of a user interface that includes address information as input and an intent action (setting a destination to a location indicated by the address information and performing navigation) as output. Alternatively, the fusion model may be trained by using content of a user interface that does not include address information as input and corresponding user actions (e.g., actions performed by a user when electronic device 100 displays the user interface) as output. Without being limited thereto. Alternatively, the fusion model may be trained by using content of a user interface that does not include address information as input and information indicating no navigation intent as output.

[0117] This is not limited to the example of FIG. 1B. In some other embodiments, at least one of the interface analysis module, the intention analysis module, and the intention trigger module may be a module included in the electronic device 200, not a module included in the electronic device 100. For example, the intention trigger module is a module included in the electronic device 200. For a specific example, refer to FIG. 1C. As shown in FIG. 1C, after receiving the intention information sent by the intention analysis module of the electronic device 100, the intention trigger module of the electronic device 200 performs an intention operation based on the intention information, and sends multimedia data corresponding to the intention operation to the output module, and the output module outputs the multimedia data. The other description is similar to the description of FIG. 1B, and the details will not be described again.

[0118] The electronic device 100, the electronic device 200, and the network device 300 according to the embodiments of this application will be described below.

[0119] FIG. 2A illustrates an example schematic diagram of the hardware configuration of electronic device 100.

[0120] Hereinafter, the embodiments will be described in detail using the electronic device 100 as an example. The electronic device 100 shown in Fig. 2A is only an example and may have more or less components than those shown in Fig. 2A, a combination of two or more components, or a different arrangement of components. The components shown in Fig. 2A may be implemented using hardware, including one or more signal processing and / or application specific integrated circuits, software, or a combination of hardware and software.

[0121] As shown in FIG. 2A , the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identity module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, an optical proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0122] It may be understood that the structure shown in this embodiment of the invention does not constitute a particular limitation on the electronic device 100. In some other embodiments of this application, the electronic device 100 may include more or fewer components than those shown in the figures, a combination of some components, splits of some components, or a different arrangement of components. The components shown in the figures may be implemented by hardware, software, or a combination of software and hardware.

[0123] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), etc. The different processing units may be separate components or may be integrated into one or more processors.

[0124] The controller may generate operation control signals according to the instruction operation code and the time sequence signal to complete the complete control of instruction reading and instruction execution.

[0125] The memory may be disposed within the processor 110 and is configured to store instructions and data. In some embodiments, the memory within the processor 110 is a cache memory. The memory may store instructions or data that are just used or are used periodically by the processor 110. If the processor 110 needs to use the instructions or data again, the processor may retrieve the instructions or data directly from the memory. This avoids repeated accesses and reduces the latency of the processor 110, improving the efficiency of the system.

[0126] In some embodiments, the processor 110 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM), a universal serial bus (USB) interface, or the like.

[0127] The I2C interface is a bidirectional synchronous serial bus and includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus groups. The processor 110 may be separately coupled to the touch sensor 180K, the charger, the flashlight, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through an I2C interface, such that the processor 110 communicates with the touch sensor 180K through the I2C bus interface to implement the touch function of the electronic device 100.

[0128] The MIPI interface may be configured to connect the processor 110 to peripheral components such as a display 194 or a camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 via the CSI to implement a picture taking function of the mobile device 100. The processor 110 communicates with the display 194 via the DSI interface to implement a display function of the electronic device 100.

[0129] It may be understood that the interface connection relationships between modules shown in this embodiment of the present invention are merely examples for the purpose of explanation and do not constitute limitations on the structure of the electronic device 100. It should be noted that in other embodiments of the present invention, the electronic device 100 may use an interface connection method different from that in the above embodiment, or may use a combination of multiple interface connection methods.

[0130] The charging management module 140 is configured to receive a charging input from a charger.

[0131] The power management module 141 is configured to connect to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, etc.

[0132] The wireless communication functionality of the electronic device 100 may be implemented via antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, a baseband processor, and the like.

[0133] Antenna 1 and Antenna 2 are configured to transmit and receive electromagnetic signals. Each antenna in electronic device 100 may be configured to cover one or more communication frequency bands. Different antennas may be further multiplexed to improve antenna utilization. For example, Antenna 1 may be multiplexed as a diversity antenna in a wireless local area network. In some other embodiments, the antennas may be used in combination with tuning switches.

[0134] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves via the antenna 1, perform processing such as filtering or amplification on the received electromagnetic waves, and send the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 may further amplify the signal modulated by the modem processor and convert the amplified signal to electromagnetic waves for emission via the antenna 1. In some embodiments, at least some functional modules in the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 may be disposed in the same device as at least some modules of the processor 110.

[0135] The modem processor may include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-frequency signal. The demodulator is configured to demodulate the received electromagnetic signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to a baseband processor for processing. The low-frequency baseband signal is processed by the baseband processor and then transmitted to an application processor. The application processor outputs a sound signal via an audio device (such as, but not limited to, speaker 170A, receiver 170B, etc.) or displays an image or video on a display 194. In some embodiments, the modem processor may be an independent component. In some other embodiments, the modem processor may be independent from the processor 110 and may be disposed in the same device as the mobile communication module 150 or another functional module.

[0136] The wireless communication module 160 may provide solutions for wireless communication applied to the electronic device 100, including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, infrared (IR) technology, etc. The wireless communication module 160 may be one or more components integrating at least one communication processor module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 further receives signals to be transmitted from the processor 110, performs frequency modulation and amplification on the signals, and converts the processed signals into electromagnetic waves for emission via the antenna 2.

[0137] In some embodiments, in the electronic device 100, the antenna 1 and the mobile communication module 150 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, and the electronic device 100 can communicate with a network and other devices by using a wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a BeiDou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation system (SBAS).

[0138] The electronic device 100 may implement display functionality via a GPU, a display 194, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric computations and render images. The processor 110 may include one or more GPUs to execute program instructions that generate or modify display information.

[0139] The display 194 is configured to display images, videos, etc. The display 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0140] The electronic device 100 may implement imaging functionality via an ISP, a camera 193, a video codec, a GPU, a display 194, an application processor, and the like.

[0141] The ISP is configured to process data fed back by the camera 193. For example, during shooting, the shutter is pressed and light is transferred to the camera's photoreceptor through the lens. The light signal is converted to an electrical signal, which is then transferred to the ISP for processing to convert the electrical signal into a visible image. The ISP may further perform algorithmic optimization for image noise, brightness, etc. The ISP may further optimize parameters such as exposure and color temperature for the shooting scenario. In some embodiments, the ISP may be disposed within the camera 193.

[0142] The camera 193 is configured to capture still images or video. An optical image of an object is generated through a lens and projected onto a photoreceptor. The photoreceptor may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photoreceptor converts the optical signal into an electrical signal, which is then transferred to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB, YUV, etc. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0143] The digital signal processor is configured to process digital signals, and may process other digital signals in addition to the digital image signal.

[0144] Video codecs are configured to compress or decompress digital video.

[0145] The NPU is a neural-network (NN) computing processor, which can rapidly process input information by referring to the structure of biological neural networks, for example, by referring to the transfer mode between human brain neurons, and can also perform self-learning continuously. Applications such as intelligent cognition of the electronic device 100 can be implemented through the NPU, for example, image recognition, face recognition, speech recognition, and text understanding.

[0146] The external memory interface 120 may be configured to connect to an external memory card, such as a microSD card, to expand the storage capabilities of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, music, video, and other files may be stored on the external memory card.

[0147] The internal memory 121 may be configured to store computer executable program code. The executable program code includes instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., audio playback function or image playback function), and the like. The data storage area may store data created during use of the electronic device 100 (audio data, address book, and the like), and the like. Additionally, the internal memory 121 may include a high-speed random access memory, and may include a non-volatile memory, such as at least one magnetic disk storage device, flash memory, or universal flash storage (UFS). The processor 110 operates the instructions stored in the internal memory 121 and / or the instructions stored in the memory disposed within the processor to perform various functional applications and data processing of the electronic device 100.

[0148] The electronic device 100 may implement audio functions, such as music playback and recording, via an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, an application processor, etc.

[0149] The audio module 170 is configured to convert digital audio information to analog audio signals for output, and is also configured to convert analog audio inputs to digital audio signals, and may further be configured to encode and decode audio signals.

[0150] The speaker 170A, also referred to as a "loudspeaker," is configured to convert audio electrical signals into sound signals. The electronic device 100 may listen to music or receive calls in a hands-free mode through the speaker 170A.

[0151] Receiver 170B, also referred to as an "earpiece," is configured to convert electrical audio signals into sound signals. When receiving an incoming call or speech information via electronic device 100, receiver 170B may be placed near a human ear to hear the sound.

[0152] Microphone 170C, also referred to as "mike" or "mic", is configured to convert sound signals into electrical signals. When a user makes a call or sends a voice message, the user may emit a sound near microphone 170C through the user's mouth to input the sound signal into microphone 170C. At least one microphone 170C may be disposed within the electronic device.

[0153] The headset jack 170D is configured to connect to a wired headset.

[0154] The pressure sensor 180A is configured to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display 194. When a touch operation is performed on the display 194, the electronic device 100 detects the intensity of the touch operation via the pressure sensor 180A. The electronic device 100 may also calculate a touch position based on the detection signal of the pressure sensor 180A.

[0155] The gyroscope sensor 180B may be configured to determine a motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., axes x, y, and z) may be determined by using the gyroscope sensor 180B. The gyroscope sensor 180B may further be used in image stabilization scenarios, navigation scenarios, and physical gaming scenarios.

[0156] The air pressure sensor 180C is configured to measure air pressure. In some embodiments, the electronic device 100 calculates altitude via air pressure values ​​measured by the air pressure sensor 180C to assist in positioning and navigation.

[0157] The magnetic sensor 180D includes a Hall sensor. The electronic device may detect the opening and closing of the flip cover by using the magnetic sensor 180D.

[0158] The acceleration sensor 180E may detect acceleration in various directions (usually three axes) of the electronic device 100. The magnitude and direction of gravity may be detected when the electronic device 100 is stationary. The acceleration sensor 180E may be further configured to identify the orientation of the electronic device, which is used in applications such as switching between landscape and portrait modes, pedometers, etc.

[0159] The distance sensor 180F is configured to measure distance. The electronic device 100 may measure distance in an infrared or laser manner.

[0160] The ambient light sensor 180L is configured to sense the brightness of the ambient light.

[0161] The fingerprint sensor 180H is configured to collect a fingerprint, and the electronic device 100 may use collected fingerprint characteristics to implement fingerprint-based unlocking, application lock access, fingerprint-based photography, fingerprint-based call answering, and the like.

[0162] The temperature sensor 180J is configured to detect a temperature.

[0163] The touch sensor 180K may also be referred to as a "touch device." The touch sensor 180K may be disposed on the display 194, and the touch sensor 180K and the display 194 may comprise a touch screen, also referred to as a "touch screen." The touch sensor 180K is configured to detect touches occurring at or near the touch sensor. The touch sensor may forward the detected touches to an application processor to determine a type of touch event. A visual output associated with the touches may be provided via the display 194. In some other embodiments, the touch sensor 180K may alternatively be disposed on the surface of the mobile device 100 at a location different from the display 194.

[0164] The bone conduction sensor 180M may acquire a vibration signal.

[0165] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive the button input and generate button signal inputs related to user settings and function control of the electronic device 100.

[0166] The motor 191 may generate a vibration prompt.

[0167] Indicator 192 may be an indicator light and may be configured to indicate charging status and power changes, or may be configured to indicate messages, missed calls, notifications, and the like.

[0168] The SIM card interface 195 is configured to connect to a SIM card.

[0169] In some embodiments, the electronic device 100 may detect a user operation via the sensor module 180. In response to the user operation, the processor 110 may perform intent recognition based on a user interface displayed by the display 194. The electronic device 100 transmits instruction information to the electronic device 200 via the mobile communication module 150 and / or the wireless communication module based on the recognized intent information. After receiving the instruction information, the electronic device 200 may output multimedia data corresponding to the intent information, for example, display a navigation interface corresponding to the navigation intent.

[0170] For example, the electronic device 100 detects a touch operation performed by a user on the electronic device 100, such as tapping the display 194 with a knuckle or sliding the display 194 with a knuckle, two fingers, or three fingers, via the pressure sensor 180A and / or the touch sensor 180K. In another example, the electronic device 100 detects a user's shaking operation and hand swing operation via the gyroscope sensor 180B and / or the acceleration sensor 180E. In another example, the electronic device 100 detects a user's gesture operation via the camera 193. The module for detecting a user operation is not limited in this application.

[0171] FIG. 2B illustrates an example schematic diagram of the hardware configuration of electronic device 200.

[0172] Hereinafter, the embodiments will be described in detail using the electronic device 200 as an example. The electronic device 200 shown in Fig. 2B is only an example, and may have more or less components than those shown in Fig. 2B, may have two or more components in combination, or may have a different arrangement of components.

[0173] 2B, the electronic device 200 may include a processor 201, a memory 202, a wireless communication module 203, an antenna 204, and a display 205. Optionally, the electronic device 200 may further include a wired communication module (not shown).

[0174] Specifically, the processor 201 may be configured to read and execute computer-readable instructions. In a particular implementation, the processor 201 may mainly include a controller, an arithmetic logic unit, and a register. The controller is mainly responsible for decoding instructions and sending control signals for operations corresponding to the instructions. The arithmetic logic unit is mainly responsible for storing register operation amounts, intermediate operation results, etc. that are temporarily stored during instruction execution. In a particular implementation, the hardware architecture of the processor 201 may be an application specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, an NP architecture, etc. In some embodiments, the processor 201 may be further configured to generate a signal, such as a Bluetooth broadcast signal or a beacon signal, to be transmitted to the outside by the wireless communication module 203.

[0175] The memory 202 is coupled to the processor 201 and configured to store various software programs and / or instruction groups. In certain implementations, the memory 202 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash storage devices, or another non-volatile solid-state storage device. The memory 202 may store an operating system, for example, an embedded operating system, such as uCOS, VxWorks, or RTLinux. The memory 202 may further store a communication program. The communication program may be used to communicate with the electronic device 100 or another device.

[0176] The wireless communication module 203 may include one or more of a WLAN communication module 203A and a Bluetooth communication module 203B. Optionally, the Bluetooth communication module 203B may be integrated with another communication module (e.g., the WLAN communication module 203A).

[0177] In some embodiments, one or more of the WLAN communication module 203A and the Bluetooth communication module 203B may monitor a signal, e.g., a measurement signal or a scanning signal, transmitted by another device and transmit a response signal, e.g., a measurement response or a scanning response, to enable the other device to discover the electronic device 200 and establish a wireless communication connection to the other device using one or more of Bluetooth and WLAN, or other short-range wireless communication technologies, for data transmission.

[0178] In some other embodiments, the WLAN communication module 203A may transmit a signal, for example broadcasting a discovery signal or a beacon signal, so that the router may discover the electronic device 200 and establish a wireless communication connection to the router using WLAN and connect to the electronic device 100 and the network device 300.

[0179] The wired communication module (not shown) may be configured to establish a connection to a device such as a router via a network cable and to be connected to the electronic device 100 and the network device 300 via the router.

[0180] The antenna 204 is configured to transmit and receive electromagnetic signals. The antennas of different communication modules may be multiplexed to improve antenna utilization or may be independent of each other. For example, the antenna of the Bluetooth communication module 203B may be multiplexed with the antenna of the WLAN communication module 203A.

[0181] Display 205 may be configured to display images, videos, etc. Display 205 includes a display panel. The display panel may be a liquid crystal display, an organic light emitting diode, an active matrix organic light emitting diode, a flexible light emitting diode, a quantum dot light emitting diode, etc. In some embodiments, electronic device 200 may include one or N displays 205, where N is a positive integer greater than 1.

[0182] In some embodiments, the computing device 200 may further include a sensor. For a specific example, see the sensor module 180 shown in FIG. 2A. The details will not be described again here.

[0183] In some embodiments, the electronic device 200 may receive the instruction information transmitted by the electronic device 100 using the wireless communication module 203 and / or a wired communication module (not shown). The processor 201 may use the display 205 to display a user interface corresponding to the intent information based on the instruction information, e.g., a navigation interface corresponding to the navigation intent.

[0184] FIG. 2C illustrates an example schematic diagram of the hardware configuration of the network device 300.

[0185] 2C, the network device 300 may include one or more processors 301, a communication interface 302, and a memory 303. The processor 301, the communication interface 302, and the memory 303 may be connected via a bus or in another manner. In the embodiment of this application, an example in which the processor 301, the communication interface 302, and the memory 303 are connected via a bus 304 will be described.

[0186] Specifically, the processor 301 may include one or more general-purpose processors, such as a CPU, and may be configured to run program code related to a device control method.

[0187] The communication interface 302 may be a wired interface (e.g., an Ethernet interface) or a wireless interface (e.g., a cellular network interface or a wireless local area network interface) and is configured to communicate with another node. In this embodiment of the application, the communication interface 302 may be specifically configured to communicate with the electronic device 100 and the electronic device 200.

[0188] The memory 303 may include a volatile memory, such as a RAM. Alternatively, the memory may include a non-volatile memory, such as a ROM, a flash memory, a HDD, or a solid-state drive SSD. Alternatively, the memory 303 may include a combination of the aforementioned types of memory. The memory 303 may be configured to store a group of program codes, such that the processor 201A calls the program codes stored in the memory 203A to implement the method implemented by the server in the embodiment of this application. In this embodiment of this application, the memory 303 may alternatively be a storage array, etc.

[0189] In some embodiments, the network device 300 may include multiple servers, such as a web server, a background server, a download server, etc. For the hardware structure of the multiple servers, refer to the hardware structure of the network device 300 shown in FIG. 2C.

[0190] It should be understood that the network device 300 shown in FIG. 2C is only one implementation of an embodiment of this application. In an actual application, the network device 300 may alternatively include more or less components. This is not limited herein.

[0191] The software system of the electronic device 100 may use a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. For example, the software system of the layered architecture may be an Android system, a Huawei mobile services (HMS) system, or another software system. In this embodiment of the application, the Android system of the layered architecture is used as an example to describe the software structure of the electronic device 100.

[0192] FIG. 2D illustrates an example schematic diagram of the hardware configuration of electronic device 100.

[0193] In a layered architecture, the software is divided into several layers, each with a distinct role and task. These layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers from top to bottom: application layer, application framework layer, Android runtime and system / libraries, and kernel layer.

[0194] The application layer may include a set of application packages.

[0195] As shown in FIG. 2D, the application package may include applications such as a camera, a map, a HiCar, a music, a chat application, an entertainment application, a home application, and a learning application.

[0196] The application framework layer provides the application programming interface (API) and programming framework for the applications in the application layer. The application framework layer contains some predefined functionality.

[0197] As shown in FIG. 2D, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, an intent forwarding service, and the like.

[0198] A window manager is configured to manage window programs. A window manager may get the size of the display, determine if there is a status bar, perform screen locking, take screenshots, etc.

[0199] Content providers are configured to store and retrieve data and make the data accessible by applications. Data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0200] The view system includes visual controls, such as a control for displaying text and a control for displaying images. The view system may be configured to build an application. A display interface may include one or more views. For example, a display interface that includes an SMS message notification icon may include a text display view and an image display view.

[0201] The telephone manager is configured to provide communication functions for the electronic device 100, such as, for example, managing call status (including accepting calls, blocking calls, etc.).

[0202] The resource manager provides various resources such as localized strings, icons, pictures, layout files, video files, etc. for an application.

[0203] A notification manager may be configured to carry notification messages, allowing applications to display notification information in the status bar. A communication manager may disappear automatically after a short pause without user interaction.

[0204] The intent transfer service may perform intent recognition based on an application at the application layer. In some embodiments, the intent transfer service may perform intent recognition based on a user interface of an application displayed by electronic device 100. Electronic device 100 may implement the recognized intent via electronic device 200. In some cases, a service that is on electronic device 100 and that is used to implement the intent may be transferred to electronic device 200. In other cases, electronic device 100 may send the recognized intent to electronic device 200, and electronic device 200 may implement the recognized intent.

[0205] In some embodiments, the intent transfer service may provide services to a system application at the application layer to perform intent recognition for a third-party application at the application layer, for example, the system application is a HiCar application and the third-party application is a map application, a chat application, an entertainment application, a home application, a learning application, etc.

[0206] Not limited thereto. In some other embodiments, the intention transfer service may be an embedded service of an application in the application layer. For example, a server corresponding to an application (sometimes referred to as an application server for short) may provide the intention transfer service to the application. When receiving a user operation, the electronic device 100 may send the content on the currently displayed user interface to the application server. The application server performs intention recognition based on the interface content and sends the recognized intention information to the electronic device 100. The electronic device 100 implements the intention information via the electronic device 200.

[0207] In some embodiments, the intent transfer service may correspond to the intent analysis module, optional page analysis module, and optional intent trigger module shown in Figure 1B. For details, see the description of Figure 1B. The details will not be described again here.

[0208] In some embodiments, the application in the application layer may correspond to the intent trigger module shown in Figure 1B. In some embodiments, the application in the application layer may correspond to the display module shown in Figure 1B.

[0209] The Android runtime includes the kernel library and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0210] The core library contains two parts: the functions that need to be called in the Java language, and the Android kernel library.

[0211] The application layer and the application framework layer run on a virtual machine. The virtual machine executes the java files in the application layer and the application framework layer as binary files. The virtual machine is configured to implement functions such as object life cycle management, stack management, thread management, security and exception management, garbage collection, etc.

[0212] The system library may include multiple functional modules, such as a surface manager, a media library, a 3D graphics processing library (e.g., OpenGL ES), and a 2D graphics engine (e.g., SGL).

[0213] The Surface Manager is configured to manage the display subsystem and provide a blend of 2D and 3D layers for multiple applications.

[0214] The Media Library supports playback and recording of multiple commonly used audio and video formats, still image files, etc. The Media Library may support multiple audio and video coding formats, such as MPEG-4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0215] The 3D graphics processing library is configured to implement 3D graphics drawing, image rendering, compositing, layer processing, etc.

[0216] The 2D Graphics Engine is a drawing engine for 2D drawing.

[0217] The kernel layer is a layer between the hardware and the software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver. In some embodiments, the sensor driver may correspond to the detection module shown in FIG. 1B.

[0218] An example of the software and hardware operations of the electronic device 100 will now be described with reference to a navigation scenario.

[0219] Assume that the display 194 displays a user interface of a chat application, and the user interface is used to display address information of a location 1. When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including information such as touch coordinates and a timestamp of the touch operation). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies a control corresponding to the input event. For example, the touch operation is a touch-tap operation, and the control corresponding to the tap operation is a navigation control. The map application invokes an interface of the application framework layer to start the map application, and then invokes the kernel layer to start the display driver to display the navigation interface via the display 194. The destination of the navigation interface is the location 1.

[0220] The software architecture of the electronic device 200 is similar to the software architecture of the electronic device 100. See Figure 2D for a specific example.

[0221] Hereinafter, the interaction method in the embodiment of this application will be described with reference to an application scenario.

[0222] Scenario 1: The electronic device 100 is a smartphone, and the electronic device 200 is an on-board computer. When the electronic device 100 displays a user interface including address information, if a user operation (e.g., a shake operation) is received, the electronic device 100 may perform an intention operation, i.e., set the location indicated by the address information as a destination, perform navigation, and transmit audio and video streams (hereinafter referred to as audio and video streams for short) corresponding to the performed intention operation to the electronic device 200 for output. In this way, the user does not need to manually input address information into a map application on the electronic device 200 and operate navigation controls. In other words, the user does not need to manually trigger the execution of the intention operation to implement a more efficient and convenient interaction.

[0223] 3A-1 through 3C show example embodiments of user interfaces in an application scenario (e.g., scenario 1 described above).

[0224] As shown in FIG. 3A-1 , the electronic device 100 may display a user interface 310 of a chat application, and the user interface 310 may include a session name 311 and a chat window 312. Suppose the current session is a two-party session. The session name 311 may include a name of a chat participant, such as "Xiao Wang". Not limited thereto. If the current session is a multi-party session, the session name 311 may include a name of the current session, such as a group name. The chat window 312 may be configured to display a chat history of the current session, such as a message 3121 and a message 3122 sent by the chat participant. Message 3121 contains the text "Meet here", message 3122 contains a place name 3122A (containing the text "Beijing Railway Station") and location information for "Beijing Railway Station" 3122B (containing the text "A13 Maojiawan Hutong, Dongcheng District, Beijing"), and message 3122 is a location card showing the geographic location "Beijing Railway Station".

[0225] 3A-1, electronic device 100 may be connected to electronic device 200 using, for example, a HUAWEI HiCar function. Electronic device 200 may display a home screen 320. Home screen 320 may include one or more application icons, such as a map application icon, a phone application icon, a music application icon, a radio application icon, a dashboard camera application icon, and a settings application icon. Home screen 320 may further include a main menu control, which may be used to return to home screen 320.

[0226] As shown in FIG. 3A-1 , the electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and recognize a currently displayed user interface 310 in response to the user operation. In some embodiments, the electronic device 100 may recognize the message 3122 to obtain location information of the geographical location “Beijing Railway Station”, and determine, based on the location information, intent information, i.e., to perform navigation to the geographical location “Beijing Railway Station”. Optionally, the electronic device 100 may alternatively refer to the message 3121 to determine that the user wants to go to the geographical location “Beijing Railway Station”, and determine the intent information based on the user's intent. Optionally, the intent information may be understood to correspond to a navigation service, or the intent information may be understood to correspond to the message 3122 (location card). The electronic device 100 may perform, based on the obtained intent information, an intent operation corresponding to the intent information, where the intent operation is to set a destination to the location information of the geographical location “Beijing Railway Station”, and perform navigation. The electronic device 100 may then transmit audio and video streams corresponding to the performed intent operation to the electronic device 200 for output. For details, see FIG. 3A-2. Optionally, the intent operation may be understood to be used to implement a navigation service, or the intent operation may be understood to correspond to the message 3122 (location card).

[0227] 3A-2, the electronic device 200 may display a map application user interface 330. The user interface 330 is used to display information related to navigation services. The user interface 330 may include a map window 331, a route window 332, and a prompt box 333.

[0228] Specifically, the map window 331 is used to display a schematic diagram of the selected navigation route on a map.

[0229] The route window 332 includes navigation information 332A, a route 332B, a route 332C, and a navigation control 332D. The navigation information 332A includes a text "Go to A13 Maojiawan Hutong, Dongcheng..." indicating the location information of the navigation destination. The navigation information 332A indicates only a part of the location information of the destination. The electronic device 200 may also display all the location information of the destination in response to a touch operation (e.g., a tap operation) performed on the navigation information 332A. The route 332B and the route 332C may indicate two navigation routes. Compared with the route 332C, the route 332B is highlighted (e.g., the text of the route 332B is highlighted in bold, but the text of the route 332C is not highlighted in bold), which indicates that the currently selected navigation route is the navigation route indicated by the route 332B. In this case, the map window 331 is used to display a schematic diagram of the navigation route indicated by the route 332B on the map. In response to a touch operation (e.g., a tap operation) performed on the path 332C, the electronic device 200 may remove the highlighting of the path 332B and highlight the path 332C. In this case, the selected navigation path is the navigation path indicated by the path 332C, and the map window 331 displays a schematic diagram of the navigation path indicated by the path 332C on the map. The navigation control 332D may be configured to enable a navigation function. In response to a touch operation (e.g., a tap operation) performed on the navigation control 332D, the electronic device 200 may perform navigation based on the currently selected path (the navigation path indicated by the path 332B in the user interface 330).

[0230] The prompt box 333 is used to display information about the current navigation service. The prompt box 333 contains the text "Navigating to A13 Maojiawan Hutong, Dongcheng District, Beijing in chat with Xiao Wang" which may indicate detailed location information of the destination of the navigation service. The navigation service is triggered by a chat session with the chat participant "Xiao Wang" in the chat application, and the detailed location information of the destination is obtained from the chat session.

[0231] In the example shown in Figures 3A-1 and 3A-2, the address information (i.e., message 3122) included in the user interface 310 displayed by the electronic device 100 is displayed in a card format. This is not limiting. In some other examples, the address information may alternatively be displayed in a text format. See Figure 3B for a specific example. This is not limiting in this application.

[0232] As shown in FIG. 3B, the electronic device 100 may display a user interface 340 of a chat application. The user interface 340 is similar to the user interface 310 shown in FIG. 3A-1, and the two user interfaces differ in the chat history of the current session. The user interface 340 may include a message 341 and a message 342. The message 341 includes the text "Where shall we meet" and the message 342 includes the text "Meet at the Beijing Railway Station". The electronic device 100 may be connected to the electronic device 200, and the electronic device 200 may display the home screen 320 shown in FIG. 3A-1. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and recognize the currently displayed user interface 340 in response to the user operation. In some embodiments, the electronic device 100 recognizes the message 342 to obtain information indicating that the user wants to go to the geographic location "Beijing Railway Station", and determines, based on the obtained information, the intention information, i.e., to perform navigation to the geographic location "Beijing Railway Station". Optionally, the intention information may be understood to correspond to the message 342. The electronic device 100 may perform an intention operation corresponding to the intention information based on the obtained intention information, where the intention operation is to set a destination to the location information of the geographic location "Beijing Railway Station" and perform navigation. Then, the electronic device 100 may transmit audio and video streams corresponding to the performed intention operation to the electronic device 200 for output. For details, see FIG. 3A-2. Optionally, the intention operation may be understood to correspond to the message 342.

[0233] In the above examples, the address information displayed by electronic device 100 is displayed in the form of a chat message (i.e., message 3122 of user interface 340 or message 342 of user interface 310). In some other examples, the address information may alternatively be displayed in a location description. See FIG. 3C for a specific example. This is not a limitation of this application.

[0234] As shown in FIG. 3C, the electronic device 100 may display a user interface 350 of an entertainment application. The user interface 350 includes a place name 351 and a location control 352. The place name 351 includes text "Capital Museum", which is the name of the place displayed in the user interface 350. The location control 352 includes text "16 Fuxingmen Outer Street, Xicheng District", which is location information of the place displayed in the user interface 350, and may indicate address information of the place "Capital Museum". The electronic device 100 may be connected to the electronic device 200, and the electronic device 200 may display the home screen 320 shown in FIG. 3A-1. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100), and in response to the user operation, recognize the currently displayed user interface 350 to obtain location information of the place named "Capital Museum", and determine, based on the location information, intent information, i.e., to navigate to the place "Capital Museum". Optionally, the intention information may be understood to correspond to the location control 352. The electronic device 100 may send instruction information to the electronic device 200 based on the acquired intention information, and the electronic device 200 may perform an intention operation corresponding to the intention information based on the instruction information, where the intention operation is to set a destination to the location information of the place "Capital Museum" and perform navigation. A specific example is similar to that of FIG. 3A-2. There is a difference in the destination and the navigation route. Optionally, the intention operation may be understood to correspond to the location control 352.

[0235] In the embodiment shown in FIG. 3A-1 to FIG. 3B, the electronic device 100 performs an intention operation corresponding to the intention information, and then transmits audio and video streams corresponding to the intention operation to the electronic device 200 for output. It may be understood that the content on the electronic device 100 is projected on the electronic device 200 for output, and the service on the electronic device 100 is actually triggered. In the embodiment shown in FIG. 3C, the electronic device 100 indicates to the electronic device 200 to perform an intention operation corresponding to the intention information, and the service on the electronic device 200 is actually triggered. This is not limited. In a specific embodiment, in the embodiment shown in FIG. 3A-1 to FIG. 3B, the service on the electronic device 200 may be alternatively triggered, and in the embodiment shown in FIG. 3C, the service on the electronic device 100 may be alternatively triggered. In the following embodiment, an example in which a service on the electronic device 200 is triggered is used for explanation. However, this is not limited in a specific implementation.

[0236] In a possible implementation, the service type corresponding to the intent information determined by the electronic device 100 is related to the device type of the electronic device 200 connected to the electronic device 100 .

[0237] Scenario 2: When the electronic device 100 is a smartphone and displays a user interface including address information and video information, the electronic device 100 receives a user operation (e.g., a shake operation). If the electronic device 200 connected to the electronic device 100 is an on-board computer, the electronic device 100 may send instruction information to the electronic device 200 in response to the user operation, and the electronic device 200 may perform an intention operation corresponding to a navigation service based on the instruction information, the intention operation being to set a location indicated by the address information as a destination and perform navigation. For a specific example, see Figures 4A-1 and 4A-2. If the electronic device 200 connected to the electronic device 100 is a smart TV, the electronic device 100 may send instruction information to the electronic device 200 in response to the user operation, and the electronic device 200 may perform an intention operation corresponding to a video service based on the instruction information, the intention operation being to play a video indicated by the video information. For a specific example, see Figures 4B-1 and 4B-2. In this way, it can better meet user requirements in real application scenarios and further improve dialogue accuracy.

[0238] 4A-1 to 4B-2 show example embodiments of user interfaces in an application scenario (e.g., scenario 2 described above).

[0239] As shown in FIG. 4A-1, the electronic device 100 may display a user interface 410 of a chat application. The user interface 410 is similar to the user interface 310 shown in FIG. 3A-1, and the two user interfaces differ in different chat histories of the current session. The user interface 410 may include a message 411, a message 412, a message 413, and a message 414. The messages 411 and 412 are the messages 3121 and 3122, respectively, of the user interface 310 shown in FIG. 3A-1. The details will not be described again here. The message 413 includes the text "Look at this", and the message 414 is a message for displaying a video in a card format, and the message 414 includes the text "My Day", which is the name of the video being displayed. The electronic device 100 may be connected to the electronic device 200 (on-board computer), and the electronic device 200 (on-board computer) may display the home screen 320 shown in FIG. 3A-1. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and recognize a currently displayed user interface 410 in response to the user operation. In some embodiments, the electronic device 100 recognizes the user interface 410 to obtain information indicating that the message 412 corresponds to a navigation service and the message 414 corresponds to a video service. The electronic device 100 may determine a corresponding navigation service based on a device type (on-board computer) of the connected device. For example, a correspondence between the on-board computer and the navigation service is preset. In this case, the electronic device 100 recognizes the message 412 and determines intent information corresponding to the navigation service, where the intent information indicates to navigate to the geographic location "Beijing Railway Station."The electronic device 100 may send instruction information to the electronic device 200 (on-board computer) based on the acquired intention information, and the electronic device 200 (on-board computer) may perform an intention operation corresponding to the intention information based on the instruction information, where the intention operation is to set a destination to the location information of the geographic location "Beijing Railway Station" and perform navigation. For details, see FIG. 4A-2. The user interface displayed by the electronic device 200 in FIG. 4A-2 is consistent with the user interface displayed by the electronic device 200 in FIG. 3A-2.

[0240] As shown in FIG. 4B-1, the electronic device 100 may display the user interface 410 shown in FIG. 4A-1. The electronic device 100 may be connected to the electronic device 200 (smart TV), which may display a home screen 420. The home screen 420 may include one or more categories, such as a TV series category, a video category, an anime category, a kids category, and a game category. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and recognize the currently displayed user interface 410 in response to the user operation. In some embodiments, the electronic device 100 recognizes the user interface 410 to obtain information indicating that the message 412 corresponds to a navigation service and the message 414 corresponds to a video service. The electronic device 100 may determine the corresponding navigation service based on the device type (smart TV) of the connected device. For example, a correspondence between a smart TV and a video service is preset. In this case, the electronic device 100 recognizes the message 414 and determines the intention information corresponding to the video service, where the intention information indicates to play a video named "My Day". The electronic device 100 may send instruction information to the electronic device 200 (smart TV) based on the obtained intention information, and the electronic device 200 (smart TV) may perform an intention operation corresponding to the intention information based on the instruction information, where the intention operation is to play a video named "My Day". For details, see FIG. 4B-2. As shown in FIG. 4B-2, the electronic device 200 may display a user interface 430. The user interface 430 includes a title 431. The title 431 includes the text "My Day", which is the name of the currently played video.

[0241] This is not limited to the above examples. In some other example scenarios, the user may select the service information to be recognized, and the service type corresponding to the intent information is determined based on the user selection. For specific examples, see Figures 4C-1 and 4C-2.

[0242] As shown in Fig. 4C-1, the electronic device 100 may display the user interface 410 shown in Fig. 4A-1. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and, in response to the user operation, display the user interface 440 shown in Fig. 4C-2.

[0243] As shown in FIG. 4C-2, the user interface 440 may include prompt information 441, a prompt box 442, and a prompt box 443. The prompt information 441 includes the text “Please select the service you need to transfer” that is used to prompt the user to select the recognized service information.

[0244] The prompt box 442 includes a service name 442A and service information 442B, where the service name 442A includes the text "Map Navigation", and the service information 442B is the message 412 of the user interface 410 shown in FIG. 4C-1. In response to a touch operation (e.g., a tap operation) performed on the prompt box 442, the electronic device 100 may determine that the recognized service information is the message 412 of the user interface 410, recognize the message 412, and obtain intention information corresponding to a navigation service, where the intention information indicates performing navigation to the geographic location "Beijing Railway Station". The electronic device 100 may send instruction information to the connected electronic device 200 based on the obtained intention information, and the electronic device 200 may perform an intention operation corresponding to the intention information based on the instruction information. For example, the interfaces displayed before and after the electronic device 200 receives the instruction information refer to the user interface 320 shown in FIG. 4A-1 and the user interface 330 shown in FIG. 4A-2.

[0245] The prompt box 443 includes a service name 443A and service information 443B, where the service name 443A includes the text "Play Video", and the service information 443B is the message 414 of the user interface 410 shown in FIG. 4C-1. In response to a touch operation (e.g., a tap operation) performed on the prompt box 443, the electronic device 100 may determine that the recognized service information is the message 414 of the user interface 410, and recognize the message 414 to obtain intention information corresponding to the video service, where the intention information indicates to play a video named "My Day". The electronic device 100 may transmit instruction information to the connected electronic device 200 based on the obtained intention information, and the electronic device 200 may perform an intention operation corresponding to the intention information based on the instruction information. For example, the interfaces displayed before and after the electronic device 200 receives the instruction information refer to the user interface 420 shown in FIG. 4B-1 and the user interface 430 shown in FIG. 4B-2.

[0246] Scenario 3: The electronic device 100 is a smartphone and the electronic device 200 is a smart TV. When the electronic device 100 displays a user interface including video information, if a user operation (e.g., a shake operation) is received, the electronic device 100 may transmit instruction information to the electronic device 200. The electronic device 200 may perform an intention operation, i.e., playing a video shown by the video information, based on the instruction information. In this way, the user does not need to manually trigger the execution of the intention operation to implement a more efficient and convenient interaction.

[0247] 5A and 5B show example embodiments of a user interface in another application scenario (e.g., scenario 3 above).

[0248] As shown in FIG. 5A, the electronic device 100 may display a user interface 510 of an entertainment application. The user interface 510 includes a name 521. The name 521 includes the text "Movie 1", which is the name of the movie being displayed in the user interface 510. The user interface 510 is used to display details about "Movie 1", such as related videos, stills, and video reviews. The electronic device 100 may be connected to the electronic device 200, which may display a home screen 420 shown in FIG. 4B-1. The home screen 420 further includes a search control 421. The search control 421 is configured to implement a search function. A user may input and play a desired video to be viewed based on the search function. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100), and in response to the user operation, recognize the currently displayed user interface 510 to obtain information about the movie named "Movie 1", and based on the information, determine intention information, i.e., to play the movie named "Movie 1". The electronic device 100 may send instruction information to the electronic device 200 based on the obtained intention information, and the electronic device 200 may perform an intention operation corresponding to the intention information based on the instruction information, where the intention operation is to play the movie named "Movie 1". For details, see FIG. 5B.

[0249] As shown in FIG. 5B, the electronic device 200 may display a user interface 520. The user interface 520 includes a title 521. The title 521 includes the text "Movie 1", which is the name of the currently playing video. In some cases, the electronic device 100 may retrieve a video stream of "Movie 1" from a video application and continuously transmit the video stream to the electronic device 200 for playing. This may be understood as projecting the video on the electronic device 100 to the electronic device 200 for playing. In this way, the user does not need to start a video application and a video playing interface on the electronic device 100 (smartphone), operate the projection control, and select the device (smart TV) to which the video is to be projected. In other cases, the electronic device 200 searches for and plays the video after receiving the instruction information. This may be understood as playing the video on the electronic device 200. In this way, the user does not need to search for videos on the electronic device 200 (smart TV) (e.g., search for videos using the search control 421 in the user interface 420 shown in FIG. 5A ), thus simplifying user operations and greatly improving interaction efficiency.

[0250] In the examples shown in Figures 5A and 5B, the video information displayed by the electronic device 100 is displayed in a video introduction. In other examples, the video information may alternatively be displayed in the form of a chat message, such as message 414 of the user interface 410 shown in Figure 4B-1. See Figures 4B-1 and 4B-2 for specific example scenarios, which are not limiting in this application.

[0251] Scenario 4: The electronic device 100 is a smartphone, and the electronic device 200 is a smart food processor. When the electronic device 100 displays a user interface including recipe information, if a user operation (e.g., a shake operation) is received, the electronic device 100 may send instruction information to the electronic device 200. The electronic device 200 may perform an intention operation, i.e., working based on the recipe information, based on the instruction information. In this way, the user does not need to search for a recipe on the smart food processor to perform cooking. In other words, the user does not need to manually trigger the execution of the intention operation to implement a more efficient and convenient interaction.

[0252] 6A and 6B show example embodiments of a user interface in another application scenario (e.g., scenario 4 above).

[0253] As shown in FIG. 6A, the electronic device 100 may display a user interface 610 of a home application. The user interface 610 includes a title 611. The title 611 includes the text “Crispy Pork Belly”, which is the name of the recipe displayed in the user interface 610. The user interface 610 is used to display details about the recipe named “Crispy Pork Belly”, such as ingredient information 612 and cooking steps 613. The electronic device 100 may be connected to the electronic device 200, and the electronic device 200 may display a home page 620. The home page 620 may include one or more categories, such as a daily recipe category, a Chinese category, a Western category, etc. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and, in response to the user operation, recognize the currently displayed user interface 610 to obtain information about the recipe named “Crispy Pork Belly”, and based on the information, determine intent information, i.e., to cook a dish corresponding to the recipe. The electronic device 100 may send instruction information to the electronic device 200 based on the acquired intention information, and the electronic device 200 may perform an intention operation corresponding to the intention information based on the instruction information, where the intention operation is to work according to the recipe. For details, see FIG. 6B.

[0254] As shown in FIG 6B, the electronic device 200 may display a user interface 630. The user interface 630 includes a title 631 and step information 632. The title 631 includes the text "Crispy Pork Belly", which is the name of the currently used recipe. The step information 632 indicates a cooking step of the currently used recipe, which corresponds to cooking step 613 in the user interface 610 shown in FIG 6A. The user interface 630 may indicate that the electronic device 200 is currently working based on a recipe named "Crispy Pork Belly".

[0255] This is not limited to the above examples. In some other examples, when the electronic device 100 recognizes the currently displayed user interface 610, it may only recognize the dish name "crispy pork belly" on the recipe, and determine the intention information, i.e., to cook the area named "crispy pork belly" based on the dish name. After receiving the instruction information, the electronic device 200 may perform an intention operation corresponding to the intention information, and the intention operation is to search for the dish name to obtain the corresponding recipe, and work based on the found recipe.

[0256] Scenario 5: The electronic device 100 is a smartphone used by a parent, and the electronic device 200 is a tablet computer used by a child for learning. When the electronic device 100 displays a user interface including learning information, if a user operation (e.g., a shake operation) is received, the electronic device 100 may transmit instruction information to the electronic device 200. Based on the instruction information, the electronic device 200 may perform an intention operation, i.e., displaying all or part of the learning information. In this way, the parent does not need to search the tablet computer for learning information for the child to learn and use. In other words, the user does not need to manually trigger the execution of the intention operation to implement a more efficient and convenient interaction.

[0257] 7A-1 and 7A-2 show example embodiments of user interfaces in another application scenario (e.g., scenario 5 above).

[0258] As shown in FIG. 7A-1, the electronic device 100 may display a user interface 710 of a learning application. The user interface 710 includes a title 711. The title 711 includes text "English test paper" indicating that the user interface 710 is used to display details about a test paper named "English test paper". The user interface 710 further includes details about a plurality of exercises, such as exercise 712, exercise 713, etc. The exercise 712 includes a question 712A and an answer 712B, and the exercise 713 includes a question 713A and an answer 713B. The user interface 710 further includes a test control 714. The test control 714 is configured to provide a function to take a practice test on the current test paper. The electronic device 100 may be connected to the electronic device 200, and the electronic device 200 may display a home screen 720. The home screen 720 may include one or more application icons, such as a clock application icon, a calendar application icon, a gallery application icon, and a settings application icon. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100), and in response to the user operation, recognize the currently displayed user interface 720 to obtain information about a test paper named "English test paper", and based on the information, determine intention information, i.e., to take a practice test on the test paper. The electronic device 100 may send instruction information to the electronic device 200 based on the obtained intention information, and the electronic device 200 may perform an intention operation corresponding to the intention information based on the instruction information, where the intention operation is to enable a function of taking a practice test on the test paper. For details, see FIG. 7A-2.

[0259] As shown in FIG. 7A-2, the electronic device 200 may display a user interface 730. The user interface 730 includes a title 731, a submission control 732, question information 733, and a switching control 734. The title 731 includes the text "English test paper", which is the name of the test paper for the currently taking practice test. The submission control 732 is configured to end the current practice test and display the results of the practice test. The question information 733 displays information about the currently viewed question, and the switching control 734 is configured to switch information about the currently viewed question. The user interface 730 may indicate that the ability to take a practice test is currently enabled for the test paper named "English test paper".

[0260] In some embodiments, the electronic device 200 responds to a touch operation (e.g., a tap operation) made on the submission control 732 by displaying the results of the practice test and transmitting the results of the practice test to the electronic device 100, thereby allowing parents to efficiently and conveniently learn about their child's learning progress.

[0261] In the examples shown in Figures 7A-1 and 7A-2, the user interface 710 displayed by the electronic device 100 includes questions and answers, but the user interface 730 displayed after the electronic device 200 receives instruction information includes only questions and no answers. In this way, the parent does not need to search for corresponding practice questions on the electronic device 200 and does not need to manually enable the function of taking a mock test. This further reduces user operations and improves interaction efficiency.

[0262] This is not limited to the examples shown in Figures 7A-1 and 7A-2. In some other examples, the electronic device 100 may recognize the exercises 712 and / or exercises 713 of the currently displayed user interface 710 in response to an operation of shaking the electronic device 100, and determine the intention information, i.e., to practice the exercises 712 and / or exercises 713 based on the exercises 712 and / or exercises 713. In this case, the electronic device 200 may perform a corresponding intention operation, i.e., displaying the questions 712A of the exercises 712 and / or the questions 713A of the exercises 713 for use by the child for practice after receiving the instruction information transmitted by the electronic device 100 based on the intention information. A specific example is similar to that of the user interface 730 shown in Figure 7A-2.

[0263] This is not limited to the above examples. In some other example scenarios, the user may select the service information to be recognized, and the service content corresponding to the intent information is determined based on the user selection. For specific examples, see Figure 7B-1 and Figure 7B-2.

[0264] As shown in Fig. 7B-1, the electronic device 100 may display the user interface 710 shown in Fig. 7A-1. The electronic device 100 may receive a user operation (e.g., shaking the electronic device 100) and, in response to the user operation, display the user interface 740 shown in Fig. 7B-2.

[0265] 7B-2, the user interface 740 may include prompt information 741, a prompt box 742, and a prompt box 743. The prompt information 741 includes the text "Please select the content you want to transfer" that is used to prompt the user to select the recognized service information.

[0266] The prompt box 742 is a question 712A of the exercise 712 in the user interface 740 shown in FIG. 7B-1. In addition, the electronic device 100 may determine that the recognized service information is the exercise 712 in the user interface 740 in response to a touch operation (e.g., a tap operation) performed on the prompt box 742, recognize the exercise 712, and obtain the intention information, i.e., to practice the exercise 712. In this case, the electronic device 200 may perform a corresponding intention operation, i.e., to display the question 712A of the exercise 712, after receiving the instruction information transmitted by the electronic device 100 based on the intention information. A specific example is similar to that of the user interface 730 shown in FIG. 7A-2.

[0267] The prompt box 743 is a question 713A of the exercise 713 in the user interface 740 shown in FIG. 7B-1. In response to a touch operation (e.g., a tap operation) performed on the prompt box 743, the electronic device 100 may determine that the recognized service information is the exercise 713 in the user interface 740, recognize the exercise 713, and obtain the intention information, i.e., to practice the exercise 713. In this case, the electronic device 200 may perform a corresponding intention operation, i.e., to display the question 713A of the exercise 713, after receiving the instruction information transmitted by the electronic device 100 based on the intention information. A specific example is similar to that of the user interface 730 shown in FIG. 7A-2.

[0268] In the above scenario 5, the electronic device 200 may alternatively be a learning machine-like device.

[0269] In the above example, the user operation that triggers the intention transfer (shortly referred to as a trigger operation) is a shake operation. In some other examples, the trigger operation may alternatively be a knuckle slide operation. See FIG. 8A for a specific example. In some other examples, the trigger operation may alternatively be a two-finger slide operation. See FIG. 8B for a specific example. In some other examples, the trigger operation may alternatively be a gesture operation. See FIG. 8C for a specific example. This is not limited thereto. Alternatively, the trigger operation may be a knuckle tap operation, a hand swing operation, or another operation. The specific type of the trigger operation is not limited in this application.

[0270] The display method provided in the embodiment of this application will be described below.

[0271] FIG. 9 shows an example of a schematic flowchart of a display method according to an embodiment of the present application.

[0272] This display method may be applied to the above-mentioned communication system 10. The communication system 10 may include an electronic device 100, an electronic device 200, and a network device 300.

[0273] As shown in FIG. 9, the method includes, but is not limited to, the following steps.

[0274] S101: The electronic device 100 establishes a connection to the electronic device 200.

[0275] In some embodiments, electronic device 100 may be directly connected to electronic device 200 in a wired and / or wireless manner, for example using Bluetooth or Wi-Fi. In some other embodiments, electronic device 100 may be connected to electronic device 200 via network device 300. See the description of the connection between electronic device 100 and electronic device 200 in FIG. 1A for more details.

[0276] S102: The electronic device 100 displays a first interface including first service information.

[0277] In some embodiments, the first service information corresponds to a first service and the different service information corresponds to a different service. Specific examples are described below.

[0278] For example, the first service information is address information corresponding to a navigation service. The message 3122 in the user interface 310 shown in FIG. 3A-1, the message 342 in the user interface 310 shown in FIG. 3B, or the location control 352 in the user interface 340 shown in FIG. 3C is the first service information.

[0279] For example, the first service information is video information corresponding to a video service (e.g., playing a video). The message 414 in the user interface 410 shown in FIG. 4A-1 or the information (e.g., name 521) included in the user interface 510 shown in FIG. 5A is the first service information.

[0280] For example, the first service information is recipe information corresponding to a cooking service (e.g., cooking based on a recipe). Information included in the user interface 610 shown in FIG. 6A (e.g., title 611) is the first service information.

[0281] For example, the first service information is learning information corresponding to a learning service (e.g., practice questions). Information included in the user interface 710 shown in FIG. 7A-1 (e.g., practice 712, practice 713) is the first service information.

[0282] S103: The electronic device 100 receives a first user operation.

[0283] The first user operation type includes, but is not limited to, a touch operation performed on a display, a voice, a movement posture (e.g., a gesture), and an electroencephalogram. For example, the first user operation is an operation of shaking the electronic device 100. In another example, the first user operation is a sliding operation with a knuckle of a finger shown in (A) of FIG. 8. In another example, the first user operation is a sliding operation with two fingers shown in (B) of FIG. 8. In another example, the first user operation is a gesture operation shown in (C) of FIG. 8. The specific type of the first user operation is not limited in this application.

[0284] In some embodiments, the electronic device 100 may detect the first user action via a detection module shown in FIG. 1B.

[0285] In some embodiments, the electronic device 100 may detect a first user action via the sensor module 180 shown in Figure 2A. For a specific example, see the description of how the electronic device 100 may detect a user action via the sensor module 180 in Figure 2A.

[0286] In some embodiments, the electronic device 100 may train a fusion model that is used to recognize user intent, such as to perform S107.

[0287] In some other embodiments, the network device 300 trains a fusion model. For a description of training the fusion model, see the description of training the fusion model in FIG. 1B and training the interface analysis model and / or the intent analysis model. The details will not be described again here.

[0288] When the electronic device 100 receives the fusion model transmitted by the network device 300 before S103, the display method may include, but is not limited to, the following three steps after S103:

[0289] S104: The electronic device 100 sends a first request message to the network device 300.

[0290] In some embodiments, the first request message is used to request obtaining setting information of the fusion model.

[0291] S105: The network device 300 sends a first configuration message to the electronic device 100.

[0292] In some embodiments, the first configuration message includes configuration information for the fusion model.

[0293] S106: The electronic device 100 updates the fusion model based on the first configuration message.

[0294] In another case where the electronic device 100 does not receive the fusion model transmitted by the network device 300 before the process of S103, the electronic device 100 may request to obtain the fusion model from the network device. The specific process is similar to the above-mentioned steps S104 to S106. The details will not be described again here.

[0295] S107: The electronic device 100 recognizes the first interface based on the fusion model, and determines intention information corresponding to the first service information.

[0296] In some embodiments, the electronic device 100 may use the interface content in the first interface as an input for the fusion model to obtain an output, i.e., intent information. Some examples of intent information are provided below.

[0297] For example, the first interface is the user interface 310 shown in FIG. 3A-1 or the user interface 340 shown in FIG. 3B, the message 3122 in the user interface 310 or the message 342 in the user interface 340 is the first service information, and the first service information is address information indicating a geographical location named "Beijing Railway Station". In this case, the intention information corresponding to the first service information is to navigate to the geographical location named "Beijing Railway Station".

[0298] For example, the first interface is the user interface 350 shown in Fig. 3C, the location control 352 in the user interface 350 is the first service information, and the first service information is address information indicating a place named "Capital Museum." In this case, the intent information corresponding to the first service information is to navigate to the place named "Capital Museum."

[0299] For example, the first interface may be the user interface 410 shown in FIG. 4A-1, the message 414 in the user interface 410 may be the first service information, and the first service information may indicate a video named "My Day". In this case, the intention information corresponding to the first service information is to play the video named "My Day".

[0300] For example, the first interface may be the user interface 510 shown in Fig. 5A, and the information included in the user interface 510 (e.g., name 521) may be first service information, and the first service information may indicate a movie named "Movie 1". In this case, the intention information corresponding to the first service information is to play the movie named "Movie 1".

[0301] For example, the first interface may be the user interface 610 shown in Fig. 6A, the information (e.g., title 611) included in the user interface 610 may be the first service information, and the first service information may indicate a recipe named "crispy pork belly." In this case, the intention information corresponding to the first service information is to cook a dish corresponding to the recipe.

[0302] For example, the first interface may be the user interface 710 shown in FIG. 7A-1, and information included in the user interface 710 (e.g., exercise 712 and exercise 713) may be first service information, and the first service information may indicate one or more exercises (e.g., at least one exercise included in a test paper named "English test paper", e.g., exercise 712 and exercise 713). In this case, the intention information corresponding to the first service information is to practice the one or more exercises.

[0303] S108: The electronic device 100 transmits instruction information to the electronic device 200 based on the intention information.

[0304] In some embodiments, the electronic device 100 may perform an intention operation based on the intention information and send multimedia data corresponding to the performed intention operation to the electronic device 200. The instruction information may indicate the electronic device 200 to output the multimedia data. For example, in FIG. 1B , the intention analysis module of the electronic device 100 sends the intention information to the intention trigger module, which performs the intention operation based on the intention information and sends audio and video streams corresponding to the intention operation to a display module of the electronic device 200 for output.

[0305] In some other embodiments, the instructional information sent by electronic device 100 to electronic device 200 may include the intent information, and the instructional information may indicate to electronic device 200 to implement the intent information. For example, in FIG. 1C , the intent analysis module of electronic device 100 sends the intent information to the intent trigger module of electronic device 200.

[0306] S109: The electronic device 200 outputs the multimedia data.

[0307] In some embodiments, when electronic device 200 receives the multimedia data and the instruction information transmitted by electronic device 100, it may output the multimedia data based on the instruction information, such as in the embodiment shown in FIG. 1B.

[0308] In some other embodiments, when receiving the instruction information sent by the electronic device 100, if the instruction information includes intention information, the electronic device 200 can perform an intention operation based on the intention information and output multimedia data corresponding to the performed intention operation, such as the embodiment shown in FIG. 1C. In some embodiments, the intention operation corresponds to first service information in the first interface. In some embodiments, the electronic device 200 outputting multimedia data corresponding to the performed intention operation may also be referred to as the electronic device 200 outputting multimedia data corresponding to the first service information.

[0309] Below, some examples of intentional operations are shown.

[0310] For example, the first interface is the user interface 310 shown in FIG. 3A-1 and FIG. 3A-2, or the user interface 340 shown in FIG. 3B, and the message 3122 in the user interface 310 or the message 342 in the user interface 340 is the first service information, and the first service information is address information indicating a geographical location named "Beijing Railway Station". In this case, the intention operation corresponding to the first service information is to set a destination to the location information of the geographical location "Beijing Railway Station" and perform navigation. For multimedia data corresponding to the intention operation and output by the electronic device 200, see the multimedia data of the user interface 330 shown in FIG. 3A-2. For a description of a specific scenario, see the description of FIG. 3A-1 and FIG. 3A-2 or FIG. 3B.

[0311] For example, the first interface is the user interface 350 shown in FIG. 3C, the location control 352 in the user interface 350 is the first service information, and the first service information is address information indicating a place named "Capital Museum". In this case, the intention operation corresponding to the first service information is to set a destination to the location information of the place "Capital Museum" and perform navigation. The multimedia data output by the electronic device 200 corresponding to the intention operation is similar to the user interface 330 shown in FIG. 3A-2, with a difference in the navigation destination. For a description of the specific scenario, please refer to the description of FIG. 3C.

[0312] For example, the first interface may be the user interface 410 shown in FIG. 4A-1, the message 414 in the user interface 410 may be the first service information, and the first service information may indicate a video named "My Day". In this case, the intention operation corresponding to the first service information is to play the video named "My Day". For multimedia data corresponding to the intention operation and output by the electronic device 200, see the multimedia data of the user interface 430 shown in FIG. 4B-2. For a description of a specific scenario, see the description of FIG. 4B-1 and FIG. 4B-2.

[0313] For example, the first interface is the user interface 510 shown in FIG. 5A, and the information (e.g., name 521) included in the user interface 510 is the first service information, and the first service information may indicate a movie named "Movie 1". In this case, the intention operation corresponding to the first service information is to play the movie named "Movie 1". For the multimedia data corresponding to the intention operation and output by the electronic device 200, see the multimedia data of the user interface 520 shown in FIG. 5B. For the description of a specific scenario, see the description of FIG. 5A and FIG. 5B.

[0314] For example, the first interface is the user interface 610 shown in FIG. 6A, and the information (e.g., title 611) included in the user interface 610 is the first service information, and the first service information may indicate a recipe named "crispy pork belly". In this case, the intention operation corresponding to the first service information is to work according to the recipe. For multimedia data corresponding to the intention operation and output by the electronic device 200, see the multimedia data of the user interface 630 shown in FIG. 6B. For a description of a specific scenario, see the description of FIG. 6A and FIG. 6B.

[0315] For example, the first interface may be the user interface 710 shown in FIG. 7A-1, and the information included in the user interface 710 (e.g., exercise 712 and exercise 713) may be first service information, and the first service information may indicate one or more exercises (e.g., at least one exercise included in a test paper named "English test paper", e.g., exercise 712 and exercise 713). In this case, the intention operation corresponding to the first service information is to display questions (not display answers) in one or more exercises. For multimedia data corresponding to the intention operation and output by the electronic device 200, see the multimedia data of the user interface 730 shown in FIG. 7A-2. For a description of a specific scenario, see the description of FIG. 7A-1 and FIG. 7A-2.

[0316] This is not limited to the above example. In another embodiment, if the first interface does not include the first service information, the electronic device 100 cannot recognize the intention information corresponding to the first service information, so it does not send the instruction information to the electronic device 200, and the electronic device 200 does not perform the intention operation corresponding to the first service information. For example, the electronic device 100 and the electronic device 200 continue to display the current interface without changing it. This is not limited to the above. Alternatively, the electronic device 100 may display prompt information, that there is no service that can be currently transferred. For example, the user interface 410 (first interface) shown in FIG. 4A-1 includes only the message 411 and the message 413, and does not include the message 412 (address information) and the message 414 (video information). In this case, the electronic device 100 and the electronic device 200 may continue to display the current interface without changing it.

[0317] For examples of the display method shown in FIG. 9, see FIGS. 3A-1 to 3C, 4A-1 to 4C-2, 5A, 5B, 6A, 6B, and 7A-1 to 7B-2.

[0318] In the method shown in Fig. 9, the electronic device 100 may perform intent recognition based on the currently displayed user interface when receiving a first user operation, and the electronic device 200 implements the recognized intent. In this way, the user does not need to manually trigger the implementation of the intent. This reduces the user's operations and implements a more efficient and convenient interaction.

[0319] This is not limited to the example of FIG. 9. In other embodiments, in S107, the electronic device 100 may recognize a first interface to obtain an interface recognition result. Optionally, the electronic device 100 may obtain an interface recognition result based on an interface analysis model. For details, see the description of the interface analysis module in FIG. 1B. Optionally, the manner in which the electronic device 100 obtains the interface analysis model is similar to the manner in which the electronic device 100 obtains the fusion model shown in FIG. 9. The electronic device 100 may perform intention recognition based on the interface recognition result to obtain intention information. Optionally, the electronic device 100 may obtain intention information based on the intention analysis model. For details, see the description of the intention analysis module in FIG. 1B. Optionally, the manner in which the electronic device 100 obtains the intention analysis model is similar to the manner in which the electronic device 100 obtains the fusion model shown in FIG. 9.

[0320] This is not limited to the example of FIG. 9. In some other embodiments, the electronic device 100 may not perform S107 and S108. After receiving the first user operation, the electronic device 100 may recognize the first interface to obtain an interface recognition result, and send the interface recognition result and instruction information to the electronic device 200. The instruction information may indicate the electronic device 200 to implement the intention information corresponding to the interface recognition result. The electronic device 200 may perform intention recognition based on the interface recognition result, obtain the intention information, perform an intention operation based on the intention information, and output multimedia data corresponding to the performed intention operation. For example, the electronic device 100 includes a detection module and an interface analysis module shown in FIG. 1B, and the electronic device 200 includes an intention analysis module and an intention trigger module shown in FIG. 1B. Optionally, the electronic device 200 may obtain the intention information based on an intention analysis model. Optionally, the manner in which the electronic device 200 obtains the intention analysis model is similar to the manner in which the electronic device 100 obtains the fusion model shown in FIG. 9.

[0321] This is not limited to the example of FIG. 9. In some other embodiments, the electronic device 100 may not perform S107 and S108. After receiving the first user operation, the electronic device 100 may send the interface content displayed by the electronic device 100 and the instruction information to the electronic device 200. The instruction information may indicate the electronic device 200 to implement the intention information corresponding to the interface content. The electronic device 200 may perform S107 of FIG. 9 to obtain the intention information, perform the intention operation based on the intention information, and output multimedia data corresponding to the performed intention operation. For example, the electronic device 100 includes a detection module shown in FIG. 1B, and the electronic device 200 includes an interface analysis module, an intention analysis module, and an intention trigger module shown in FIG. 1B. Optionally, the electronic device 200 may obtain an interface recognition result based on the interface analysis model. Optionally, the electronic device 200 may obtain the intention information based on the intention analysis model. Optionally, the manner in which the electronic device 200 obtains the interface analysis model and / or the intention analysis model is similar to the manner in which the electronic device 100 obtains the fusion model shown in Fig. 9. Optionally, the electronic device 200 may obtain intention information based on the fusion model and the interface content. Optionally, the manner in which the electronic device 200 obtains the fusion model is similar to the manner in which the electronic device 100 obtains the fusion model shown in Fig. 9.

[0322] 10 is a schematic flow chart of another display method of this application. The first device in this method may be the aforementioned electronic device 100, and the second device in this method may be the aforementioned electronic device 200. The method may include, but is not limited to, the following steps:

[0323] S201: The electronic device displays a first interface.

[0324] In some embodiments, the first interface includes first information, and the first information is related to a first service. For an example of the first information, see the example of the first service information in S102 of FIG. 9.

[0325] S202: A first device receives a first user operation.

[0326] In some embodiments, S202 is similar to S103 in Figure 9. For details, see the description of S103 in Figure 9.

[0327] S203: In response to a first user operation, the first device recognizes a first interface and determines intention information.

[0328] In some embodiments, the intent information indicates to execute first instructions, the first instructions being used to implement a first service.

[0329] In some other embodiments, the first instruction is obtained by parsing the intent information. In some other embodiments, the first instruction is included in the intent information.

[0330] In some embodiments, the intention information includes first information. For example, the first information is information indicating a first location, and the intention information indicates performing navigation to the first location. In some embodiments, the intention information includes information related to the first information. For example, the first information is information indicating a first video. A manner of playing the first video (e.g., a playback source of the first video) may be obtained based on the first information, and the intention information indicates to play the first video in the obtained manner of playing the first video.

[0331] In some embodiments, see the description of S107 in FIG. 9 for a description of recognizing the first information by the first device to determine the intent information.

[0332] S204: The first device sends the intention information to the second device.

[0333] S205: The second device executes the first instruction based on the intention information to generate second information.

[0334] In some embodiments, executing the first instruction by the second device may correspond to performing the intention operation described above. For examples of the intention operations, see the intention operations illustrated in FIG. 9.

[0335] In some embodiments, the second information is multimedia data generated by executing the first instructions, such as audio data, video data, or image data.

[0336] S206: The second device displays a second interface based on the second information.

[0337] In some embodiments, the second device may output the second information, for example, play audio data included in the second information, display image data included in the second information, or play video data included in the second information.

[0338] In some embodiments, for an example of the second device displaying a second interface, see the example of the electronic device 200 outputting multimedia data corresponding to the intended operation in the description of the intended operation shown in FIG. 9 .

[0339] In some embodiments, the first information is information indicating a first location. For example, the first information is a message 3122 in the user interface 310 shown in FIG. 3A-1. In this case, the first location indicated by the message 3122 is the geographic location "Beijing Railway Station". In another example, the first information is a message 342 in the user interface 340 shown in FIG. 3B or a location control 352 in the user interface 350 shown in FIG. 3C. In this case, the first location indicated by the location control 352 is a place called "Capital Museum". The first service is a navigation service. The second information is display information generated by performing a navigation operation on the first location. For example, the second information is multimedia data generated by performing navigation by setting a destination to the location information of the geographic location "Beijing Railway Station". In this case, the second interface displayed by the second device based on the second information is the user interface 330 shown in FIG. 3A-2. In another example, the second information is multimedia data generated by performing navigation with a destination set to location information of a place such as "Capital Museum." For a description of a particular scenario, see the description of FIG. 3A-1 and FIG. 3A-2, FIG. 3B, or FIG. 3C.

[0340] In some other embodiments, the first information is information indicating a first video. For example, the first information is message 414 in user interface 410 shown in FIG. 4A-1. In this case, the name of the first video indicated by message 414 is "My Day". In another example, the first information is information (e.g., name 521) included in user interface 510 shown in FIG. 5A, for example. In this case, the name of the first video indicated by the information is "Movie 1". The first service is a navigation service. The second information is display information generated by playing the first video. For example, the second information is multimedia data generated by playing the video "My Day". In this case, the second interface displayed by the second device based on the second information is user interface 430 shown in FIG. 4B-2. For example, the second information is multimedia data generated by playing the video "Movie 1". In this case, the second interface displayed by the second device based on the second information is user interface 520 shown in Figure 5B. For a description of the particular scenario, see the description of Figures 4B-1 and 4B-2, or Figures 5A and 5B.

[0341] In some other embodiments, the first information is information indicative of a first recipe, such as information (such as title 611) included in the user interface 610 shown in FIG. 6A. In this case, the name of the first recipe indicated by the information is "crispy pork belly". The first service is a cooking service. The second information is display information generated by implementing a cooking service corresponding to the first recipe, such as multimedia data generated by working on the recipe "crispy pork belly". In this case, the second interface displayed by the second device based on the second information is the user interface 630 shown in FIG. 6B. See the description of FIGS. 6A and 6B for a description of the particular scenario.

[0342] In some other embodiments, the first information is information indicating a first question and an answer to the first question, for example, an exercise 712 in the user interface 710 shown in FIG. 7A-1, where the exercise 712 includes a question 712A and an answer 712B. In this case, the first service is a test paper generation service. In this application, an example is used in which the test paper includes at least one question and does not include an answer. The second interface includes the first question but does not include an answer to the first question. For example, the second interface is the user interface 730 shown in FIG. 7A-2. The user interface 730 includes the question 712A (question information 733 in the user interface 730) but does not include the answer 712B. For a description of the specific scenario, see the description of FIG. 7A-1 and FIG. 7A-2.

[0343] In some embodiments, the first interface further includes third information, and the third information is related to a second service. The third information and the description of the second service are similar to the first information and the description of the first service. S203 specifically includes the first device recognizing the first information to determine fourth information, recognizing the third information to determine fifth information, and determining that the intention information is the fourth information from the fourth information and the fifth information according to a first preset rule. The fourth information indicates to execute a first instruction, and the fifth information indicates to execute a second instruction, and the second instruction is used to implement the second service. The description of the second instruction is similar to the description of the first instruction.

[0344] Optionally, the first preset rule may include that the device type of the second device is the preset device type, which may be understood as the first device may determine the intention information to be implemented based on the device type of the connected second device. For example, in the above-mentioned scenario 2, the first interface is a chat interface, the first information and the third information are the message 412 and the message 414 in the user interface 410 shown in FIG. 4A-1, respectively, the first information is location information, and the third information is video information. In this case, the first service corresponding to the first information is a navigation service, the fourth information indicates to perform navigation to the geographical location "Beijing Railway Station", the second service corresponding to the third information is a video playback service, and the fifth information indicates to play a video named "My Day". The first device is the electronic device 100 (smartphone), and the second device is the electronic device 200. If the second device is an on-board computer, the first device may determine that the intention information is the fourth information. For example scenarios, see Figures 4A-1 and 4A-2. If the second device is a smart TV, the first device may determine that the intent information is the fifth information. For example scenarios, see Figures 4B-1 and 4B-2.

[0345] Optionally, the first preset rule may include that the services supported by the second device include a first service. For example, the first service is a navigation service. If the second device is a device that has a map application installed and can perform a navigation service based on the map application, the first device may determine that the intent information is the first information.

[0346] Optionally, the first pre-set rule is that the priority of the first service is higher than the priority of the second service.

[0347] Optionally, the first information and the third information are instant messaging messages, and the first preset rule includes that a receiving time of the first information is later than a receiving time of the third information. For example, in the above-mentioned scenario 2, the first interface is a chat interface, and the first information and the third information are message 412 and message 414, respectively, in user interface 410 shown in FIG. 4A-1. Because message 414 is received later, the first device may determine that the intent information is fifth information corresponding to message 414, and the fifth information indicates to play a video named "My Day". See FIG. 4B-1 and FIG. 4B-2 for an example scenario.

[0348] The method shown in Fig. 10 is, for example, applied to the communication system 10 shown in Fig. 1C, where the first device is the electronic device 100 and the second device is the electronic device 200. For details, see the description of Fig. 1C.

[0349] 11 is a schematic flowchart of yet another display method according to an embodiment of this application. The first device in this method may be the aforementioned electronic device 100, and the second device in this method may be the aforementioned electronic device 200. The method may include, but is not limited to, the following steps:

[0350] S301: The electronic device displays a first interface.

[0351] S302: A first device receives a first user operation.

[0352] S303: In response to a first user operation, the first device recognizes a first interface and determines intent information.

[0353] S301 to S303 correspond to S201 to S203 in Fig. 10. For details, see the description of S201 to S203 in Fig. 10.

[0354] S304: The first device executes a first instruction based on the intention information to generate second information.

[0355] In some embodiments, S304 is similar to S205 of Fig. 10. The difference is that the device performing S304 is the first device instead of the second device.

[0356] S305: The first device transmits the second information to the second device.

[0357] S306: The second device displays a second interface based on the second information.

[0358] S306 corresponds to S206 in Figure 10. For details, see the description of S206 in Figure 10.

[0359] The example of Figure 11 is similar to the example of Figure 10, except that in Figure 11, the device that executes the first instruction to generate the second information is the first device, not the second device. For details, see the example of Figure 10.

[0360] The method shown in Fig. 11 is, for example, applied to the communication system 10 shown in Fig. 1B, where the first device is the electronic device 100 and the second device is the electronic device 200. For details, see the description of Fig. 1B.

[0361] This is not limited to the case of FIG. 10 and FIG. 11. In some other embodiments, the device that recognizes the first interface and determines the intention information may be the second device instead of the first device. For example, the first device transmits multimedia data (e.g., image data) related to the first interface to the second device in response to the first user operation. The second device performs intention recognition based on the received data. The specific process is similar to the above process in which the first device recognizes the first interface and determines the intention information. The details will not be described again here.

[0362] When any one of the above modules or units is implemented by software, the software exists in the form of computer program instructions and is stored in a memory. The processor may be configured to execute the program instructions to implement the above method steps. The processor may be a computational device that operates the software, i.e., at least one of various computing devices that operate the software, such as, but not limited to, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), an artificial intelligence processor, etc. Each computing device may include one or more cores that are used to execute the software instructions to perform an operation or process. The processor may be an independent semiconductor chip or may be integrated into the semiconductor chip with another circuit. For example, the processor may constitute a system-on-a-chip (SoC) with other circuits (e.g., codec circuits, hardware acceleration circuits, or various bus and interface circuits). Alternatively, the processor may be integrated into the ASIC as an embedded processor of the ASIC. The ASIC integrated with the processor may be packaged separately or with other circuits. In addition to a core for executing software instructions to perform an operation or process, a processor may further include necessary hardware accelerators, such as a field programmable gate array (FPGA), a programmable logic device (PLD), or logic circuitry for implementing special-purpose logic operations.

[0363] When the above modules or units are implemented by hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, dedicated digital circuitry, hardware accelerator, or non-integrated discrete devices. The hardware may run the necessary software or may perform the steps of the above method without software.

[0364] Those skilled in the art may understand that all or part of the steps of the method in the above embodiments may be implemented by a computer program instructing relevant hardware. The computer program may be stored in a computer-readable storage medium. When the computer program is executed, the steps in the above method embodiments are executed. The above storage medium includes any medium capable of storing computer program code, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. 1. A display method applied to a first device, the first device being connected to a second device, displaying a first interface, the first interface including first information, the first information relating to a first service; Receiving a first user operation; recognizing the first interface to determine intent information in response to the first user operation, the intent information indicating to execute a first instruction, the first instruction being used to implement the first service; and sending the intent information to the second device, wherein the intent information is used by the second device to execute the first instructions to generate second information, and the second information is used by the second device to display a second interface.

2. the first interface further includes third information, the third information relating to a second service; Recognizing the first interface to determine intent information includes: determining fourth information from recognizing the first information and determining fifth information from recognizing the third information, the fourth information indicating to execute the first instructions and the fifth information indicating to execute second instructions, the second instructions being used to implement the second service; 2. The method of claim 1, determining that the intent information is the fourth information from the fourth information and the fifth information according to a first preset rule, the first preset rule including at least one of: a device type of the second device is a preset device type; services supported by the second device include the first service; and a priority of the first service is higher than a priority of the second service.

3. 3. The method of claim 2, wherein the first information is location information, the first service is a navigation service, the second service is different from the first service, and the first preset rule includes that a device type of the second device is the preset device type, and the preset device type is an in-vehicle device.

4. The method according to any one of claims 1 to 3, wherein the first information is information indicating a first location, the first service is the navigation service, and the second information is display information generated by performing a navigation operation on the first location.

5. The method according to claim 1 or 2, wherein the first information is information indicating a first video, the first service is a video playback service, and the second information is display information generated by playing the first video.

6. 3. The method according to claim 1 or 2, wherein the first information is information indicating a first recipe, the first service is a cooking service, and the second information is display information generated to implement the cooking service corresponding to the first recipe.

7. 3. The method of claim 1 or 2, wherein the first information is information indicating a first question and an answer to the first question, the first service is a test paper generation service, and the second interface includes the first question but does not include the answer to the first question.

8. 1. A display method applied to a first device, the first device being connected to a second device, displaying a first interface, the first interface including first information, the first information relating to a first service; Receiving a first user operation; recognizing the first interface and determining intent information in response to the first user operation; executing first instructions based on the intent information to generate second information, the first instructions being used to implement the first service; and transmitting the second information to the second device, the second information being used by the second device to display a second interface.

9. the first interface further includes third information, the third information relating to a second service; Recognizing the first interface to determine intent information includes: recognizing the first information to determine fourth information and recognizing the third information to determine fifth information, the fourth information indicating execution of the first instructions and the fifth information indicating execution of second instructions, the second instructions being used to implement a second service; and 9. The method of claim 8, comprising: determining, from the fourth information and the fifth information, that the intention information is the fourth information according to a first preset rule, the first preset rule comprising: a device type of the second device being a preset device type; and / or a priority of the first service being higher than a priority of the second service.

10. A display method applied to a second device, the second device being connected to a first device, receiving intent information transmitted by the first device, the intent information being determined by recognizing a first interface displayed when the first device receives a first user action, the first interface including first information, the first information relating to a first service; executing first instructions based on the intent information to generate second information, the first instructions being used to implement the first service; and and displaying a second interface based on the second information.

11. A display method applied to a second device, the second device being connected to a first device, receiving first information transmitted by the first device, the first information being information generated by executing first instructions, the first instructions being used to implement a first service, the first instructions being instructions to be executed as indicated by intent information, the intent information being determined by recognizing a first interface displayed when the first device receives a first user action, the first interface including second information, the second information being related to the first service; and displaying a second interface based on the first information.

12. An electronic device comprising a transceiver, a processor and a memory, the memory configured to store a computer program, the processor invoking the computer program to perform the method of any one of claims 1 to 11.

13. A computer storage medium storing a computer program, the computer program implementing the method according to any one of claims 1 to 11 when executed by a processor.

Images