Electronic equipment and data processing method

Abstract

The embodiment of the invention relates to electronic equipment and a data processing method, and the electronic equipment comprises a communicator which is configured to receive an equipment list obtaining request sent by a HomeKit client after establishing connection with the HomeKit client based on a HomeKit protocol; the processor is configured to determine equipment information of the electronic equipment and equipment information of each piece of target equipment based on the equipment list acquisition request; target information is generated based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder is used for indicating one unknown device to be accessed, and the device information corresponding to each device placeholder is null; and the communicator is further configured to send the target information to the HomeKit client, so that the HomeKit client displays the equipment list based on the target information.

Description

Electronic device and data processing method

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the application number 202211477456.8 filed with the China Patent Office on November 23, 2022, and the application number 202211658353.1 filed with the China Patent Office on December 22, 2022, and the application number 202211714328.0 filed with the China Patent Office on December 29, 2022, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to the field of terminal technology, and in particular, to an electronic device and a data processing method for the electronic device. Background Art

[0004] With the development of smart home technology, users can control all HomeKit-enabled devices in their home, including lights, locks, thermostats, smart plugs, and other devices, through the HomeKit app on their devices.

[0005] Electronic devices equipped with the HomeKit protocol can also be discovered and controlled by the HomeKit app. For example, after device A is discovered and connected to the HomeKit app, the user can control device A through the HomeKit app. The HomeKit app can control not only device A, but also device B, which is connected to it, and even device C, which is connected to B. This allows users to control multiple electronic devices through the HomeKit app, providing great convenience.

[0006] According to the HomeKit protocol, the HomeKit app acts as the control end. The list of controlled devices displayed on the HomeKit app is obtained during initialization after the HomeKit app connects to electronic device A. However, once the device list is initialized, additional devices that subsequently connect to electronic device A cannot be added to the list. In other words, after the HomeKit app connects to electronic device A and initializes, the device list displayed on the HomeKit app cannot be dynamically updated.

[0007] Summary of the Invention

[0008] In a first aspect, according to an embodiment of the present application, an electronic device includes: a communicator, configured to: after establishing a connection with a HomeKit client based on the HomeKit protocol, receive a device list acquisition request sent by the HomeKit client, the device list acquisition request being used to indicate acquisition of device information of each target device connected to the electronic device, the device information including an identification of the device; a processor, configured to: based on the device list acquisition request, determine the device information of the electronic device and the device information of each target device, the device information including an identification of the corresponding device; generate target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder being used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder being empty; the communicator is further configured to: send the target information to the HomeKit client so that the HomeKit client displays a device list based on the target information, the device list including the identification of the electronic device and the identification of each target device.

[0009] In a second aspect, according to an embodiment of the present application, a data processing method is provided, which is applied to an electronic device, comprising: after establishing a connection with a HomeKit client based on the HomeKit protocol, receiving a device list acquisition request sent by the HomeKit client, the device list acquisition request being used to indicate acquisition of device information of each target device connected to the electronic device, the device information including an identification of the device; based on the device list acquisition request, determining the device information of the electronic device and the device information of each target device, the device information including an identification of the corresponding device; generating target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder being used to indicate an unknown device to be connected, the device information corresponding to each device placeholder being empty; sending the target information to the HomeKit client so that the HomeKit client displays a device list based on the target information, the device list including the identification of the electronic device and the identification of each target device.

[0010] In a third aspect, according to an embodiment of the present application, a computer-readable non-volatile storage medium is provided, comprising: a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, the device list acquisition method shown in the second aspect is implemented.

[0011] In a fourth aspect, according to an embodiment of the present application, a computer program product is provided, comprising: when the computer program product is run on a computer, the computer is enabled to implement the device list acquisition method shown in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG1 illustrates one of the operation scenarios between a control device and an electronic device according to some embodiments;

[0013] FIG2 is a block diagram of a hardware configuration of an electronic device 200 according to some embodiments;

[0014] FIG3 illustrates a second operation scenario between a control device and an electronic device according to some embodiments;

[0015] FIG4 illustrates a third operation scenario between a control device and an electronic device according to some embodiments;

[0016] FIG5 is a flowchart of a method for obtaining a device list according to some embodiments;

[0017] FIG6 is a second flow chart of a method for obtaining a device list according to some embodiments;

[0018] FIG7 is a third flow chart of a method for obtaining a device list according to some embodiments;

[0019] FIG8 is a block diagram of a structure between a mobile phone and a television according to some embodiments;

[0020] FIG9 is a fourth flow chart of a method for obtaining a device list according to some embodiments;

[0021] FIG10 is a fifth flow chart of a method for obtaining a device list according to some embodiments;

[0022] FIG11 illustrates an operation scenario between a display device and a control device according to some embodiments;

[0023] FIG12 is a block diagram of a hardware configuration of a control device 001 according to some embodiments;

[0024] FIG13 is a schematic diagram of a screen projection connection interface according to some embodiments;

[0025] FIG14 is a second schematic diagram of a screen projection connection interface according to some embodiments;

[0026] FIG15 is a flowchart of a screen projection processing method according to some embodiments;

[0027] FIG16 is a schematic diagram of a screen projection data flow according to some embodiments;

[0028] FIG17 is a schematic diagram of a screen projection data flow according to some embodiments;

[0029] FIG18 is a second flowchart of a screen projection processing method according to some embodiments;

[0030] FIG19 is a third flow chart of a screen projection processing method according to some embodiments;

[0031] FIG20 is a fourth flowchart of a screen projection processing method according to some embodiments;

[0032] FIG21 is a fifth flowchart of a screen projection processing method according to some embodiments;

[0033] FIG22 is a sixth flowchart of a screen projection processing method according to some embodiments;

[0034] FIG23 is a schematic diagram of software configuration in a display device according to some embodiments;

[0035] FIG24 is a schematic diagram showing an icon control interface of an application in a display device according to some embodiments;

[0036] FIG25 is a system framework diagram for performing screen projection processing according to some embodiments;

[0037] FIG26 is a schematic flow chart of a screen projection processing method according to some embodiments;

[0038] FIG27 is a schematic diagram of an interface of a display when the current playback state of the display device is in a target mode according to some embodiments;

[0039] FIG28 is a schematic diagram of an interface of a display when a display switch of the display is in an on state and the display displays non-audio data according to some embodiments;

[0040] FIG29 is a flowchart of another screen projection processing method according to some embodiments;

[0041] FIG30 is a schematic diagram of an interface when a display shows prompt information according to some embodiments;

[0042] FIG31 is a schematic diagram of a display interface when a display device enters an input page according to some embodiments;

[0043] FIG32 is a schematic diagram of a screen projection process of a display device according to some embodiments;

[0044] Figure 33 is a schematic diagram of the overall process of a screen projection processing method according to some embodiments. DETAILED DESCRIPTION

[0045] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0046] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0047] The electronic device provided in the embodiments of the present application can have various implementation forms, for example, it can be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, etc.

[0048] Figure 1 is a schematic diagram of an operation scenario between a control device and an electronic device according to an embodiment, wherein the HomeKit app is installed in the control device 100, and the electronic device 200 supports the HomeKit protocol. Therefore, after the control device 100 establishes a connection with the electronic device 200 through the HomeKit app, the user can control the electronic device 200 that supports the HomeKit protocol and other devices directly or indirectly connected to the electronic device 200 through the HomeKit app installed in the control device 100.

[0049] In some embodiments, the control device 100 can be an iOS device with the HomeKit app installed and running the iOS operating system, such as a handheld device like an iPhone or iPad. It can also be a device with the HomeKit app installed and running other operating systems. The control device 100 can connect to the same wired or wireless network as the electronic device 200 and establish a connection through device discovery. Users can control the device 100 by inputting user commands through voice input, control panel input, gesture input, and the like.

[0050] As shown in FIG2 , electronic device 200 includes at least one of a tuner and demodulator 210 , a communicator 220 , a detector 230 , an external device interface 240 , a processor 250 , a display 260 , an audio output interface 270 , a user interface 280 , an external memory, and a power supply.

[0051] In some embodiments, the processor includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0052] The display 260 includes a display screen component for presenting images, a driving component for driving image display, a component for receiving image signals output from a processor, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0053] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0054] Communicator 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. Electronic device 200 can use communicator 220 to send and receive control signals and data signals with external control device 100 or server 400.

[0055] The user interface 280 can be used to receive control signals from the control device 100 (e.g., an infrared remote control). It can also be used to directly receive user input commands and convert the commands into commands that the electronic device 200 can recognize and respond to. In this case, it can be called a user input interface.

[0056] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0057] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0058] The tuner-demodulator 210 receives broadcast television signals via a wired or wireless reception method, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0059] In some embodiments, the processor 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the processor 250 is located, such as an external set-top box.

[0060] Processor 250 controls the operation of the electronic device and responds to user operations through various software control programs stored in memory (internal memory or external memory). Processor 250 controls the overall operation of electronic device 200. For example, in response to receiving a user command to select a UI object to be displayed on display 260, processor 250 may perform operations related to the object selected by the user command.

[0061] In some embodiments, the processor includes a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), and at least one of a random access memory (RAM), a read-only memory (ROM), a first interface to an nth interface for input / output, a communication bus, etc.

[0062] RAM, also known as main memory, is the internal storage that directly exchanges data with the processor. It can be read and written at any time (except when refreshing) and is very fast. It is often used as a temporary data storage medium for operating systems or other running programs. The biggest difference between RAM and ROM is the volatility of the data; the stored data will be lost if the power is lost. RAM is used in computers and digital systems to temporarily store programs, data, and intermediate results. ROM operates in a non-destructive read mode and can only read information but not write it. Once written, information is fixed and will not be lost even if the power is cut off, hence the name fixed memory.

[0063] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific voice or gesture, and the user input interface may recognize the voice or gesture through a sensor to receive the user input command.

[0064] With the development of smart home technology, users can control all HomeKit-enabled devices in their home, including lights, locks, thermostats, smart plugs, and other devices, through the HomeKit app on their iOS devices.

[0065] Electronic devices equipped with the HomeKit protocol can also be discovered and controlled by the HomeKit app on iOS devices. For example, after device A is discovered and connected to the HomeKit app, the user can control device A through the HomeKit app on the iOS device. The HomeKit app can control not only device A, but also device B, which is connected to it, and even device C, which is connected to B. This allows users to control multiple electronic devices through the HomeKit app on their iOS device, providing great convenience.

[0066] According to the HomeKit protocol, the iOS device acts as the controller. The list of controlled devices displayed on the iOS device is obtained during initialization after the iOS device connects to electronic device A. However, once the device list is initialized, additional devices that subsequently connect directly or indirectly to electronic device A cannot be added to the list. In other words, after the iOS device connects to electronic device A and initializes, the device list displayed on the iOS device cannot be dynamically updated.

[0067] In order to solve the above technical problems, according to an embodiment of the present application, a device list acquisition method is provided, specifically a method for dynamically updating a device list by pre-setting device placeholders based on the HomeKit protocol, specifically, after establishing a connection with a HomeKit client based on the HomeKit protocol, receiving a device list acquisition request sent by the HomeKit client, the device list acquisition request is used to indicate the acquisition of device information of each target device connected to the electronic device, the device information including the device identification; based on the device list acquisition request, determining the device information of the electronic device and the device information of each target device, the device information including the identification of the corresponding device; generating target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder, each device placeholder being used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty; sending the target information to the HomeKit client so that the HomeKit client displays a device list based on the target information, the device list including the identification of the electronic device and the identification of each target device. In this way, by adding device placeholders to the target information, since each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty, not only will the user not be misled because the device list generated by the HomeKit client based on the target information includes the identifiers of some unconnected devices, but a place is also reserved in the device list for unknown devices to be connected. Therefore, when other devices subsequently establish a connection with the electronic device, a device placeholder in the target information can be updated with the device information of the other device to update the target information, and then the HomeKit client updates the device list based on the updated target information, so as to dynamically update the device list displayed on the HomeKit client.

[0068] According to an embodiment of the present application, an electronic device and a device list acquisition method are provided. The electronic device can implement the device list acquisition method according to the embodiment of the present application, or a functional module or functional entity within the electronic device can implement the device list acquisition method according to the embodiment of the present application. The electronic device includes: a communicator and a processor, corresponding to the communicator 220 and the processor 250 in FIG. 2 , respectively. The HomeKit client can be the control device 100 shown in the figure, or it can be a HomeKit app installed in the control device 100, and the specific method can be determined based on actual conditions.

[0069] In some embodiments, in order to enable the HomeKit client to dynamically update the device list of devices controlled by the electronic device, after establishing a connection with the HomeKit client based on the HomeKit protocol, the communicator receives a device list acquisition request sent by the HomeKit client to instruct the acquisition of device information (the device information includes the identification of the device) of each target device connected to the electronic device; the processor determines the device information of the electronic device and the device information of each target device based on the device list acquisition request, and the device information includes the identification of the corresponding device; then generates target information based on the device information of the electronic device, the device information of each target device and at least one device placeholder; the communicator sends the target information to the HomeKit client so that the HomeKit client displays a device list based on the target information, and the device list includes the identification of the electronic device and the identification of each target device. In an embodiment of the present application, by adding device placeholders to the target information, since each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty, not only will the user not be misled because the device list generated by the HomeKit client based on the target information includes the identifiers of some unconnected devices, but a position is also reserved in the device list for the unknown devices to be connected. Therefore, when other devices subsequently establish a connection with the electronic device, a device placeholder in the target information can be updated with the device information of the other device to update the target information, and then the HomeKit client updates the device list based on the updated target information to dynamically update the device list displayed on the HomeKit client.

[0070] Each device placeholder indicates an unknown device to be connected. It can be understood as a location reserved for a device in the target information. This means that a location is reserved for the device in the device list generated by the HomeKit client based on the target information. A device to be connected can be any device that is directly or indirectly connected to the electronic device.

[0071] Among them, HomeKit client devices are used to discover and control electronic devices and various other devices connected to the electronic devices (directly or indirectly). After the HomeKit client and the electronic device are connected to the same wired or wireless network, the HomeKit client performs device discovery. Once the HomeKit client discovers the electronic device, it establishes a connection with the electronic device. The specific process can be referred to in existing related technologies and will not be repeated here.

[0072] Among them, after the HomeKit client establishes a connection with the electronic device, it sends a device list acquisition request to the electronic device, and then receives the target information sent by the electronic device. The HomeKit client generates a device list based on the target information. Specifically, it generates a device list based on the identifier of the electronic device in the target information and the identifier of each target device. Since the device information corresponding to the device placeholder is empty, the device list does not include the identifier of the unknown device to be connected indicated by each device placeholder. In other words, the HomeKit client displays the identifier of the device whose device information in the target information is not empty in the device list, ignores the device placeholder whose device information in the target information is empty, and does not display it in the device list.

[0073] A direct connection between an electronic device and device A refers to a direct wired or wireless connection between the electronic device and device A, without any other connection. An indirect connection between an electronic device and device B refers to a connection between the electronic device and device B via device C. This can also be understood as a direct connection between the electronic device and device C, and a direct connection between device C and device B. Devices A and C can be referred to as the inherent connection of the electronic devices, and device B can be referred to as an extended connection of the electronic devices.

[0074] In some embodiments, when the electronic device detects that a connection (direct connection or indirect connection) has been established with another device, the identifier of the other device can be dynamically updated to the device list so that the HomeKit client can control the newly connected other device.

[0075] In some embodiments, after sending the target information to the HomeKit client, the processor detects that the first device establishes a connection with the electronic device; then updates one of the device placeholders in the target information to the device information of the first device to update the target information; the communicator sends the updated target information to the HomeKit client so that the HomeKit client updates the device list based on the updated target information, and the updated device list includes the identifier of the electronic device, the identifiers of each target device, and the identifier of the first device.

[0076] In an embodiment of the present application, when a first device establishes a connection with an electronic device, a device placeholder in the target information is replaced with the device information of the first device to obtain updated target information, and then the updated target information is sent to the HomeKit client, so that the HomeKit client adds the identifier of the first device to the device list based on the information of the first device in the updated target information, thereby dynamically updating the device list displayed by the HomeKit client when a new device is connected to the electronic device. In this process, there is no need for the electronic device to disconnect from the HomeKit client and then reconnect. Instead, the dynamic update of the device list displayed by the HomeKit client is achieved while the electronic device and the HomeKit client remain connected.

[0077] In an embodiment of the present application, when an electronic device detects that it is disconnected from a target device (directly or indirectly), the identifier of the target device can be dynamically deleted from the device list to prevent the user from controlling the disconnected target device through the HomeKit client.

[0078] In some embodiments, after sending the target information to the HomeKit client, the processor detects that a second device is disconnected from the electronic device, the second device being any one of the target devices; then the device information of the second device in the target information is updated to a placeholder for the device to update the target information; then the communicator sends the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information, the updated device list including the identifier of the electronic device and the identifiers of each device among the target devices except the second device.

[0079] In some embodiments, after sending the target information to the HomeKit client, the processor detects that a second device is disconnected from the electronic device, the second device being any one of the target devices; then the device information of the second device is deleted from the target information to update the target information; then the communicator sends the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information, the updated device list including the identifier of the electronic device and the identifiers of each device among the target devices except the second device.

[0080] In an embodiment of the present application, when the second device is disconnected from the electronic device, the device information of the second device in the target information can be replaced with a device placeholder, or the device information of the second device can be deleted from the target information to obtain updated target information, and then the updated target information is sent to the HomeKit client, so that the HomeKit client deletes the identifier of the second device from the device list based on the updated target information that does not include the device information of the second device, thereby dynamically updating the device list displayed by the HomeKit client when an already connected device connects to the electronic device. In this process, there is no need for the electronic device to disconnect from the HomeKit client and then reconnect, but the dynamic update of the device list displayed by the HomeKit client is achieved while the electronic device remains connected to the HomeKit client.

[0081] In the embodiment of the present application, the number of devices directly connected to the electronic device in each target device is not limited, nor is the number of devices indirectly connected to the electronic device limited. The specific number can be determined based on actual conditions and is not limited here.

[0082] In some embodiments, each target device includes: at least one third device, each third device is directly connected to the electronic device, that is, each target device is a third device, each target device is a device directly connected to the electronic device.

[0083] In some embodiments, as shown in FIG3 , the HomeKit client is a mobile phone 301 installed with the HomeKit app, the electronic device is a TV 201 that supports the HomeKit protocol, and the at least one third device is three third devices, each of which is an HDMI device 302 directly connected to the TV via an HDMI interface.

[0084] In some embodiments, the target devices include at least one third device and at least one fourth device; each third device is directly connected to the electronic device, and each fourth device is connected to the electronic device through a corresponding third device. That is, a target device can be a third device or a fourth device. The at least one fourth device can be connected to the electronic device through the same third device or through different third devices, depending on the actual situation.

[0085] In some embodiments, as shown in Figure 4, the HomeKit client is a mobile phone 301 installed with the HomeKit app, the electronic device is a TV 302 that supports the HomeKit protocol, the at least one third device is two third devices, and the at least one fourth device is two fourth devices. Each third device is an HDMI device 303 directly connected to the TV via an HDMI interface, and each fourth device is an HDMI device 304 directly connected to a third device via an HDMI interface.

[0086] In the embodiments of the present application, the connection method between the target device directly connected to the electronic device and the electronic device is not limited, nor is the connection method between the target device indirectly connected to the electronic device and the electronic device limited. It can be a wired interface connection, such as a High Definition Multimedia Interface (HDMI) connection, a USB interface connection, etc., or a wireless connection, such as a Bluetooth connection, a WiFi connection, etc., and the specific method can be determined according to actual conditions.

[0087] In some embodiments, the electronic device is connected to each third device via an HDMI interface; when each target device includes the at least one fourth device, each third device is connected to the corresponding third device via an HDMI interface.

[0088] HDMI is a digital video / audio interface technology, a specialized digital interface suitable for image transmission. It can transmit both audio and video signals simultaneously, with a maximum data transfer speed of 2.25GB / s, without the need for digital-to-analog or analog-to-digital conversion before signal transmission. HDMI can be combined with High-bandwidth Digital Content Protection (HDCP) to prevent unauthorized copying of copyrighted audio and video content. The additional space provided by HDMI can be used in future audio and video formats.

[0089] In an embodiment of the present application, when the electronic device and each target device are devices with a display screen, and when the transmitted data is audio and video data, connecting through an HDMI interface can improve the data transmission efficiency, protect the security of the data during the transmission process, and support audio and video formats that will be upgraded in the future.

[0090] In some embodiments, the device information also includes the type of the corresponding device; the device list also includes the type of the electronic device and the type of each target device.

[0091] In some embodiments, when the electronic device is connected to and initialized with the HomeKit client, the device list includes the identification and type of the electronic device, and the identification and type of each target device; when it is detected that the electronic device establishes a connection with the first device, the device list is updated to include the identification and type of the electronic device, the identification and type of each target device, and the identification and type of the first device; when it is detected that the electronic device is disconnected from the second device, the device list is updated to include the identification and type of the electronic device, and the identification and type of each target device other than the second device.

[0092] In an embodiment of the present application, the device information includes the type of the device, and the device identifier and type can be displayed in the device list. This allows the user to better understand the identifier and type of each device based on the device list displayed by the HomeKit client, thereby better controlling each device according to user needs.

[0093] In some embodiments, the sum of the number of the target devices and the number of the at least one device placeholder is equal to a first preset value; or, the number of the at least one device placeholder is equal to a second preset value.

[0094] The first preset value and the second preset value can be determined according to actual conditions.

[0095] In some embodiments, the first preset value is greater than the second preset value.

[0096] In an embodiment of the present application, the total number of devices controlled by the HomeKit client based on the connection with the electronic device (i.e., the total number of electronic devices, target devices, and at least one device placeholder, i.e., the sum of the first preset value and 1) can be set based on the properties of the electronic device, the total number of devices controlled by the HomeKit client input by the user, and other information. The number of at least one device placeholder (i.e., the second preset value) can also be set based on the properties of the electronic device, the total number of devices controlled by the HomeKit client input by the user, and other information. In this way, the number of device placeholders to be added in the target information can be set according to actual needs, which can improve the management efficiency of the device list.

[0097] In order to explain the present solution in more detail, the following will be explained in an exemplary manner with reference to Figures 5 to 10. It can be understood that the steps involved in Figures 5 to 10 may include more steps or fewer steps in actual implementation, and the order of these steps may also be different, so as to be able to implement the device list acquisition method provided in the embodiment of the present application. The execution subject of the device list acquisition method can be an electronic device, or it can be a functional module or functional entity in the electronic device that can implement the device list acquisition method. Moreover, the specific description of the device list acquisition method according to the embodiment of the present application can refer to the relevant description of the above-mentioned electronic device, and the same or similar technical effects can be achieved, which will not be repeated here.

[0098] FIG5 is a flowchart of the steps of implementing a method for obtaining a device list according to one or more embodiments of the present application. The method for obtaining a device list may include:

[0099] S501: After establishing a connection with a HomeKit client based on the HomeKit protocol, receive a device list acquisition request sent by the HomeKit client.

[0100] The device list acquisition request is used to instruct acquisition of device information of each target device connected to the electronic device, where the device information includes an identifier of the device.

[0101] S502: Determine the device information of the electronic device and the device information of each target device based on the device list acquisition request.

[0102] The device information includes the identifier of the corresponding device.

[0103] S503: Generate target information based on the device information of the electronic device, the device information of each target device, and at least one device placeholder.

[0104] Each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty.

[0105] S504: Send the target information to the HomeKit client, so that the HomeKit client displays a device list based on the target information.

[0106] The device list includes the identifier of the electronic device and the identifiers of each target device.

[0107] In an embodiment of the present application, by adding device placeholders to the target information, since each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty, not only will the user not be misled because the device list generated by the HomeKit client based on the target information includes the identifiers of some unconnected devices, but a position is also reserved in the device list for the unknown devices to be connected. Therefore, when other devices subsequently establish a connection with the electronic device, a device placeholder in the target information can be updated with the device information of the other device to update the target information, and then the HomeKit client updates the device list based on the updated target information to dynamically update the device list displayed on the HomeKit client.

[0108] In some embodiments, in combination with FIG. 5 , as shown in FIG. 6 , after the above S504 , the device list acquisition method according to the embodiment of the present application may further include the following S505 to S507 .

[0109] S505: Detect that the first device establishes a connection with the electronic device.

[0110] S506: Update a device placeholder in the target information to the device information of the first device, so as to update the target information.

[0111] S507: Send the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information.

[0112] The updated device list includes the identifier of the electronic device, the identifiers of each target device, and the identifier of the first device.

[0113] In some embodiments, in combination with FIG. 5 , as shown in FIG. 7 , after the above S504 , the device list acquisition method according to the embodiment of the present application may further include the following S508 to S510 .

[0114] S508: Detect that the second device is disconnected from the electronic device.

[0115] The second device is any one of the target devices.

[0116] S509: Update the device information of the second device in the target information to a placeholder for the device, so as to update the target information.

[0117] S510: Send the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information.

[0118] The updated device list includes the identifier of the electronic device and the identifiers of each target device except the second device.

[0119] In some embodiments, the respective target devices include: at least one third device; or, the respective target devices include: at least one third device and at least one fourth device; wherein each third device is directly connected to the electronic device, and each fourth device is connected to the electronic device through the corresponding third device.

[0120] In some embodiments, the electronic device is connected to each third device via an HDMI interface; when each target device includes the at least one fourth device, each third device is connected to the corresponding third device via an HDMI interface.

[0121] In some embodiments, the device information also includes the type of the corresponding device; the device list also includes the type of the electronic device and the type of each target device.

[0122] In some embodiments, the sum of the number of the target devices and the number of the at least one device placeholder is equal to a first preset value; or, the number of the at least one device placeholder is equal to a second preset value.

[0123] For example, consider an iOS device such as a mobile phone and an electronic device such as a TV, connected via an HDMI port. According to the HomeKit protocol, the device list displayed on the mobile phone is initialized after the phone and TV are connected. Once the list is initialized, new devices cannot be added. Although a TV has a fixed number of HDMI ports, this list is fixed after the phone and TV are connected. However, devices connected to the external device's HDMI port can also be connected to other HDMI devices, and this data is dynamic. Once a connection is established between the phone and TV, this data cannot be dynamically added to the mobile phone, resulting in the mobile phone being unable to dynamically display newly connected devices.

[0124] In some embodiments, as shown in FIG8 , a mobile phone serves as the control terminal, on which the HomeKit app is installed, and a television serves as the controlled terminal supporting the HomeKit protocol. The system comprises a HomeKit proxy module 61, a data processing module 62, and a device management module 63. The HomeKit proxy module 61 is responsible for establishing a network connection and data communication with the mobile phone. Receive control instructions and data (including device list acquisition requests) sent by the mobile phone; Data processing module 62: Responsible for parsing control instructions and data, receiving device list acquisition requests forwarded by HomeKit Proxy, forwarding the parsed device list acquisition requests to the device management module 63, and receiving the device information of the TV returned by the device management module, as well as the device information of each device directly or indirectly connected to the TV. Then, the data processing module is also used to add at least one device placeholder based on the device information of the TV and the device information of each device, reorganize the data, generate target information, and return the target information to HomeKit Proxy module 61 so that HomeKit Proxy module 61 can return it to the mobile phone; The device management module 63 is used to obtain the device information of the TV, the device information of each device connected to the TV, and determine the number of devices connected to the TV, etc., and report the obtained information to the data processing module 62.

[0125] In the following description, the inherent device list information includes the device information of the TV and the device information of the device directly connected to the TV; the extended device list information includes the device information of the device indirectly connected to the TV. When the TV is directly or indirectly connected to the target device via the HDMI interface, the device management module 63 is used to identify the connection information corresponding to each HDMI interface of the TV, determine the third device connected to the inherent HDMI interface of the TV, and obtain the device information of the corresponding third device. Then, based on the device information of the TV and the device information of each third device, the inherent device list information is determined; then, from the third device directly connected to the TV via the HDMI interface, the fourth device connected to the third device via the HDMI interface is obtained, and the device information of the fourth device is obtained. Then, based on the device information of each fourth device, the extended device list is obtained.

[0126] In some embodiments, as shown in FIG9 , the device list acquisition method according to an embodiment of the present application includes the following S701 to S710 .

[0127] S701. After the mobile phone and TV are connected to the same network, use the HomeKit app to discover the device and establish a connection with the TV.

[0128] S702: After the connection between the mobile phone and the TV is established, the mobile phone sends a device list acquisition request to the TV.

[0129] S703: After receiving the device list acquisition request, the HomeKit Proxy module of the TV sends the device list acquisition request to the data processing module.

[0130] S704: The data processing module of the television sends a device list acquisition request to the device management module.

[0131] After receiving the device list acquisition request, the TV's HomeKit Proxy module calls the TV's data processing module to request the device list from the device management module.

[0132] S705: The device management module obtains inherent device list information and extended device list information.

[0133] It can be understood that the device management module obtains the number of local HDMI interfaces of the TV and the device information of the TV, as well as the device information of the device connected to the HDMI interface, to obtain the inherent device list information, and at the same time obtains whether the device connected to the HDMI interface is connected to an extension device, and obtains the device information of the extension device connected to the device connected to the HDMI interface, to obtain the extension device list information.

[0134] S706: The device management module returns the inherent device list information and the extended device list information to the data processing module.

[0135] S707: The data processing module adds N device placeholders based on the inherent device list information and the extended device list information, and sets the device name to be empty to generate target information.

[0136] S708: The data processing module returns the target information to the HomeKit Proxy module.

[0137] S709. The HomeKit Proxy module sends the target information to the mobile phone.

[0138] S710: After receiving the target information, the mobile phone displays a device list based on the target information.

[0139] It can be understood that the mobile phone displays the identifier of each inherent device or the identifier and type of each inherent device based on the device information of each device in the inherent device list information according to the target information; and displays the identifier of each extended device or the identifier and type of each extended device based on the device information of each device in the extended device list information, and ignores each device placeholder. It can also be understood that the mobile phone displays the device information of devices whose device information in the target information is not empty, and ignores the device placeholders whose device information is empty. In this way, the initialization of the device list is completed, and the initialized device list reserves a place for the device to be connected.

[0140] In some embodiments, after the device list is initialized, if the user newly connects an expansion device during use of the TV, the mobile HomeKit app needs to be notified to update the list, as shown in Figure 10. The specific process includes the following S801 to S808.

[0141] S801. The HomeKit Proxy module sends a monitoring instruction to the data processing module to monitor whether there is a device connected to the TV or a device connected to the TV.

[0142] S802: The data processing module forwards the monitoring instruction to the device management module.

[0143] S803: The device management module responds to the monitoring instruction and obtains device information of the newly connected expansion device when monitoring that a new expansion device is connected to the device connected to the television.

[0144] S804: The device management module returns the device information of the newly connected expansion device to the data processing module.

[0145] The device management module may update the device information of the newly connected expansion device into the expansion device list information and then send it to the data processing module, or may send the device information of the newly connected expansion device to the data processing module separately.

[0146] S805: The data processing module receives the device information of the newly connected extended device and replaces a device placeholder in the target information with the newly connected extended device information to update the target information.

[0147] S806: The data processing module sends the updated target information to the HomeKit Proxy module.

[0148] S807. The HomeKit Proxy module sends the updated target information to the mobile phone.

[0149] S808. After receiving the updated target information, the mobile phone displays an updated device list based on the updated target information.

[0150] It can be understood that after the device management module recognizes that a new expansion device has been connected, it returns the device information of the newly connected expansion device to the data processing module. The data processing module updates the device information of the newly connected expansion device to a device placeholder in the target information, and sets the name, type and other information of the newly connected expansion device. In this way, the device placeholder becomes a real device; the updated target information is sent to the mobile phone through the HomeKit Proxy module. After the mobile phone receives the updated target information, it refreshes the device placeholder in the list. Since the placeholder has real information such as name and type, it can be refreshed and displayed on the device list displayed on the mobile phone.

[0151] In addition to the above-mentioned embodiment for solving the technical problem of being unable to dynamically update the device list displayed on the iOS device after the iOS device and electronic device A are connected and initialized, this application also provides examples of some embodiments that can further solve other technical problems when the electronic device is used as a display device, as follows:

[0152] The pseudo-standby state refers to the state in which the display device operating system is running, but the display device screen is powered off and in a dark state, and the output port stops outputting. The power-on state refers to the state in which the display device operating system is running, and the display device screen is powered on and in a bright state.

[0153] The Android system is constantly being updated. To protect user privacy, permissions for apps are being tightened across the device to prevent abuse that can lead to performance degradation and user complaints. The current security measures were recently integrated by Google, and many apps haven't yet adapted. This can result in push notifications failing to wake the TV from a false standby state, resulting in a poor user experience.

[0154] In order to solve the above technical problems, the electronic device described in the embodiment of the present application is used as a display device by receiving a screen projection request sent by a smart device; in response to the screen projection request, when the display device is in a pseudo standby state, the display device is controlled to switch from the pseudo standby state to the power-on state based on the AirPlay2 application; the screen projection operation corresponding to the screen projection request is executed based on the AirPlay2 application, so that when the screen projection request sent by the smart device is received in the pseudo standby state, the display device can be woken up first based on the AirPlay2 application, and then the corresponding screen projection operation is executed, so that screen projection can be realized in the pseudo standby state, reducing the user's operating cost from the entire link, improving the efficiency of screen projection processing, and enhancing the user's operating experience.

[0155] Figure 11 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment, wherein the control device includes a smart device or a control apparatus. As shown in Figure 11, a user can operate the display device 200 via a smart device 300 or a control apparatus 001. Both the smart device 300 and the control apparatus 001 can be considered as the control device 100.

[0156] In some embodiments, the control device 001 may be a remote controller. Communication between the remote controller and the display device may include infrared protocol communication, Bluetooth protocol communication, or other short-range communication methods, and the display device 200 may be controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, and the like.

[0157] In some embodiments, a smart device 300 (such as a mobile terminal, tablet computer, computer, laptop computer, etc.) can also be used to control the display device 200. For example, the display device 200 is controlled using an application running on the smart device.

[0158] In some embodiments, the display device may not use the aforementioned smart device or control apparatus to receive instructions, but may receive user control through touch or gestures.

[0159] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 001 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0160] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 400 may provide various content and interactions to the display device 200.

[0161] Figure 12 is a block diagram of the configuration of a control device 001 according to an exemplary embodiment. As shown in Figure 12, the control device 001 includes a processor 110, a communication interface 130, a user input / output interface 140, an external memory, and a power supply. The control device 001 receives user input commands and converts them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200.

[0162] According to an embodiment of the present application, a display device and a screen projection processing method are provided, wherein the display device can implement the screen projection processing method according to the embodiment of the present application or the functional module or functional entity in the display device can implement the screen projection processing method according to the embodiment of the present application. The display device includes: a communicator and a processor, which correspond to the communicator 220 and the processor 250 in Figure 2 above, respectively. The smart device can be the smart device 300 in Figure 11, and the smart device can be a device that supports the AirPlay2 protocol and can perform screen mirroring (screen projection connection) to the display device, such as an iOS device such as an iPhone or iPad.

[0163] In some embodiments, the communicator receives a screen projection request sent by a smart device; the processor responds to the screen projection request and controls the display device to switch from the pseudo standby state to the power-on state based on the AirPlay2 application when the display device is in the pseudo standby state; and executes the screen projection operation corresponding to the screen projection request based on the AirPlay2 application.

[0164] In some embodiments, the display device is controlled to switch from a pseudo-standby state to a power-on state based on the AirPlay2 application. It can be understood that the standard interface for waking up the screen is called based on the AirPlay2 application to power on the screen of the display device, so that the screen changes from a dark screen state to a bright screen state, thereby waking up the screen.

[0165] In some embodiments, the screen projection request can be a screen projection connection request, and the corresponding screen projection operation is to establish a screen projection connection; the screen projection request can be a screen projection push request, and the corresponding screen projection operation is to play the projection content, and the projection content can be pictures, audio or video, etc.; the specific details can be determined according to actual conditions and are not limited here.

[0166] Among them, the AirPlay2 application is an application built based on the AirPlay2 protocol that can realize the AirPlay2 screen projection function between smart devices and display devices. The specific name of the AirPlay2 application is not limited in the embodiment of this application and can be determined according to actual conditions.

[0167] In an embodiment of the present application, when a screen projection request is received from a smart device in a pseudo-standby state, the display device can be first woken up based on the AirPlay2 application, and then the corresponding screen projection operation can be performed, so that screen projection can be achieved in the pseudo-standby state, reducing the user's operating cost from the entire link, improving the efficiency of screen projection processing, and enhancing the user's operating experience.

[0168] In some embodiments, the screen projection request is a screen projection connection request, and the screen projection operation is to establish a screen projection connection; the processor is configured to perform the screen projection operation corresponding to the screen projection request based on the AirPlay2 application, specifically: establish a screen projection connection between the display device and the smart device based on the AirPlay2 application. It can be understood that the communicator receives the screen projection connection request sent by the smart device; the processor responds to the screen projection connection request, and when the display device is in a pseudo-standby state, controls the display device to switch from the pseudo-standby state to the power-on state based on the AirPlay2 application; and establishes a screen projection connection between the display device and the smart device based on the AirPlay2 application.

[0169] In an embodiment of the present application, when a screen projection connection request is received from a smart device in a pseudo-standby state, the display device can be first woken up based on the AirPlay2 application, and then the corresponding screen projection connection operation can be performed, so that the screen projection connection can be successfully established in the pseudo-standby state, reducing the user's operating cost from the entire link, improving the efficiency of establishing the screen projection connection, and enhancing the user's operating experience.

[0170] In some embodiments, the screen projection request is a screen projection push request, and the screen projection operation is to play the projection content; the processor is further configured to: before the communicator receives the screen projection request sent by the smart device, establish a screen projection connection between the display device and the smart device based on the AirPlay2 application; when the target condition is met, control the display device to switch from the power-on state to the pseudo-standby state; the processor is configured to perform the screen projection operation corresponding to the screen projection request based on the AirPlay2 application, specifically: play the projection content corresponding to the screen projection push request based on the AirPlay2 application. It can be understood that the processor establishes a screen projection connection between the display device and the smart device based on the AirPlay2 application; when the target condition is met, control the display device to switch from the power-on state to the pseudo-standby state; the communicator receives the screen projection connection request sent by the smart device; in response to the screen projection connection request, the processor controls the display device to switch from the pseudo-standby state to the power-on state based on the AirPlay2 application when the display device is in the pseudo-standby state; and plays the projection content corresponding to the screen projection push request based on the AirPlay2 application.

[0171] In some embodiments, when the display device is connected to a Wi-Fi network and is powered on, pressing the power button will cause the device to enter a pseudo-standby state. When the display device is powered on and has no audio or video output (e.g., in a paused playback state), and has been stationary for a certain period of time (e.g., 1 hour, 24 hours, etc.) without any key operation, the device will also enter a pseudo-standby state.

[0172] When a user uses AirPlay2 to cast the screen to watch videos or audio, and leaves the screen for too long or presses the power button, the Android TV will enter a pseudo-standby state. When the user returns and wants to cast the screen to watch the program again, he or she can operate the smart device to send a casting push request to the display device. The display device will respond to the casting push request, wake up the screen, and play the corresponding video or audio.

[0173] In an embodiment of the present application, when a screen projection push request is received from a smart device in a pseudo-standby state, the display device can be first woken up based on the AirPlay2 application, and then the projection content corresponding to the screen projection push request can be played based on the AirPlay2 application, so that the corresponding projection content can be played in response to the screen projection push in the pseudo-standby state, thereby reducing the user's operating cost from the entire link, improving the efficiency of screen projection push processing, and enhancing the user's operating experience.

[0174] In some embodiments, the target condition includes any one of the following: the playback content of the display device does not change within a preset time period; a user input for entering a pseudo standby state is received.

[0175] In some embodiments, when the display device is powered on, the playback content of the display device does not change within a preset time period, that is, the display device does not play any content or the playback content does not change (pause playback state) within the preset time period, and the display device switches from the powered on state to the pseudo-standby state.

[0176] In some embodiments, the user input for entering the pseudo standby state can be a user clicking a power button or a button for entering the pseudo standby state, or the display device can display a prompt message indicating whether to enter the pseudo standby state, and the user selecting an option to enter the pseudo standby state. When the user receives the user input for entering the pseudo standby state while the device is powered on, the display device switches from the powered on state to the pseudo standby state.

[0177] In some embodiments, a user can configure an AirPlay 2 application to have a wake-up function in a pseudo-standby state, i.e., a function to wake up the display device screen and switch the display device from the pseudo-standby state to the on state. The processor is further configured to: before the communicator receives a screen projection request sent by a smart device, in response to a received target user input authorizing the AirPlay 2 application to have the wake-up function, determine that the AirPlay 2 application has the wake-up function; wherein the wake-up function is used to control the display device to switch from the pseudo-standby state to the on state.

[0178] In some embodiments, the target user input can be the input of setting the AirPlay2 application to have a wake-up function in the interface corresponding to the application permission setting item under the system setting interface; the target user input can also be the input of selecting the input to authorize the AirPlay2 application to have a wake-up function when the AirPlay2 application is first turned on or turned on for a preset number of times and an authorization prompt message is displayed, prompting whether to authorize the AirPlay2 application to have a wake-up function. The specific input can be determined based on actual conditions and is not limited here.

[0179] In an embodiment of the present application, the AirPlay2 application is set to have a wake-up function through target user input, so that when the display device receives a screen projection request sent by a smart device in a pseudo-standby state, it can control the display device from the pseudo-standby state to the power-on state based on the AirPlay2 application, and then perform the corresponding screen projection operation, thereby reducing the user's operating cost from the entire link, improving the efficiency of screen projection push processing, and enhancing the user's operating experience.

[0180] In some embodiments, when an AirPlay 2 application is launched, an authorization prompt is displayed to enable the AirPlay 2 application to have a wake-up function. The processor is further configured to: display the authorization prompt, the authorization prompt including a first option and a second option, the first option being used to indicate authorization for the AirPlay 2 application to have a wake-up function, and the second option being used to indicate disauthorization for the AirPlay 2 application to have a wake-up function; the target user input is an input selecting the first option.

[0181] In some embodiments, when the user selects the second option for the authorization prompt information, the AirPlay2 application does not have a wake-up function. When the display device receives a screen projection request sent by the smart device in a pseudo-standby state, the AirPlay2 application cannot control the display device to switch from the pseudo-standby state to the power-on state, and the display device remains in the pseudo-standby state, and thus cannot perform the corresponding screen projection operation.

[0182] In the embodiment of the present application, whether the AirPlay2 application has the wake-up function is set by authorization prompt information, which is simple to operate. Moreover, by interacting with the user through the authorization prompt information, the user can set whether the AirPlay2 application has the wake-up function according to his or her own needs, which can improve the user experience.

[0183] In some embodiments, an AirPlay 2 application can be pre-installed in a system directory, and the AirPlay 2 application can then have a wake-up function without user authorization. Specifically, the AirPlay 2 application is installed in the system directory of the display device, and any application in the system directory has a wake-up function; wherein the wake-up function is used to control the display device to switch from a pseudo-standby state to an on state.

[0184] In some embodiments, the system directory can be the system / app directory, the system_ext / priv_app directory, or other system directories. In this case, no user authorization is required, that is, the TV can be woken up through Airplay2 screen casting in pseudo-standby state to perform screen casting demonstration.

[0185] The smart device is an iPhone and the display device is a TV. Screen mirroring is performed on the iPhone, and an interface as shown in Figure 13 is displayed. This interface includes the device identifiers of various devices that can be used for Airplay2 screen projection with the iPhone (which can be device name, device model, device number, etc.). As shown in the figure, it includes multiple identifiers such as "Living Room TV", "Bedroom TV", and "Smart TV". Among them, the user selects the "Smart TV" identifier corresponding to the TV, and the screen projection connection is established between the iPhone and the TV. The TV displays the interface shown in Figure 14, indicating that the screen projection connection between the iPhone and the TV is successfully established.

[0186] The display device includes the Airplay daemon (AirPlayDaemon), the Airplay system service (AirPlaySystemService), and the Airplay2 application (AirPlayApk). Among them, AirPlayDaemon is used to parse the screen projection request. AirPlaySystemService is used to switch to the AirPlay channel and pull up AirPlayApk according to the screen projection parsed by AirPlayDaemon; AirPlaySystemService is also used to forward messages between AirPlayDaemon and AirPlayApk to establish a screen projection connection and screen projection push. AirPlayDaemon corresponds to the iOS system language, AirPlayApk corresponds to the Android system language, and AirPlaySystemService is used to convert between the iOS system language and the Android system language. As shown in Figure 15, the business process of establishing a screen projection connection and completing the screen projection push in the display device includes the following steps.

[0187] S601. AirPlayDaemon receives a screen projection connection request sent by a smart device.

[0188] AirPlayDaemon parses the screen casting connection request to obtain the screen casting connection control instruction, and sends a socket message to AirPlaySystemService. The socket message carries the screen casting connection control instruction, which is used to notify AirPlaySystemService to pull up AirPlayApk, and then let AirPlayApk complete the screen casting connection initialization, realize the screen casting connection, and display the screen casting screen (Curtain) page. The screen casting screen page is used to indicate that the connection is successfully established. The screen casting screen page can be the pairing page on the smart device.

[0189] S602. AirPlayDaemon sends a socket message to AirPlaySystemService.

[0190] S603. AirPlaySystemService executes JNI to trigger Java to callback AirPlayApk based on the socket message.

[0191] JNI (Java Native Interface) is a programming framework used to call Java programs in the Java virtual machine and native applications or libraries. Native applications generally refer to programs written in languages ​​such as C, C++, or assembly and compiled to native hardware and operating systems.

[0192] JNI triggers the Java callback AirPlayApk, that is, JNI calls the logic of starting the application to switch to the AirPlay channel and launch the AirPlay application.

[0193] S604. AirPlayApk broadcasts a first request message for requesting the screen projection connection to initialize startup parameters.

[0194] S605. AirPlayApk broadcasts a second request message for requesting drawing parameters of the projection screen page.

[0195] The first request message and the second request message may be the same request message or two independent request messages.

[0196] S606. AirPlaySystemService broadcasts the projection connection initialization startup parameters and the drawing parameters of the projection screen page.

[0197] AirPlaySystemService can broadcast the casting connection initialization startup parameters and the casting screen page drawing parameters simultaneously through one broadcast message, or it can broadcast the casting connection initialization startup parameters and the casting screen page drawing parameters separately through two broadcast messages.

[0198] After receiving the first request message and the second request message, AirPlaySystemService can forward the first request message and the second request message to AirPlayDaemon. AirPlayDaemon sends the projection connection initialization startup parameters and the drawing parameters of the projection screen page to AirPlaySystemService. AirPlaySystemService then sends the projection connection initialization startup parameters and the drawing parameters of the projection screen page to AirPlayApk.

[0199] S607. AirPlayApk initializes the screen projection based on the screen projection connection initialization startup parameters to complete the screen projection connection; AirPlayApk draws the screen projection page based on the drawing parameters of the screen projection page and displays the screen projection page.

[0200] When the TvInputService onCreate cycle function in AirPlayApk is started, the broadcast AirPlay.check.breakin (first request message) is sent to inform that the channel switching is successful and request the screen projection connection to initialize the startup parameters. When the InputService.Session.onCreateOverLay function is triggered, the broadcast Request.CurtainPage.TIS (second request message) is sent to request the drawing parameters of the projection screen page. After obtaining the drawing parameters of the projection screen page, the projection screen page is drawn and displayed.

[0201] S608. AirPlayDaemon receives a screen projection push request sent by the smart device.

[0202] AirPlayDaemon parses the screen casting push request. If the screen casting data corresponding to the screen casting push request is audio data, the screen casting push request is parsed to obtain the audio data and the screen casting push control instruction. AirPlayDaemon processes the audio data and sends the screen casting push control instruction to AirPlaySystemService; if the screen casting data corresponding to the screen casting push request is a video uniform resource locator (Uniform Resource Locator, URL), the screen casting push request is parsed to obtain the video URL and the screen casting push control instruction. AirPlayDaemon sends the video URL and the screen casting push control instruction to AirPlaySystemService, and AirPlaySystemService downloads the video data according to the video URL.

[0203] S609. AirPlaySystemService receives the screen projection push control instruction sent by AirPlayDaemon.

[0204] AirPlaySystemService receives the screen push control instruction of video or audio, and sends the broadcast AirPlay.from.breakin (screen push control instruction) to notify AirPlayApk to play.

[0205] S610, AirPlaySystemService broadcasts screen projection and pushes control instructions.

[0206] S611. AirPlayApk receives the screen projection push control instruction and broadcasts a third request message for requesting screen projection playback startup parameters.

[0207] AirPlayApk receives the screen projection push control instruction, creates a session, and sends a broadcast AirPlay.reqbreakin.parameter (the third request message) to AirPlaySystemService to request startup parameters.

[0208] S612, AirPlaySystemService broadcasts screen casting and playback startup parameters.

[0209] AirPlaySystemService sends the startup parameters to AirPlayApk using the broadcast airplay.tis.action function.

[0210] After receiving the third request message, AirPlaySystemService can forward the third request message to AirPlayDaemon. AirPlayDaemon sends the screen casting and playback startup parameters to AirPlaySystemService, and AirPlaySystemService then sends the screen casting and playback startup parameters to AirPlayApk.

[0211] S613, AirPlayApk performs screen casting based on the screen casting startup parameters.

[0212] AirPlay calls launchAirPlayAppTIS to inform the underlying screen casting that the playback has started and is playing.

[0213] Figure 16 shows the audio data push process based on the AirPlay2 application. The sender device is a smart device, AirPlay Daemon and Luna are display devices, and Luna is part of AirPlayApk. When the display device receives the audio push from the sender device, the audio buffer data is directly sent to the AirPlay Daemon, and the metadata data is forwarded to Luna in the AirPlay app for upper-level UI drawing. The sender device and AirPlay Daemon node synchronize audio through Clock Sync to achieve simultaneous playback.

[0214] Metadata, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource search, and file records.

[0215] Figure 17 shows the video data push process for AirPlay 2 applications. When a display device receives a video push from a sending device, it sends the video URL to the AirPlay Daemon, which then forwards it to the Luna engine in the AirPlay app. Luna then completes media resource download, video playback, and upper-layer UI rendering.

[0216] In order to explain the present solution in more detail, the following will be explained in an exemplary manner in conjunction with Figures 18 to 22. It can be understood that the steps involved in Figures 18 to 22 may include more steps, or fewer steps in actual implementation, and the order of these steps may also be different, so as to be able to implement the screen projection processing method provided in the embodiment of the present application. The execution subject of the screen projection processing method can be a display device, or it can be a functional module or functional entity in the display device that can implement the screen projection processing method, which is not limited here. Moreover, the specific description of the screen projection processing method according to the embodiment of the present application can refer to the relevant description of the above-mentioned display device, and the same or similar technical effects can be achieved, which will not be repeated here.

[0217] FIG18 is a flowchart of steps for implementing a screen projection processing method according to one or more embodiments of the present application. The screen projection processing method may include:

[0218] S901. Receive a screen projection request sent by a smart device.

[0219] S902. In response to the screen projection request, when the display device is in a pseudo-standby state, control the display device based on the AirPlay2 application to switch from the pseudo-standby state to the power-on state.

[0220] S903. Execute the screen projection operation corresponding to the screen projection request based on the AirPlay2 application.

[0221] In some embodiments, the screen projection request is a screen projection connection request, and the screen projection operation is to establish a screen projection connection; in combination with Figure 18, as shown in Figure 19, the above S903 can be specifically implemented through the following S903a.

[0222] S903a. Establish a screen projection connection between the display device and the smart device based on the AirPlay2 application.

[0223] In some embodiments, the screen projection request is a screen projection push request, and the screen projection operation is to play the projection content; combined with Figure 18, as shown in Figure 20, before the above S901, the screen projection processing method according to the embodiment of the present application can also include the following S904 and S905, and the above S903 can be specifically implemented through the following S903b.

[0224] S904. Establish a screen projection connection between the display device and the smart device based on the AirPlay2 application.

[0225] S905: When the target condition is met, control the display device to switch from the power-on state to the pseudo-standby state.

[0226] S903b. Play the projection content corresponding to the projection push request based on the AirPlay2 application.

[0227] In some embodiments, the target condition includes any one of the following: the playback content of the display device does not change within a preset time period; a user input for entering a pseudo standby state is received.

[0228] In some embodiments, in combination with Figure 18, as shown in Figure 21, before the above-mentioned S901, the screen projection processing method according to the embodiment of the present application may also include the following S906.

[0229] S906 : In response to the received target user input authorizing the AirPlay2 application to have the wake-up function, determine that the AirPlay2 application has the wake-up function.

[0230] The wake-up function is used to control the display device to switch from a pseudo-standby state to a power-on state.

[0231] In some embodiments, in combination with Figure 21, as shown in Figure 22, before the above-mentioned S901, the screen projection processing method according to the embodiment of the present application may also include the following S907.

[0232] S907: Display authorization prompt information.

[0233] Among them, the authorization prompt information includes a first option and a second option. The first option is used to indicate that the AirPlay2 application is authorized to have a wake-up function, and the second option is used to indicate that the AirPlay2 application is not authorized to have a wake-up function; the target user input is the input of selecting the first option.

[0234] In some embodiments, the AirPlay2 application is installed in the system directory of the display device, and any application in the system directory has a wake-up function; wherein, the wake-up function is used to control the display device to switch from a pseudo-standby state to a power-on state.

[0235] In addition to the above-mentioned embodiments for solving the technical problem of being unable to dynamically update the device list displayed on the iOS device after the iOS device and electronic device A are connected and initialized, and the embodiments for enabling screen projection in a pseudo-standby state, thereby reducing the user's operating costs across the entire link, improving screen projection processing efficiency, and enhancing the user's operating experience, this application also provides examples of some embodiments that can further solve other technical problems when the electronic device is used as a display device, as follows:

[0236] With the rapid development of the internet and the widespread adoption of artificial intelligence (AI) technology in recent years, display devices such as smart TVs have gradually become essential devices in people's daily lives and the preferred entertainment medium for living rooms. Especially during family gatherings, the large screen size of smart TVs becomes increasingly prominent, making them a popular entertainment option.

[0237] At the same time, as user needs gradually increase, the requirements for large screens are also higher. For example, users hope to share real-time game images on a large screen; for another example, users hope to play their favorite videos on a large screen. In order to meet user needs, the screen projection function can solve this problem very well. Current display devices can simultaneously support multiple screen projection protocols, such as Miracast protocol, AirPlay (air playback) protocol, etc.

[0238] When using a display device to play a data stream via screen projection, if the user only wants to hear the audio of the data stream, they can operate the display device to enter target mode. In target mode, the display device only plays the audio data, and the display switch of the monitor is in the off state. However, if the user casts the screen again to play other data streams, such as playing a video, the display device will still be in target mode and the display switch of the monitor will still be in the off state, resulting in the user being unable to view the video content, which is a poor user experience.

[0239] In response to the above problem, when the above electronic device is used as a display device, an embodiment of the present application also proposes a screen projection processing method, wherein the processor of the display device first obtains the type of the currently received screen projection data stream and the current playback status of the display device, and then determines that the type of the currently received screen projection data stream is non-audio data, and determines that the current playback status of the display device is in target mode, controls the display switch of the display to be in an open state, and controls the display to display non-audio data, wherein, when the playback status is in target mode, the display switch of the display is in a closed state. In the above technical solution, the display device can automatically control the display switch of the display to be in an open state when determining that the type of the currently received screen projection data stream is non-audio data, and determining that the current playback status of the display device is in target mode, and controls the display to display non-audio data, thereby avoiding the situation where the display switch of the display is in a closed state when non-audio data is currently being played due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the screen projection process, thereby improving the user experience.

[0240] Figure 23 is a schematic diagram of the software configuration in a display device according to one or more embodiments of the present application. As shown in Figure 23, the system is divided into three layers: from top to bottom, the application layer (abbreviated as "application layer"), the application framework layer (abbreviated as "framework layer"), and the operating system layer.

[0241] In some embodiments, at least one application is running in the application layer. These applications may be window programs, system settings programs, clock programs, etc. that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the applications in the application layer include but are not limited to the above examples. In this embodiment, the application layer may include TV Input (the input module of the TV, also known as the channel list module) and the AirPlay Android Application Package (AirPlay APK), wherein the AirPlay APK is mainly used to obtain the properties of the entire machine (display device), such as obtaining the volume of the entire machine, etc., which are not limited here.

[0242] In some embodiments, the framework layer primarily includes: switching channels (switching inputs, switching TISs), AirPlay services, and audio and video middleware. The AirPlay service can interact with the AirPlay process and AirPlay APK, control the AirPlay APK's lifecycle, and interact with the switching channels and audio and video middleware through interface calls. The AirPlay service interacts with the AirPlay APK through broadcasts, and the switching channel interacts with the TV Input through the Java Native Interface (JNI).

[0243] In some embodiments, the operating system layer mainly includes: the AirPlay process and the implementation of functional modules such as mirroring, video, and audio. Among them, the operating system here can be Linux. The AirPlay process is mainly a resident native process, including the AirPlay protocol stack, which can call the specific AirPlay playback function module (mirror, video or audio, etc.) to implement the corresponding function; the AirPlay process and the AirPlay service realize the interaction between the two through network communication. The network communication here can be implemented based on a network communication protocol, such as a socket protocol or other protocols, etc., which is not limited in this embodiment.

[0244] In some embodiments, when a user operates a mobile device to cast a screen, the AirPlay protocol stack can receive the casting information or casting request sent by the mobile device, and send the casting information or casting request to the AirPlay service. After the AirPlay service receives the casting information or casting request (including casting parameters), it can determine the type of the current data stream to be cast (such as video, audio or mirror, etc.), and can also determine whether it is the first time to cast a screen. If it is the first time to cast a screen, the interface in the Android TV Input Framework (TIF) is called to enable the TV Input to switch channels and switch the current channel of the display device to the channel corresponding to AirPlay; if it is not the first time to cast a screen, the AirPlay service will obtain the type of the currently received casting data stream, and call the audio and video middleware interface to obtain the current playback status of the display device. If the type of the currently received casting data stream is non-audio data, and it is determined that the current playback status of the display device is in target mode, then AirPlay will call the audio and video middleware interface and turn off the switch corresponding to the Audio Only function to control the display switch of the display to be in the on state. At this time, the screen of the display will light up and the display can display non-audio data.

[0245] It should be noted that FIG. 23 illustrates the software configuration in the display device by taking the display device using the Android platform as an example, and is not intended to limit it.

[0246] Figure 24 is a schematic diagram of the icon control interface display of an application in a display device according to one or more embodiments of the present application. As shown in Figure 24, the application layer includes at least one application that can display corresponding icon controls in the display, such as: live TV application icon controls, video on demand application icon controls, media center application icon controls, application center icon controls, game application icon controls, etc. Live TV applications can provide live TV through different signal sources. Video on demand applications can provide videos from different storage sources. Unlike live TV applications, video on demand provides video display from certain storage sources. Media center applications can provide applications for playing various multimedia content. Application center can provide storage for various applications.

[0247] In some embodiments, the display device is a terminal device with a display function, such as a television, a mobile phone, a computer, a learning machine, etc. In the display device:

[0248] an output interface (display 260 and / or audio output interface 270 ), configured to output user interaction information;

[0249] Communicator 220, for communicating with the server;

[0250] The processor 250 is configured to execute computer instructions in the memory to cause the device to perform:

[0251] Obtain the type of the currently received screen projection data stream and the current playback status of the display device;

[0252] Determine that the type of the currently received projection data stream is non-audio data, and determine that the current playback state of the display device is in target mode, control the display switch of the display to be in an open state, and control the display to display the non-audio data, wherein, when the playback state is in the target mode, the display switch of the display is in a closed state.

[0253] In some embodiments, the processor 250 is further configured to:

[0254] After determining that the type of the currently received projection data stream is non-audio data and determining that the current playback state of the display device is in target mode, controlling the display to display prompt information, wherein the prompt information is used to prompt the user whether to turn on the display switch of the display;

[0255] Upon receiving an operation of turning on a display switch of the display, the display switch of the display is controlled to be in an on state, and the display is controlled to display the non-audio data.

[0256] In some embodiments, the processor 250 is further configured to:

[0257] In response to the current screen projection request, determining channel information of the display device;

[0258] When it is determined based on the channel information that the channel of the display device is the channel corresponding to airplay, and the current screen projection request is a non-mirror screen projection request, the type of the currently received screen projection data stream and the current playback status of the display device are obtained.

[0259] In some embodiments, the processor 250 is further configured to:

[0260] Call the audio and video middleware interface to obtain the status of the switch corresponding to the audio-only function;

[0261] When the state is the open state, it is determined that the current playback state of the display device is in the target mode.

[0262] In some embodiments, the processor 250 is further configured to:

[0263] The audio and video middleware interface is called to turn off the switch corresponding to the Audio Only function, so as to control the display switch of the display to be in the on state.

[0264] In some embodiments, the processor 250 is further configured to:

[0265] Determine that the current screen projection request is a first screen projection request, establish a screen projection connection between the display device and the sender of the current screen projection request, and switch the channel of the display device to the channel corresponding to the airplay;

[0266] Based on the screen projection connection and the channel corresponding to the airplay, the data stream sent by the sender is received and the data stream is played.

[0267] In some embodiments, the screen projection protocol between the display device and the sender of the current screen projection request is: AirPlay protocol.

[0268] To sum up, the embodiment of the present application executes the above-mentioned screen projection processing method on the display device. The processor of the display device first obtains the type of the currently received screen projection data stream and the current playback status of the display device, and then determines that the type of the currently received screen projection data stream is non-audio data, and determines that the current playback status of the display device is in the target mode, controls the display switch of the display to be in the open state, and controls the display to display non-audio data, wherein, when the playback status is in the target mode, the display switch of the display is in the closed state. In the above-mentioned technical solution, the display device can automatically control the display switch of the display to be in the open state and control the display to display non-audio data when determining that the type of the currently received screen projection data stream is non-audio data and that the current playback status of the display device is in the target mode, thereby avoiding the situation where the display switch of the display is in the closed state when the non-audio data is currently playing due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the screen projection process, thereby improving the user experience.

[0269] FIG25 is a system framework diagram for performing screen projection processing according to one or more embodiments of the present application. As shown in FIG25 , the system may include an acquisition module 401, a determination module 402, and a control module 403. The system first obtains the type of the currently received screen projection data stream and the current playback state of the display device through the acquisition module 401, then the determination module 402 determines that the type of the currently received screen projection data stream is non-audio data, and determines that the current playback state of the display device is in target mode. Finally, the control module 403 controls the display switch of the display to be in an open state and controls the display to display non-audio data, wherein when the playback state is in target mode, the display switch of the display is in an off state. In the above technical solution, the display device can automatically control the display switch of the display to be in an open state and control the display to display non-audio data when determining that the type of the currently received screen projection data stream is non-audio data and that the current playback state of the display device is in target mode, without the need for manual operation by the user, thereby avoiding the situation where the display switch of the display is in an off state when the non-audio data is currently playing due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the screen projection process, thereby improving the user experience.

[0270] In order to explain the screen projection processing scheme in the embodiment of the present application in more detail, the following will be explained in an exemplary manner in combination with Figure 26. It can be understood that the steps involved in Figure 26 may include more steps or fewer steps in actual implementation, and the order of these steps may also be different, so as to implement the screen projection processing method provided in the embodiment of the present application. The embodiment of the present application does not limit it.

[0271] FIG26 is a flow chart of a screen projection processing method according to an embodiment of the present application. This embodiment can be applied to prevent the display switch of the display from being in the off state during the screen projection process, resulting in the inability to view the projected content. As shown in FIG26, the screen projection processing method specifically includes the following steps:

[0272] S51, obtain the type of the currently received screen projection data stream and the current playback status of the display device.

[0273] Among them, the currently received screen projection data stream can be understood as the screen projection data stream currently received by the display device and sent by the mobile device. It can be understood that: the user can operate the mobile device to cast the screen. During the screen projection process, the user can cast the screen multiple times, for example, first cast audio data, then cast video data, and then cast image data, etc. Correspondingly, the mobile device will send multiple screen projection data streams to the display device. In this embodiment, there is no specific limitation on the mobile device that sends the screen projection data stream to the display device during the screen projection process. It can be the same mobile device or different mobile devices. The mobile device can include a mobile phone, iPad or macbook, etc., which are client devices that support the Airplay protocol. The types of screen projection data streams can include audio data and non-audio data, wherein audio data is sound-related data, such as music, recordings, etc.; non-audio data can include video streams, live streams or broadcast streams, etc., which contain picture content. The current playback status of the display device can include: the display device is currently playing only audio, the display device is currently playing only picture content, or the display device is currently playing audio and picture content at the same time, etc.

[0274] In order to solve the problem that the user cannot watch the projected content because the display switch of the display is in the off state during the screen projection process, the processor of the display device obtains the type of the currently received projection data stream and the current playback status of the display device, so as to determine whether the type of the currently received projection data stream is non-audio data, and whether the current playback status of the display device is in target mode, which is conducive to the execution of subsequent steps.

[0275] S52, determine that the type of the currently received projection data stream is non-audio data, and determine that the current playback state of the display device is in target mode, control the display switch of the display device to be in the on state, and control the display to display non-audio data.

[0276] When the playback state is in target mode, the display switch of the display is in the off state, that is, in target mode, the display is in a black screen state, and the user cannot view the screen content. The display switch of the display can be understood as a switch that controls the display to be in a black screen state or a bright screen state.

[0277] The processor of the display device determines that the type of the currently received projection data stream is non-audio data, and that the current playback state of the display device is in target mode based on the type of the currently received projection data stream and the current playback state of the display device. This indicates that the display device is projecting and playing non-audio data at this time, but the display switch of the display is in the off state, and the screen content of the non-audio data cannot be played normally. Therefore, the processor of the display device controls the display switch of the display device to be in the on state, and controls the display to display the non-audio data (screen content), thereby achieving the effect of playing the non-audio data normally.

[0278] In this embodiment, the processor of the display device first obtains the type of the currently received projection data stream and the current playback status of the display device, and then determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback status of the display device is in target mode, controls the display switch of the display to be in an open state, and controls the display to display non-audio data, wherein, when the playback status is in target mode, the display switch of the display is in a closed state. In the above technical solution, the display device can automatically control the display switch of the display to be in an open state and control the display to display non-audio data when determining that the type of the currently received projection data stream is non-audio data and that the current playback status of the display device is in target mode, without the need for manual operation by the user, thereby avoiding the situation where the display switch of the display is in a closed state when non-audio data is currently playing due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the projection process, thereby improving the user experience.

[0279] In some embodiments, determining that the current playback state of the display device is in the target mode may specifically include:

[0280] Call the audio and video middleware interface to obtain the status of the switch corresponding to the audio-only function;

[0281] When the state is the open state, it is determined that the current playback state of the display device is in the target mode.

[0282] Among them, the Audio Only function is a function provided by the display device to play only sound and turn off the display device screen. Turning off the display device screen means controlling the display switch of the display to be in the off state. The switch corresponding to the Audio Only function can be understood as a module used to control turning this function on or off. The state of the switch corresponding to the Audio Only function can be controlled by the user operating the remote control or the mobile device connected to the display device. For example, by clicking the Menu button of the remote control (it can also be other buttons, which is not limited in this embodiment), the switch corresponding to the Audio Only function can be called up to control its state. The audio and video middleware interface can be understood as an interface for recording and controlling the state of the switch corresponding to the Audio Only function.

[0283] Specifically, the processor of the display device calls the audio and video middleware interface to obtain the status of the switch corresponding to the Audio Only function. When the status of the switch corresponding to the Audio Only function is on, it can be determined that the current playback status of the display device is in the target mode.

[0284] In this embodiment, determining that the current playback state of the display device is in the target mode by the above method is simple, fast and efficient.

[0285] In some embodiments, controlling the display switch of the display to be in an on state may specifically include:

[0286] The audio and video middleware interface is called to turn off the switch corresponding to the Audio Only function, so as to control the display switch of the display to be in the on state.

[0287] Specifically, the processor of the display device calls the audio and video middleware interface and turns off the switch corresponding to the Audio Only function, that is, the display device only plays sound and the function of turning off the display device screen is turned off. After turning off the switch corresponding to the Audio Only function, the display switch of the display can be controlled to be in the on state, and the display can display the picture normally.

[0288] In this embodiment, the above method can automatically control the display switch of the display to be in the on state, which is simple and efficient, and does not require manual operation by the user, avoiding the situation where the display cannot display the picture normally because the display is in a black screen state when playing non-audio data.

[0289] For example, Figure 27 is a schematic diagram of the display interface when the display device is currently in target mode according to an embodiment of the present application. As shown in Figure 27, when the display device is currently in target mode (i.e., the switch corresponding to the Audio Only function is not turned on), the display switch of the display is in the off state, and at this time, the display interface is a black screen interface.

[0290] For example, Figure 28 is a schematic diagram of a display interface when the display switch of a display is in the on state and the display is displaying non-audio data according to an embodiment of the present application. As shown in Figure 28, when the display switch of the display is in the on state and the display is displaying non-audio data, the display normally displays the screen content corresponding to the non-video data.

[0291] It should be noted that the screen content shown in FIG. 28 is for exemplary purposes only. The screen content may be determined based on the playback progress corresponding to the non-video data, and this embodiment does not impose any limitation thereto.

[0292] Figure 29 is a flow chart of another screen projection processing method according to an embodiment of the present application. This embodiment is optimized based on the above embodiment. This embodiment can be applied to the process of determining that the type of the currently received screen projection data stream is non-audio data, and determining that the current playback state of the display device is in target mode. As shown in Figure 29, the screen projection processing method specifically includes the following steps:

[0293] S291, obtain the type of the currently received screen projection data stream and the current playback status of the display device.

[0294] S292, determine that the type of the currently received projection data stream is non-audio data, and determine that the current playback status of the display device is in target mode, and control the display to display prompt information.

[0295] The prompt information is used to prompt the user whether to turn on the display switch of the display.

[0296] The processor of the display device determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback status of the display device is in target mode, and controls the display to display a prompt message, through which the user can be prompted whether to turn on the display switch of the display.

[0297] S293: receiving an operation of turning on a display switch of the display, controlling the display switch of the display to be in an on state, and controlling the display to display video data.

[0298] When the processor of the display device receives an operation of turning on a display switch of the display triggered by the user, it controls the display switch of the display to be in an on state, and controls the display to display video data.

[0299] In this embodiment, the processor of the display device first obtains the type of the currently received projection data stream and the current playback status of the display device, then determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback status of the display device is in target mode, controls the display to display prompt information, and finally receives an operation to turn on the display switch of the display, controls the display switch of the display to be in the on state, and controls the display to display video data. By displaying the prompt information on the display, the user can be prompted to know that the display switch of the display was turned off during the previous screen projection, which can increase the human-computer interaction experience. At the same time, after receiving the operation to turn on the display switch of the display, the display switch of the display is controlled to be in the on state, and controls the display to display video data, avoiding the situation where the display switch of the display is in the off state (i.e., black screen state) when non-audio data is currently playing due to the target mode, and the user cannot view the content of the non-audio data, ensuring that the user can normally view the projection content during the screen projection process, thereby improving the user's usage experience.

[0300] Exemplarily, Figure 30 is a schematic diagram of the interface when a display displays prompt information according to an embodiment of the present application. As shown in Figure 30, after the processor of the display device determines that the type of the currently received projection data stream is non-audio data, and determines that the current playback state of the display device is in target mode, it can control the display to display prompt information. The prompt information may include a countdown prompt. Specifically, the processor of the display device can start a 5-minute countdown, and a prompt page as shown in Figure 30 will pop up 15 seconds after the countdown ends. The page includes: whether to turn on the display switch of the display, and the operable options include "Yes" and "No". Among them, Yes is used to turn on the display switch of the display, and No is used to not turn on the display switch of the display. The 15s in the upper right corner is a time prompt, and its value will continue to decrease with the change of time until it reaches 0.

[0301] The prompt page shown in Figure 30 can prompt the user whether to turn on the display switch to avoid the display remaining in a black screen state. If no user operation is received, the processor of the display device automatically closes the prompt page after 15 seconds by default and automatically turns on the display switch.

[0302] It is understandable that 5 minutes and 15 seconds are both custom values ​​and may be other values, which are not limited in this embodiment.

[0303] In some embodiments, obtaining the type of the currently received screen projection data stream and the current playback status of the display device may specifically include:

[0304] In response to the current screen projection request, determining channel information of the display device;

[0305] When it is determined based on the channel information that the channel of the display device is the channel corresponding to airplay, and the current screen projection request is a non-mirror screen projection request, the type of the currently received screen projection data stream and the current playback status of the display device are obtained.

[0306] The current screen casting request can be understood as the screen casting request currently received by the display device and sent by the mobile device. The screen casting request can carry relevant screen casting parameters, such as mirroring, screen casting video, or screen casting audio information.

[0307] Specifically, after the processor of the display device receives the current screen projection request sent by the mobile device, it determines the channel information of the display device in response to the current screen projection request, wherein the communication information may be the identification information of the channel currently corresponding to the display device. When the processor of the display device determines that the channel of the display device is the channel corresponding to the airplay based on the channel information (indicating that the current screen projection request is not the first screen projection request, that is, it is not the first screen projection), and the current screen projection request is a non-mirror screen projection request, the type of the currently received screen projection data stream and the current playback status of the display device are obtained.

[0308] It should be noted that: if the current screen projection request is the first screen projection request or a mirrored screen projection request, the display device will execute the logic of switching channels, that is, the channel of the display device will be switched to the channel corresponding to the airplay. After the display device executes the logic of switching channels, the display will not have a black screen. When the current screen projection request is: not the first screen projection request and not a mirrored screen projection request, the display device will not execute the logic of switching channels. In this case, if the current playback status of the display device is in target mode (the target mode can be set during the previous screen projection), the display will remain in a black screen state (that is, the display switch of the display is in the off state).

[0309] In this embodiment, under the above conditions, obtaining the type of the currently received projection data stream and the current playback status of the display device is conducive to the subsequent determination of whether to execute the step of controlling the display switch of the display to be in the on state, thereby preventing the situation where the non-audio data screen content cannot be played due to the display switch of the display being in the off state.

[0310] It should be noted that: if the current screen projection request is the first screen projection request, the display device will execute the logic of switching channels. There is no restriction on the type of screen projection data stream corresponding to the first screen projection request, that is: regardless of whether the screen projection data stream is audio data or non-audio data, the display device will execute the logic of switching channels.

[0311] For example, Figure 31 is a schematic diagram of the interface of a display when a display device enters an input page according to an embodiment of the present application. As shown in Figure 31, the user can operate the input button of the remote control or operate other mobile devices connected to the display device to enter the input page. The page may include multiple options, such as: Home, TV, AirPlay, HDMI (High Definition Multimedia Interface) 1 and HDMI2, etc. When switching from one option to another in the page, the display device will execute the logic of switching channels.

[0312] In some embodiments, when the AirPlay portal is in the settings and the AirPlay option is selected, the display device will also execute the logic of switching channels.

[0313] In some embodiments, the method may further specifically include:

[0314] Determine that the current screen projection request is a first screen projection request, establish a screen projection connection between the display device and the sender of the current screen projection request, and switch the channel of the display device to the channel corresponding to the airplay;

[0315] Based on the screen projection connection and the channel corresponding to the airplay, the data stream sent by the sender is received and the data stream is played.

[0316] Among them, the sender of the current screen projection request can be any mobile device, and this embodiment does not limit this.

[0317] Specifically, after the processor of the display device receives the current screen projection request sent by the mobile device, it determines the channel information of the display device in response to the current screen projection request. When it is determined based on the channel information that the channel of the display device is not the channel corresponding to the airplay, it can be determined that the current screen projection request is the first screen projection request. At this time, the processor of the display device establishes a screen projection connection between the display device and the sender of the current screen projection request based on the first screen projection request, and switches the channel of the display device to the channel corresponding to the airplay. At the same time, since the current screen projection request is the first screen projection request, the display device will execute the logic of switching channels. After the display device executes the logic of switching channels, the display will not have a black screen. Therefore, the processor of the display device can play the data stream normally after receiving the data stream sent by the sender based on the screen projection connection and the channel corresponding to the airplay.

[0318] In this embodiment, the above process can ensure that the corresponding data stream is played normally when the screen is projected for the first time.

[0319] In some embodiments, the screen projection protocol between the display device and the sender of the current screen projection request is: AirPlay protocol.

[0320] Among them, the AirPlay protocol is a playback technology in the iOS operating system, which can transfer files (including videos, photos and mirrors) on iOS or later versions to devices that support airplay.

[0321] Specifically, there is a logic of switching channels in the AirPlay protocol. Therefore, in this embodiment, the screen projection protocol between the display device and the sender of the current screen projection request is: AirPlay protocol.

[0322] For example, FIG32 is a schematic diagram of a screen projection process of a display device according to an embodiment of the present application. As shown in FIG32 , the screen projection process of the display device may include the following steps:

[0323] Step 1: When a user casts a screen through a mobile device, the screen casting driver module of the display device can receive the screen casting parameters sent by the mobile device, such as mirroring, screen casting video or screen casting audio, and determine whether the current screen casting is the first time;

[0324] Step 2: If the current screen projection is the first time, since the display device is not currently in the AirPlay channel when the screen is projected for the first time, the screen projection driver sends a channel switching message to the channel switching module, so that the channel switching module switches the channel of the display device to the channel corresponding to AirPlay, and then calls the AirPlay function module (such as audio, video or mirroring, etc.) to implement the corresponding function; if the current screen projection is mirror projection, regardless of whether it is the first time, the screen projection driver will send a channel switching message to the channel switching module, and implement mirror projection through the mirroring function module, so that the display device synchronously plays the screen content of the mobile device;

[0325] Step 3. If the current screen projection is not the first screen projection and is not a mirror projection, the screen projection driver defaults the display device's channel to the channel corresponding to AirPlay. Then, when the type of the currently received screen projection data stream is non-audio data, the screen projection driver calls the audio and video middleware interface to obtain the status of the switch corresponding to the Audio Only function. If the status of the switch corresponding to the Audio Only function is on, the status of the switch corresponding to the Audio Only function is controlled to be off, so as to control the display switch of the monitor to be on and control the monitor to display non-audio data; if the status of the switch corresponding to the Audio Only function is off, the monitor is controlled to display non-audio data.

[0326] FIG33 is a schematic diagram of the overall process of a screen projection processing method according to an embodiment of the present application. As shown in FIG33 , the overall process may specifically include the following steps:

[0327] S331. The user selects any music player software on the mobile device to cast the audio (i.e., the user casts the screen to play the audio data). At this time, the TV will display the audio-related information in the mobile device and start playing the audio;

[0328] S332. On the Audio page of the display device, the user operates the menu button on the remote control (or other means) and clicks the "Audio Only" option in the menu. This turns on the switch corresponding to the Audio Only function. At this time, the display device only plays audio and turns off the display of the display device.

[0329] S333: When a user selects non-audio data, such as video data, for screen projection on a mobile device, because the display device has previously played audio, the logic for switching channels will not be executed when playing video. It is assumed that the display device's channel has not changed. Since the switch corresponding to the Audio Only function is on, the display screen will not light up and remain in a black state.

[0330] S334: The user plays non-audio data on the screen;

[0331] S335. The display device determines whether the type of the currently received projection data stream is non-audio data and whether the state of the switch corresponding to the Audio Only function is on; if so, that is, the type of the currently received projection data stream is non-audio data and the state of the switch corresponding to the Audio Only function is on, then execute step S336; otherwise, the type of the currently received projection data stream is audio data, or the state of the switch corresponding to the Audio Only function is not on (i.e., off), then execute step S338;

[0332] S336. If the display device determines that the type of the currently received projection data stream is non-audio data and the switch corresponding to the Audio Only function is on, turn off the switch corresponding to the Audio Only function;

[0333] S337, the display switch of the display is in the on state;

[0334] S338. Control the display device to play the current non-audio data normally.

[0335] In some embodiments, the state change of the screen projection can be determined in the following manner, at which time it can be identified that the current state of Airplay is Audio / Video / Screen.

[0336] The code is as follows:

[0337] In the above code file, the parameter ui is the current Airplay status, which can be Screen (equivalent to the Airplay homepage) / Video / Audio. Whenever the user casts the screen, the notifyUIPage method is called. For example, if the user casts the screen to play video data, then ui = video; if the user casts the screen to play audio data, then ui = audio.

[0338] The implementation logic of quitAudioOnlyWhenAudioToVideo(ui) is as follows:

[0339] First, get the last Airplay status. If it is Audio and the current Airplay status is Video, then determine whether the switch status of the Audio Only function is on. Turn off the Audio Only function and the display screen will light up.

[0340] The code is as follows:

[0341] To sum up, the embodiment of the present application executes the above-mentioned screen projection processing method on the display device. The processor of the display device first obtains the type of the currently received screen projection data stream and the current playback status of the display device, and then determines that the type of the currently received screen projection data stream is non-audio data, and determines that the current playback status of the display device is in the target mode, controls the display switch of the display to be in the open state, and controls the display to display non-audio data, wherein, when the playback status is in the target mode, the display switch of the display is in the closed state. In the above-mentioned technical solution, the display device can automatically control the display switch of the display to be in the open state and control the display to display non-audio data when determining that the type of the currently received screen projection data stream is non-audio data and that the current playback status of the display device is in the target mode, thereby avoiding the situation where the display switch of the display is in the closed state when the non-audio data is currently playing due to the target mode, and the user cannot view the content of the non-audio data, thereby ensuring that the user can normally view the projection content during the screen projection process, thereby improving the user experience.

[0342] According to an embodiment of the present application, a computer-readable non-volatile storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned device list acquisition method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0343] The computer-readable non-volatile storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0344] An embodiment of the present application further provides a computer program product, comprising: when the computer program product is run on a computer, enabling the computer to implement the above method.

[0345] For ease of explanation, the above description has been made with reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations are possible. The above embodiments are selected and described to better explain the principles and practical applications, so that those skilled in the art can better utilize the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. An electronic device comprising: The communicator is configured to: after establishing a connection with a HomeKit client based on the HomeKit protocol, receive a device list acquisition request sent by the HomeKit client, the device list acquisition request being used to instruct acquisition of device information of each target device connected to the electronic device, the device information including an identifier of the device; A processor configured to: determine device information of the electronic device and device information of each target device based on the device list acquisition request, wherein the device information includes an identifier of a corresponding device; Generate target information based on the device information of the electronic device, the device information of each target device, and at least one device placeholder, each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty; The communicator is further configured to: send the target information to the HomeKit client, so that the HomeKit client displays a device list based on the target information, where the device list includes an identifier of the electronic device and an identifier of each target device.

2. The electronic device according to claim 1, The processor is further configured to: after sending the target information to the HomeKit client, detect that a first device establishes a connection with the electronic device; updating one of the device placeholders in the target information with the device information of the first device to update the target information; The communicator is further configured to: send the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information, and the updated device list includes the identifier of the electronic device, the identifiers of each target device, and the identifier of the first device.

3. The electronic device according to claim 1, The processor is further configured to: after sending the target information to the HomeKit client, detect that a second device is disconnected from the electronic device, the second device being any one of the target devices; Updating the device information of the second device in the target information to a device placeholder to update the target information; The communicator is further configured to: send the updated target information to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information, and the updated device list includes the identifier of the electronic device and the identifier of each device in the target devices except the second device.

4. The electronic device according to claim 1, wherein each target device comprises: at least one third device; or, The target devices include: at least one third device and at least one fourth device; Each of the third devices is directly connected to the electronic device, and each of the fourth devices is connected to the electronic device through the corresponding third device.

5. The electronic device according to claim 4, wherein the electronic device is connected to each of the third devices via an HDMI interface; In a case where each of the target devices includes the at least one fourth device, each of the third devices is connected to the corresponding third device via an HDMI interface. 6 . The electronic device according to claim 1 , wherein the device information further includes a type of the corresponding device; and the device list further includes the type of the electronic device and the types of the respective target devices.

7. The electronic device according to any one of claims 1 to 6, wherein the sum of the number of each target device and the number of the at least one device placeholder is equal to a first preset value; or, The number of the at least one device placeholder is equal to a second preset value.

8. A data processing method for an electronic device, the method comprising: After establishing a connection with a HomeKit client based on the HomeKit protocol, receiving a device list acquisition request sent by the HomeKit client, the device list acquisition request being used to instruct acquisition of device information of each target device connected to the electronic device, the device information including an identifier of the device; Determining device information of the electronic device and device information of each target device based on the device list acquisition request, wherein the device information includes an identifier of the corresponding device; Generate target information based on the device information of the electronic device, the device information of each target device, and at least one device placeholder, each device placeholder is used to indicate an unknown device to be connected, and the device information corresponding to each device placeholder is empty; The target information is sent to the HomeKit client, so that the HomeKit client displays a device list based on the target information, where the device list includes an identifier of the electronic device and an identifier of each target device.

9. The method according to claim 8, after sending the target information to the HomeKit client, the method further comprises: detecting that a first device establishes a connection with the electronic device; updating one of the device placeholders in the target information with the device information of the first device to update the target information; The updated target information is sent to the HomeKit client, so that the HomeKit client updates the device list based on the updated target information, where the updated device list includes an identifier of the electronic device, identifiers of each target device, and an identifier of the first device.

10. The method according to claim 8, after sending the target information to the HomeKit client, the method further comprises: detecting that a second device is disconnected from the electronic device, the second device being any one of the target devices; Updating the device information of the second device in the target information to a device placeholder to update the target information; Sending the updated target information to the HomeKit client so that the HomeKit client updates the device list based on the updated target information, the updated device list including the identifier of the electronic device and the identifier of each device in the target devices except the second device.

11. The method according to claim 8, wherein the device information further includes the type of the corresponding device; and the device list further includes the type of the electronic device and the type of each target device.

12. The method according to any one of claims 8 to 11, wherein the sum of the number of each target device and the number of the at least one device placeholder is equal to a first preset value; or, The number of the at least one device placeholder is equal to a second preset value.

13. A computer-readable non-volatile storage medium comprising: A computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the data processing method according to any one of claims 8 to 12 is implemented.

Images