A display device

Abstract

The application provides a display device, a controller of the display device relays signaling through a protocol stack process, performs bidirectional protocol conversion and forwarding between a smart home interconnection protocol and an inter-process communication protocol, thereby completing all signaling processing functions without media processing capability. Through cross-process cooperation between the protocol stack process, the application process and the browser process, the display device can establish an end-to-end media transmission channel in three business scenarios of camera wake-up, media query and camera active push, solve the structural constraint problem of separation of protocol capability and media processing capability under a multi-process architecture, enable the protocol stack process without media processing capability to complete all signaling functions defined in the smart home interconnection protocol specification, and directly transmit media data through peer-to-peer connection between the browser process and the camera, thereby avoiding additional resource overhead due to the lack of media processing capability of the protocol stack process.

Description

Technical Field

[0001] This application relates to the field of display device technology, and more particularly to a display device. Background Technology

[0002] The Matter protocol is a unified smart home connectivity standard released by the Connectivity Standards Alliance. The protocol defines three signaling flows: user-initiated viewing stream, battery / camera standby wake-up stream, and camera-initiated push stream. However, the specification only defines the function set interfaces and command formats, without specifying how signaling processing is implemented internally at the control end. It implicitly assumes that the control end is a single entity with communication and media asset processing capabilities.

[0003] For example, in the process architecture of a mobile phone, the protocol development kit provides a single-process architecture implementation. By integrating libraries into independent applications on the mobile phone and creating peer-to-peer connection objects, the application process itself becomes a media terminal. Thus, the mobile phone can implement the three signaling processes mentioned above.

[0004] However, smart terminals, represented by display devices, mostly adopt a multi-process operating system architecture. The protocol stack runs in the underlying system service process, handling only command-level communication and lacking media asset processing capabilities. Media asset processing is provided by the browser kernel, locked within a separate browser process, and cannot be extracted and used in other processes as a standalone library. This is an architectural constraint determined by the requirements of operating system-level security isolation, content protection, and stability. Furthermore, a single-process architecture relying solely on the protocol stack for end-to-end data transmission incurs significant resource overhead because the protocol stack in the display device lacks media asset processing capabilities. In other words, due to the constraints of the multi-process operating system architecture, display devices struggle to adapt to the diverse data interaction needs in end-to-end scenarios. Summary of the Invention

[0005] This application provides a display device to solve the problem that display devices, due to the constraints of multi-process operating system architecture, are difficult to adapt to various data interaction needs in end-to-end scenarios.

[0006] In a first aspect, this application provides a display device, comprising: monitor.

[0007] The communication device is configured to establish a communication connection with the camera based on a smart home interconnection protocol.

[0008] The controller is configured with a protocol stack process, an application process, and a browser process.

[0009] The protocol stack process is configured as follows: In response to a media asset configuration complete message received from the application process, the media asset configuration complete message is sent to the camera to trigger the camera to send back a first communication connection address to the protocol stack process. The media asset configuration complete message includes target media asset configuration information, which is media asset configuration information jointly supported by the display device and the camera.

[0010] In response to the first communication connection address received from the camera, the first communication connection address is forwarded to the application process.

[0011] The application process is also configured to write the first communication connection address into a memory.

[0012] The browser process is also configured as follows: The target communication connection address is determined based on the first communication connection address and the second communication connection address. The second communication connection address is a communication connection address supported by the display device, and the target communication connection address is a communication connection address jointly supported by the display device and the camera.

[0013] Based on the target communication connection address and the target media asset configuration information, a media asset transmission channel is established for receiving media asset data from the camera.

[0014] In this way, by configuring three independently running processes—a protocol stack process, an application process, and a browser process—in the controller of the display device, a layered processing architecture for camera media transmission under the smart home interconnection protocol is realized. The protocol stack process is responsible for low-level signaling interaction and address forwarding, the application process is responsible for data relay and storage writing, and the browser process is responsible for negotiating and matching communication addresses and finally establishing the transmission channel. Each of the three processes performs its own function, ensuring the real-time interaction between the protocol stack layer and the camera, while reducing the coupling of the browser process through the relay isolation of the application process, making the establishment of the media transmission channel more stable and reliable. At the same time, the design of writing the first communication connection address into memory allows the browser process to read and match addresses as needed, avoiding synchronization problems that may arise from direct inter-process communication and improving the success rate of establishing a media transmission connection between the display device and the camera.

[0015] In some feasible implementations, before the protocol stack process executes the response to the media asset configuration complete message received from the application process and sends the media asset configuration complete message to the camera to trigger the camera to send back the first communication connection address to the protocol stack process, it is further configured to: In response to a first signaling message received from the application process, a first wake-up command is sent to the camera. The first signaling message is generated by the application process when the display device initiates the camera wake-up process.

[0016] In response to the first wake-up media asset configuration information received from the camera, the first wake-up media asset configuration information is forwarded to the application process.

[0017] The application process is configured to write the first wake-up media configuration information into the memory.

[0018] The browser process is configured as follows: Upon detecting that the first wake-up media asset configuration information has been written to the memory, target media asset configuration information is determined based on the first wake-up media asset configuration information and the second wake-up media asset configuration information. The second wake-up media asset configuration information is media asset configuration information supported by the display device.

[0019] The application process sends a media asset configuration completion message containing the target media asset configuration information to the protocol stack process.

[0020] In this way, the protocol stack process sends a first wake-up command to the camera, triggering the camera to send back the first wake-up media asset configuration information. This information is then written to memory by the application process for the browser process to read and negotiate. This process separates wake-up negotiation from media asset configuration. The protocol stack process focuses on sending and receiving commands and maintaining the session, the application process is responsible for the persistent storage of configuration information, and the browser process completes the intersection matching based on the media asset configuration information supported by both parties and generates the target configuration. This division of labor allows the wake-up process to be seamlessly connected with the subsequent media asset transmission establishment process. It ensures the timely issuance and response of wake-up commands and ensures the accuracy of media asset configuration matching through independent negotiation by the browser process. This enables the display device to efficiently and accurately complete the negotiation of media asset parameters when actively waking up the camera, laying the foundation for the establishment of the subsequent media asset transmission channel.

[0021] In some feasible implementations, when the browser process detects that the first wake-up media asset configuration information has been written to memory, it determines the target media asset configuration information based on the first wake-up media asset configuration information and the second wake-up media asset configuration information, specifically configured as follows: Obtain the second media asset configuration information supported by the display device.

[0022] The target media asset configuration information is determined based on the first wake-up media asset configuration information and the second media asset configuration information.

[0023] In this way, the target configuration is determined by comparing the second media asset configuration information supported by the display device itself with the first wake-up media asset configuration information transmitted back by the camera. This method involves the browser process uniformly performing the intersection calculation of media asset configurations, rather than relying on the protocol stack or application process. This centralizes the media asset configuration negotiation logic within the browser process, facilitating subsequent maintenance and expansion. Simultaneously, the browser process directly obtains and matches the configuration information supported by the display device, reducing data transmission in intermediate steps, improving the efficiency and accuracy of configuration negotiation, and ensuring that the final determined target media asset configuration information is the optimal parameter combination supported by both devices. This provides a reliable configuration foundation for the stable establishment of the subsequent media asset transmission channel.

[0024] In some feasible implementations, the browser process executes the sending of a media asset configuration completion message containing the target media asset configuration information to the protocol stack process through the application process, specifically configured as follows: A media asset configuration completion message is created based on the target media asset configuration information and the second communication connection address.

[0025] The message indicating that the media asset configuration is complete is sent to the application process.

[0026] The application process is configured to send the media asset configuration completion message to the protocol stack process.

[0027] In this way, the browser process creates the message based on the target media asset configuration information and the second communication connection address, and then forwards it to the protocol stack process through the application process. The browser process is responsible for assembling the message content, ensuring the integrity of the configuration information and communication address. The application process acts as a relay, decoupling the browser process from the protocol stack process. The protocol stack process is then responsible for ultimately sending the message to the camera. This layered transmission design avoids the browser process directly manipulating the protocol stack interface, reducing the coupling between processes and allowing each process to evolve and maintain independently. At the same time, the application process's relay mechanism also provides the possibility for message caching and retransmission, which helps improve the reliability of message transmission and ensures that the camera can receive the media asset configuration completion message in a timely manner and send back the communication address.

[0028] In some feasible implementations, before the protocol stack process executes the first wake-up media asset configuration information received from the camera and forwards it to the application process, it is further configured to: In response to a first wake-up negotiation command received from the camera, a first communication session is created. The first wake-up negotiation command indicates that the camera has entered the wake-up preparation phase.

[0029] Send a first communication session creation message to the application process.

[0030] The application process is configured to send a communication preparation request to the browser process in response to a first communication session creation message received from the protocol stack process.

[0031] The browser process is configured to: in response to the communication preparation request received from the application process, obtain a second communication connection address supported by the display device.

[0032] Write the second communication connection address into the memory.

[0033] This approach, by completing the creation of the communication session and the preparation of the communication address during the wake-up negotiation phase, allows the prepared address information to be directly used during subsequent media asset configuration negotiation and transmission channel establishment, thus shortening the overall connection establishment time. The protocol stack process is responsible for session management, ensuring the stability of the underlying connection. The browser process obtains and stores the second communication connection address in advance, avoiding potential delays caused by temporarily obtaining it when needed. This pre-prepared mechanism makes the wake-up negotiation process between the display device and the camera smoother, and the cooperation between processes more orderly, improving the efficiency and timeliness of media asset transmission channel establishment in wake-up scenarios.

[0034] In some feasible implementations, the protocol stack process, in response to a media asset configuration completion message received from the application process, sends the media asset configuration completion message to the camera, thereby triggering the camera to send back a first communication connection address to the protocol stack process. Specifically, this is configured as follows: The first command transmission channel between the display device and the camera is detected. This first command transmission channel is a historical command transmission channel between the display device and the camera.

[0035] If the first command transmission channel is not detected, a second command transmission channel is created. The data capacity of the second command transmission channel is greater than that of the first command transmission channel.

[0036] The media asset configuration completion message is sent to the camera via the second instruction transmission channel.

[0037] This approach fully considers the large data volume of media asset configuration completion messages, which may contain numerous configuration parameters and address information. By dynamically selecting transmission channels, it ensures that messages are transmitted to the camera completely and efficiently. The mechanism for detecting historical channels can reuse existing connection resources, reducing unnecessary channel creation overhead. The strategy of automatically creating high-capacity channels when historical channels are unavailable guarantees reliable transmission of large data volumes. This adaptive channel selection mechanism enhances the flexibility and reliability of command transmission, avoids message fragmentation or transmission failures due to insufficient channel capacity, helps accelerate the media asset configuration negotiation process, and enables the camera to send back its communication connection address more quickly.

[0038] Secondly, this application provides another display device, including: monitor.

[0039] The communication device is configured to establish a communication connection with the camera based on a smart home interconnection protocol.

[0040] The controller is configured with a protocol stack process, an application process, and a browser process.

[0041] The browser process is configured as follows: Upon detecting that the first query media asset configuration information has been written to the memory, the application process and protocol stack process send a second communication connection address to the camera, causing the camera to return the first communication connection address. The second communication connection address is a communication connection address supported by the display device. The first query media asset configuration information is the media asset configuration information returned by the camera to the display device when the display device initiates the media asset query. The first query media asset configuration information is media asset configuration information jointly supported by the display device and the camera.

[0042] The protocol stack process is also configured to forward the first communication connection address to the application process in response to the first communication connection address received from the camera.

[0043] The application process is also configured to write the first communication connection address into a memory.

[0044] The browser process is further configured to determine a target communication connection address based on the first communication connection address and the second communication connection address. The target communication connection address is a communication connection address jointly supported by the display device and the camera.

[0045] Based on the target communication connection address and the target media asset configuration information, a media asset transmission channel is established for receiving media asset data from the camera.

[0046] In this architecture, after the browser process detects the writing of the first query media asset configuration information, it sends the second communication connection address to the camera through the application process and the protocol stack process. This allows it to obtain the first communication connection address returned by the camera and complete the address matching process. Under this architecture, the negotiation results of the media asset query are passed between processes via memory. The browser process drives subsequent processes by detecting changes in memory, achieving asynchronous decoupling between processes. The protocol stack process is responsible for the low-level signaling interaction with the camera, the application process handles data relay and storage, and the browser process focuses on address negotiation and channel establishment, with clear division of labor among the three layers. This design enables the display device to orderly complete the acquisition of configuration information, exchange of communication addresses, and establishment of transmission channels when actively querying the camera's media asset capabilities, improving the stability and maintainability of transmission channel establishment in media asset query scenarios.

[0047] In some feasible implementations, the protocol stack process is also configured to: In response to a first signaling message received from the application process, a first media asset query command is sent to the camera. The first signaling message is generated by the application process when the display device initiates the media asset query. The first signaling message includes second media asset configuration information supported by the display device.

[0048] In response to the first query media asset configuration information received from the camera, the first query media asset configuration information is forwarded to the application process. The first query media asset configuration information is media asset configuration information jointly supported by the display device and the camera.

[0049] The application process is configured to write the first queried media asset configuration information into a memory.

[0050] In this way, the protocol stack process sends a first media asset query command to the camera based on the first signaling message from the application process, and forwards the first query media asset configuration information returned by the camera to the application process for writing to memory. This process distributes the initiation and response processing of media asset queries across different processes, with the protocol stack process focusing on sending commands and receiving responses, ensuring real-time interaction with the camera. The application process is responsible for storing the configuration information, providing a data foundation for subsequent processing by the browser process. The first signaling message carries the second media asset configuration information supported by the display device, allowing the camera to directly match and return a configuration supported by both parties, reducing the number of subsequent negotiation rounds. This design simplifies the media asset query interaction process, enabling the display device to quickly obtain the media asset capabilities supported by the camera, preparing for subsequent communication address negotiation and transmission channel establishment.

[0051] In some feasible implementations, the application process is further configured as follows: In response to a detected media asset query operation, a media asset configuration information request instruction is sent to the browser process.

[0052] The browser process is configured to: in response to the configuration information request instruction received from the application process, obtain the second media asset configuration information supported by the display device.

[0053] The first signaling message is created based on the second media asset configuration information.

[0054] The application process sends the first signaling message to the protocol stack process.

[0055] In this way, the application process responds to detected media asset query operations by requesting configuration information from the browser process. The browser process then obtains the second media asset configuration information, creates the first signaling message, and sends it to the protocol stack process via the application process. This design clearly separates the responsibilities of responding to user operations from assembling signaling messages. The application process is responsible for detecting user operations and forwarding requests, while the browser process is responsible for obtaining configuration information and constructing messages. This separation allows the browser process to uniformly manage the media asset configuration information of the display device, facilitating configuration updates and maintenance. The application process, acting as a bridge between user interaction and the underlying protocol, reduces the coupling between business logic and the protocol layer.

[0056] Thirdly, this application also provides a display device, comprising: monitor.

[0057] The communication device is configured to establish a communication connection with the camera based on a smart home interconnection protocol.

[0058] The controller is configured with a protocol stack process, an application process, and a browser process.

[0059] The browser process is configured as follows: Upon detecting that the first push media asset configuration information has been written to the memory, the application process and the protocol stack process send a second communication connection address to the camera, causing the camera to send back the first communication connection address. The second communication connection address is a communication connection address supported by the display device. The first push media asset configuration information is the information sent to the display device when the camera initiates the media asset push.

[0060] The protocol stack process is configured to forward the first communication connection address to the application process in response to the first communication connection address received from the camera.

[0061] The application process is configured to write the first communication connection address into the memory.

[0062] The browser process is configured as follows: If the first communication connection address is detected to be written to the memory, a target communication connection address is determined based on the first communication connection address and the second communication connection address. The target communication connection address is a communication connection address jointly supported by the display device and the camera.

[0063] Based on the target communication connection address and the target media asset configuration information, a media asset transmission channel is established for receiving media asset data from the camera. The target media asset configuration information is the intersection of the first push media asset configuration information and the second media asset configuration information, where the second media asset configuration information is the media asset configuration information supported by the display device.

[0064] In this way, configuration negotiation and address negotiation in media asset push scenarios are performed step by step. The configuration information actively pushed by the camera is written to the memory via the protocol stack and application process. After the browser process detects it, it initiates the communication address exchange process. Address matching and channel establishment are completed only after the first communication connection address of the camera is obtained. This signaling relay architecture based on the protocol stack process makes the data chain in push scenarios more orderly. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram illustrating the operation scenario between a display device and a control device provided in some embodiments of this application.

[0067] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application.

[0068] Figure 3 This is a schematic diagram illustrating the software configuration of a display device provided in some embodiments of this application.

[0069] Figure 4 This is an overall flowchart of a signaling relay method performed by a display device according to some embodiments of this application.

[0070] Figure 5 A flowchart illustrating the instruction transmission channel detection process provided in some embodiments of this application.

[0071] Figure 6 This is a schematic diagram of signaling interaction between a display device and a wake-up initiator, provided in some embodiments of this application.

[0072] Figure 7 A flowchart illustrating the camera pre-wake-up process provided for some embodiments of this application.

[0073] Figure 8 A flowchart illustrating the determination of target media asset configuration information provided in some embodiments of this application.

[0074] Figure 9 A flowchart illustrating the creation of media asset configuration completion message provided in some embodiments of this application.

[0075] Figure 10 This is an interactive flowchart of a display device acting as a media asset query initiator, as provided in some embodiments of this application.

[0076] Figure 11 A flowchart for creating a first signaling message provided for some embodiments of this application.

[0077] Figure 12 A flowchart illustrating the first query media asset configuration information negotiation process provided for some embodiments of this application.

[0078] Figure 13 The following is an interactive flowchart of a camera acting as a media asset push initiator, as provided in an embodiment of this application. Detailed Implementation

[0079] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0080] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0081] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0082] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0083] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0084] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0085] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, a user can operate the display device 200 via touch operation, a mobile terminal 300, and a control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc.

[0086] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0087] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application that controls the display device 200.

[0088] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0089] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0090] Figure 2 Provided for some embodiments of this application Figure 1Hardware configuration block diagram of display device 200.

[0091] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0092] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0093] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0094] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0095] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0096] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0097] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0098] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface 280 receives the user input commands through the graphical user interface (GUI).

[0099] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0100] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0101] To enable user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running an application. The operating system also allows users to interact with the display device 200.

[0102] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0103] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0104] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0105] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0106] like Figure 3 As shown, Figure 3 The diagram below illustrates the software configuration of a display device according to some embodiments of this application. In some embodiments, the system of the display device 200 can be divided into three layers, from top to bottom: the application layer, the middleware layer, and the hardware layer.

[0107] The application layer mainly includes TV applications and application frameworks. The applications are mainly browser-based applications, such as HTML5 apps, and native apps. An application framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, and the user interface for these functions (toolbar, status bar, menu, dialog box).

[0108] Native apps can support online or offline access, push notifications, or access to local resources.

[0109] The middleware layer includes various television protocols, multimedia asset protocols, and system components. Middleware can use the basic services (functions) provided by system software to connect different parts of application systems or different applications on the network, achieving resource sharing and function sharing.

[0110] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chips, with the specific logic implemented by each chip. The drivers mainly include: audio drivers, display drivers, Bluetooth drivers, camera drivers, Wi-Fi drivers, USB drivers, HDMI drivers, sensor drivers (such as fingerprint sensors, temperature sensors, pressure sensors, etc.), and power drivers.

[0111] based on Figures 2-3 The software and hardware architectures shown enable the display device 200 to interact with remote cameras and other devices possessing communication and data transmission capabilities at both the signaling and media asset data levels. However, because the display device 200 typically employs a multi-architecture approach, such as relying on the collaboration between protocol stack processes, browser processes, and application processes to achieve interaction with remote cameras, a single process within the display device 200 cannot simultaneously possess both communication and media asset data transmission capabilities. Therefore, it is difficult to fully support the complete functionality defined in the Matter protocol for smart home interconnection.

[0112] To address the above issues, this application provides a display device 200, which includes a display 260 and a controller 250. The controller 250 is configured with a protocol stack process, a browser process, and an application process. Figure 4 This is an overall flowchart of a signaling relay method performed by a display device according to some embodiments of this application.

[0113] like Figure 4 As shown, in some embodiments, the protocol stack process is configured as follows: S400: In response to the media asset configuration complete message received from the application process, send the media asset configuration complete message to the camera to trigger the camera to send back the first communication connection address to the protocol stack process.

[0114] The media asset configuration completion message includes target media asset configuration information, which is media asset configuration information jointly supported by the display device and the camera.

[0115] In some examples, media asset configuration information includes parameters related to media asset data transmission, such as video encoding format, video resolution, video frame rate, audio encoding format, audio sampling rate, and audio bit depth. The target media asset configuration information is the optimal combination of configurations selected within the intersection of the capabilities of the display device 200 and the camera.

[0116] The media asset configuration completion message contains media asset configuration information supported by the display device 200 provided by the browser process; therefore, the data volume of the media asset configuration completion message is larger than that of ordinary interactive messages. To ensure the transmission quality of the media asset configuration completion message, a TCP command transmission channel can be used to transmit it. However, during the historical interactions between the display device 200 and the camera, a command transmission channel established based on the UDP protocol may remain. When the protocol stack sends large amounts of media asset configuration information to the camera, it will prioritize using the historically established UDP command transmission channel, resulting in incomplete transmission of the media asset configuration information.

[0117] Figure 5 This is a flowchart illustrating the instruction transmission channel detection process provided in an embodiment of this application.

[0118] like Figure 5 As shown, in response to the media asset configuration complete message received from the application process, the protocol stack process sends the media asset configuration complete message to the camera, thereby triggering the camera to send back the first communication connection address to the protocol stack process, specifically configured as follows: S500: Detect the first instruction transmission channel between the display device and the camera.

[0119] The first instruction transmission channel is the historical instruction transmission channel between the display device and the camera.

[0120] S501: If the first instruction transmission channel is not detected, a second instruction transmission channel is created.

[0121] The data capacity of the second instruction transmission channel is greater than that of the first instruction transmission channel.

[0122] S502: Send the media asset configuration completion message to the camera based on the second instruction transmission channel.

[0123] In some examples, before sending the media asset configuration complete message, the protocol stack process actively checks whether a historical UDP command transmission channel, i.e., the first command transmission channel, exists between the display device 200 and the camera. If the first command transmission channel is not detected, a second command transmission channel is created based on the TCP protocol, i.e., a second command transmission channel that supports larger data volume transmission. Then, the protocol stack process sends the media asset configuration complete message to the camera through the second command channel.

[0124] Understandably, during the process of detecting the command transmission channel, if the protocol stack process detects the first command transmission channel, it will prioritize transmitting data through the first command transmission channel. To avoid this situation, the protocol stack process sends a pre-wake command to the camera before sending the wake-up command to the camera. This pre-wake command disconnects the historical command transmission channel (i.e., the first command transmission channel) between the display device 200 and the camera.

[0125] Furthermore, after the historical command transmission channel between the display device 200 and the camera is terminated based on the pre-wake-up instruction, the protocol stack process maintains a status flag indicating that the user has indicated that a TCP command transmission channel can be forcibly created. Thus, when the protocol stack process transmits the media configuration completion message, it can determine whether a second command transmission channel can be created by reading this status flag.

[0126] This ensures that the protocol stack process actively creates a second command transmission channel based on the TCP protocol when sending a large configuration completion message to the camera. This, in turn, guarantees the transmission quality of the media asset configuration completion message.

[0127] In addition, during the pre-wake-up phase, the first pre-wake-up instruction also includes multiple sub-wake-up instructions for the initial negotiation between the display device 200 and the camera. Based on the received multiple sub-wake-up instructions, the camera can allocate internal hardware resources to improve the efficiency of subsequent wake-up / initialization.

[0128] S401: In response to the first communication connection address received from the camera, forward the first communication connection address to the application process.

[0129] The first communication connection address is the network address information of the camera device, including its IP address, port number, transport protocol type (UDP / TCP), and a list of candidate addresses (ICE Candidates, specifically including host candidate, srflx candidate, and relay candidate) for establishing peer-to-peer connections. This communication connection address is used to locate and connect to the camera device during peer-to-peer network transmission.

[0130] In some examples, after receiving the media asset configuration completion message, the camera knows that the media asset configuration information negotiation with the display device 200 has been completed, and can then further negotiate the communication connection information, that is, feed back the first communication connection address to the protocol stack process.

[0131] After receiving the first communication connection address, the protocol stack process forwards it to the application process. The protocol stack process is configured with a requester cluster for the initiator and a provider cluster for the receiver. In the current process, when the entire flow is initiated by display device 200, the protocol stack process sends and receives commands with the camera through the requester cluster.

[0132] The application process is also configured as follows: S402: Write the first communication connection address into the memory.

[0133] In some examples, the application process writes a first communication connection address into memory to facilitate subsequent calls to the first communication connection address to establish a media asset data transmission channel with the camera. This also helps the browser process determine the intersection of the first communication connection address and a second communication connection address supported by the display device 200.

[0134] The browser process is also configured as follows: S403: Determine the target communication connection address based on the first communication connection address and the second communication connection address.

[0135] Wherein, the second communication connection address is a communication connection address supported by the display device, and the target communication connection address is a communication connection address jointly supported by the display device and the camera.

[0136] In some examples, the browser process can detect the first communication connection address being written to memory. The browser process performs interactive connectivity testing on the first and second communication connection addresses, using the ICE (Interactive Connectivity Establishment) protocol to pair candidate addresses and test connectivity, determining the address pair that both support and can communicate successfully as the target communication connection address. The target communication connection address is the foundational address information for subsequently establishing the media asset transmission channel.

[0137] S404: Based on the target communication connection address and the target media asset configuration information, establish a media asset transmission channel for receiving media asset data from the camera.

[0138] In some examples, the media asset transmission channel is an end-to-end peer-to-peer connection established between the display device and the camera: a WebRTC Peer-to-Peer channel, implemented based on the WebRTC peer-to-peer network transport protocol. Once the media asset transmission channel is established, the audio and video media asset data transmitted from the camera to the display device is directly transmitted between the camera and the browser process via the P2P channel, without passing through the protocol stack process. The protocol stack process does not participate in any audio or video data encoding / decoding or forwarding processing, thus eliminating unnecessary media asset data copying and forwarding delays.

[0139] Figure 6 This is a signaling interaction diagram illustrating a display device acting as a wake-up initiator, provided in some embodiments of this application. This scenario applies to the complete process where a display device actively sends a wake-up command to a camera in standby or low-power mode, triggering the camera to wake up from standby mode and establish a media asset transmission channel.

[0140] like Figure 6 As shown, before the controller's protocol stack process sends a media asset configuration complete message to the camera in response to the media asset configuration complete message received from the application process, it also performs the following wake-up negotiation steps: S600: In response to the first signaling message received from the application process, a first wake-up command is sent to the camera.

[0141] The first signaling message is a signaling message generated by the application process when the display device acts as the initiator of camera wake-up.

[0142] In some examples, a user triggers a camera wake-up operation by clicking on a camera device card that is in standby or low-power mode, through a list of smart home devices displayed in the device's graphical user interface. Upon detecting this operation, the application process generates a first signaling message based on the camera device identifier carried in the operation. This first signaling message is a wake-up request message that does not contain session description information; it is only used to trigger the camera to wake up from standby mode and does not carry complete media asset session parameters. The first signaling message is sent to the protocol stack process via the MQTT fromApp / solicitOffer topic.

[0143] After receiving the first signaling message, the protocol stack process parses the camera device identifier and target operation type in the message, and constructs a first wake-up command conforming to the smart home interconnection protocol format, such as SolicitOffer. The first wake-up command is sent to the camera through the smart home interconnection protocol's command transmission channel. The first wake-up command is used to notify the camera device to exit standby mode, initialize hardware modules such as the image sensor and encoder, and prepare for subsequent media asset transmission.

[0144] Figure 7A flowchart illustrating the camera pre-wake-up process provided in this application embodiment.

[0145] like Figure 7 As shown, before the protocol stack process executes the response to the first wake-up media asset configuration information received from the camera and forwards the first wake-up media asset configuration information to the application process, it is also configured as follows: S700: In response to a first wake-up negotiation command received from the camera, a first communication session is created.

[0146] The first wake-up negotiation instruction is used to indicate that the camera has entered the wake-up preparation stage.

[0147] S701: Send a first communication session creation message to the application process.

[0148] The application process is configured as follows: S702: In response to the first communication session creation message received from the protocol stack process, a communication preparation request is sent to the browser process.

[0149] The browser process is configured as follows: S703: In response to the communication preparation request received from the application process, obtain the second communication connection address supported by the display device.

[0150] S704: Write the second communication connection address into the memory.

[0151] In some embodiments, before sending the first wake-up command to the camera, the protocol stack process sends a first pre-wake-up command to the camera, so that the camera pre-allocates hardware resources and clears the residual historical UDP command transmission channel between the display device 200 and the camera. After the pre-wake-up negotiation, the camera enters the initialization and wake-up phase and sends a first wake-up negotiation command back to the protocol stack process to indicate to the display device 200 that the camera has started the wake-up process.

[0152] After receiving the first wake-up negotiation instruction, the protocol stack process creates a wake-up session related to this camera wake-up process to maintain the current camera wake-up procedure. The wake-up session is used to maintain context information for this media asset transmission process, including the session identifier WebRTCSessionID, camera device identifier Node ID, endpoint identifier Endpoint ID, media asset configuration information, and communication connection address. Furthermore, the protocol stack process forwards the first wake-up negotiation instruction to the application process, which then initiates a communication preparation request to the browser process.

[0153] After receiving a communication preparation request, the browser process can obtain a second communication connection address supported by the display device 200. For example, after receiving the communication preparation request, the browser process creates a peer connection object RTCPeerConnection and begins collecting available network address candidates (ICE Candidates) on the display device. These network address candidates include: a host candidate (the IP address and port of the display device's local network interface), a server reflection address (srflx candidate, the public IP address and port obtained through NAT mapping from the STUN server), and a relay candidate (the relay transmission address assigned by the TURN relay server).

[0154] In this way, by utilizing the camera's initialization time, the display device 200 simultaneously collects the second communication connection address, which helps reduce the total time for determining the target communication connection address in subsequent processes, thereby reducing the overall time required to wake up the camera. The browser process can write the second communication connection address into memory to facilitate the subsequent process of determining the target communication address.

[0155] In some examples, the process of the browser process collecting the second communication connection address also includes the time spent waiting for the camera to initialize and wake up. The waiting time can be preset to avoid prolonged waiting periods and unresponsiveness due to camera initialization failure.

[0156] S601: In response to the first wake-up media asset configuration information received from the camera, forward the first wake-up media asset configuration information to the application process.

[0157] In some examples, after the camera completes hardware initialization and encoder startup, it proactively generates Session Description Offer (SDP Offer) information and sends the first wake-up media asset configuration information, containing the session description information, to the media requester (WebRTC TransportRequestor Cluster) of the protocol stack process via the smart home interconnection protocol. The first wake-up media asset configuration information contains complete media asset capability parameters supported by the camera device, such as a list of supported video encoding formats, resolution options, frame rate range, a list of audio encoding formats, sampling rate options, etc.

[0158] The protocol stack process receives the first wake-up media asset configuration information actively sent by the camera through the callback interface of the Requestor Cluster. The protocol stack process updates the corresponding communication session context, records the media asset configuration information, and then converts the first wake-up media asset configuration information, including the media asset capability parameters supported by the camera, into an inter-process communication message format, and forwards it to the application process through the MQTT toApp / requestOffer topic.

[0159] The application process is configured as follows: S602: Write the first wake-up media configuration information into the memory.

[0160] In some examples, the first wake-up media configuration information written to memory can be used to confirm the target wake-up media configuration information.

[0161] The browser process is configured as follows: S603: When the first wake-up media configuration information is detected to be written to the memory, the target media configuration information is determined based on the first wake-up media configuration information and the second wake-up media configuration information.

[0162] The second wake-up media asset configuration information is the media asset configuration information supported by the display device.

[0163] In some examples, the browser process performs an intersection match between the first wake-up media asset configuration information and the second wake-up media asset configuration information, and selects the optimal configuration combination as the target media asset configuration information within the capability intersection range according to a preset priority strategy. The priority strategy can be set as needed, and this application embodiment does not limit the setting of the priority strategy, so it will not be described in detail here.

[0164] Figure 8 A flowchart for determining target media asset configuration information provided in an embodiment of this application.

[0165] like Figure 8 As shown, when the browser process detects that the first wake-up media asset configuration information has been written to memory, it determines the target media asset configuration information based on the first wake-up media asset configuration information and the second wake-up media asset configuration information, which is specifically configured as follows: S800: Obtain the second media asset configuration information supported by the display device.

[0166] S801: Determine the target media asset configuration information based on the first wake-up media asset configuration information and the second media asset configuration information.

[0167] In some examples, the browser process detects that the first wake-up media asset configuration information has been written by listening for write events in the memory. The browser process then obtains the second wake-up media asset configuration information supported by the display device, which is the display device's media asset processing capability parameters, including decodeable video encoding formats, supported video resolution range, supported audio encoding formats, audio sampling rate, and bit depth.

[0168] Furthermore, the browser process determines the target media asset configuration information that is supported by both ends by retrieving the first wake-up media asset configuration information and the second wake-up media asset configuration information. That is, both the display device 200 and the camera send and play media asset data based on the target media asset configuration information.

[0169] S604: The application process sends a media asset configuration completion message containing the target media asset configuration information to the protocol stack process.

[0170] In some examples, after determining the target media asset configuration information, the browser process can create a media asset configuration completion message based on the target media asset configuration information, and send the media asset configuration completion message to the camera through the application process and protocol stack process, so that the camera transmits data according to the media asset format indicated by the target media asset configuration information when transmitting media asset data.

[0171] Figure 9 A flowchart illustrating the media asset configuration completion message provided in this embodiment of the application.

[0172] like Figure 9 As shown, the browser process executes the sending of a media asset configuration completion message containing the target media asset configuration information to the protocol stack process through the application process, specifically configured as follows: S900: Create a media asset configuration completion message based on the target media asset configuration information and the second communication connection address.

[0173] S901: Send the media asset configuration completion message to the application process.

[0174] The application process is configured as follows: S902: Send the media asset configuration completion message to the protocol stack process.

[0175] In some examples, the browser process combines the target media asset configuration information with the previously collected second communication connection address into a media asset configuration complete message. That is, the media asset configuration complete message includes both a response to the first wake-up media asset configuration information sent by the camera and a push notification of the second communication connection address it supports. The browser process sends the media asset configuration complete message to the application process. The application process then forwards the media asset configuration complete message to the protocol stack process via the MQTT fromApp / provideAnswer topic.

[0176] Figure 10 This document provides an interactive flowchart illustrating the interaction between a display device and a camera device as the initiator of a media asset query, as provided in some embodiments of this application. This scenario applies to the complete process of a display device establishing a media asset transmission channel with a camera device, where the user actively clicks on the display device to view the online camera device.

[0177] like Figure 10 As shown, when the browser process detects that the first query media asset configuration information has been written to memory, it executes the following media asset transmission channel establishment process: S1000: Upon detecting that the first query media asset configuration information has been written to the memory, the application process and the protocol stack process send the second communication connection address to the camera so that the camera can send back the first communication connection address.

[0178] Wherein, the second communication connection address is a communication connection address supported by the display device. The first queried media asset configuration information is the media asset configuration information returned by the camera to the display device when the display device initiates the media asset query. The first queried media asset configuration information is media asset configuration information jointly supported by the display device and the camera.

[0179] In some examples, after the browser process detects that the first query media configuration information has been written by listening for write events in the memory, it creates a peer connection object RTCPeerConnection, collects network address candidates ICE Candidates available on the display device, packages the second communication connection address into a signaling message, and forwards it to the camera device level by level through the application process and the protocol stack process.

[0180] The protocol stack process is also configured as follows: S1001: In response to the first communication connection address received from the camera, forward the first communication connection address to the application process.

[0181] In some examples, after receiving the second communication connection address, the camera completes the session negotiation preparation for the peer connection, collects its own network address candidates (ICE Candidates), packages the collected address candidates into a first communication connection address, and sends it to the protocol stack process through the callback interface of the smart home interconnection protocol. The protocol stack process converts it into an inter-process communication message format and forwards it to the application process through the MQTT topic toApp / requestICECandidates.

[0182] The application process is also configured as follows: S1002: Write the first communication connection address into the memory.

[0183] In some examples, the first communication connection address is written to memory for later use, in order to facilitate subsequent target communication address filtering and the establishment of media asset transmission channels.

[0184] The browser process is also configured as follows: S1003: Determine the target communication connection address based on the first communication connection address and the second communication connection address.

[0185] The target communication connection address is a communication connection address jointly supported by the display device and the camera.

[0186] In some examples, after the browser process detects that the first communication connection address has been written to memory, it performs an ICE pairing operation on the first communication connection address and the second communication connection address, performs pairing and connectivity tests on the candidate addresses of both parties, and determines the address pair that can communicate successfully as the target communication connection address.

[0187] S1004: Based on the target communication connection address and the target media asset configuration information, establish a media asset transmission channel for receiving media asset data from the camera.

[0188] In some examples, the target media asset configuration information is the optimal parameter combination determined through negotiation between the first queried media asset configuration information and the second media asset configuration information supported by the display device. After ICE pairing is completed, the peer-to-peer connection between the browser process and the camera enters a ready state, and the media asset transmission channel is established. The camera sends audio and video media asset data directly to the browser process through the media asset transmission channel, and the browser process decodes and renders it on the display.

[0189] It is understandable that this scenario involves the display device 200 proactively initiating a media asset data query to the camera based on user actions. Specifically, the camera only sends the first query for media asset configuration information back to the display device 200 after receiving the media asset query command from the display device 200. Furthermore, the display device 200 sends a media asset configuration information request command to the browser process based on user actions detected by the application process on the user interface, thus triggering the media asset query scenario.

[0190] Figure 11 A flowchart for creating a first signaling message provided in an embodiment of this application.

[0191] like Figure 11 As shown, the application process is also configured as follows: S1100: In response to the detected media asset query operation, a media asset configuration information request instruction is sent to the browser process.

[0192] In some examples, a user clicks on an online camera device card in the graphical user interface of the display device, triggering a media asset query operation. Upon detecting this operation, the application process determines that the user is actively querying media asset data for existing online cameras. It then sends a media asset configuration information request command to the browser process, requesting the browser to provide the media asset configuration information supported by the display device and prepare to generate session description information.

[0193] The browser process is configured as follows: S1101: In response to the configuration information request instruction received from the application process, obtain the second media asset configuration information supported by the display device.

[0194] In some examples, after receiving a configuration information request instruction, the browser process queries and displays the media asset processing capabilities supported by the device's hardware and software, including a list of decodeable video encoding formats, a range of supported video resolutions, a range of supported video frame rates, a list of supported audio encoding formats, and complete capability parameters such as audio sampling rate and bit depth, which are then summarized into the second media asset configuration information.

[0195] S1102: Create the first signaling message based on the second media asset configuration information.

[0196] In some examples, the browser process encapsulates the second media asset configuration information into a first signaling message. In addition to the second media asset configuration information, the first signaling message also includes the target camera device identifier and the operation type identifier. Simultaneously, the browser process creates a peer-to-peer connection object, RTCPeerConnection, and generates corresponding session description information based on the second media asset configuration information, which is also encapsulated within the first signaling message.

[0197] S1103: The first signaling message is sent to the protocol stack process through the application process.

[0198] In some examples, the browser process first sends the first signaling message to the application process, and the application process then forwards the first signaling message to the protocol stack process through the MQTT fromApp / provideOffer topic.

[0199] Figure 12 A flowchart illustrating the negotiation process for the first query media asset configuration information provided in this application embodiment.

[0200] like Figure 12 As shown, the protocol stack process is also configured as follows: S1200: In response to the first signaling message received from the application process, send the first media asset query instruction to the camera.

[0201] The first signaling message is generated by the application process when the display device initiates a media asset query. The first signaling message includes second media asset configuration information supported by the display device.

[0202] In some examples, after the protocol stack process receives the first signaling message sent by the application process via MQTT, it parses the second media asset configuration information and session description information in the message and constructs a first media asset query instruction that conforms to the smart home interconnection protocol format.

[0203] Before sending the first media query command, the protocol stack process can send a pre-query command to the camera to pre-allocate hardware resources to the camera and pre-clear any residual command transmission channels between the display device 200 and the camera, thus providing a basis for the subsequent forced establishment of a data channel with a larger data capacity.

[0204] It is understandable that the first signaling message contains the second media asset configuration information supported by the display device 200 provided by the browser, meaning that the data volume of the first media asset query command exceeds the data capacity of the UDP command transmission channel. Therefore, when the protocol stack process sends the first media asset query command to the camera, it can refer to the historical command transmission channel detection process and the TCP command transmission channel establishment process in the camera wake-up scenario, which will not be elaborated here.

[0205] S1201: In response to the first query media asset configuration information received from the camera, forward the first query media asset configuration information to the application process.

[0206] The first query media asset configuration information is the media asset configuration information jointly supported by the display device and the camera.

[0207] In some examples, after receiving the first media asset query command, the camera parses the session description information and the second media asset configuration information. The camera matches the second media asset configuration information of the display device with its own supported media asset configuration information, determining the media asset configuration information that both support as the first queried media asset configuration information. After completing the media asset configuration negotiation, the camera responds to the protocol stack process via the smart home interconnection protocol, carrying the WebRTC session identifier, stream identifier, and the first queried media asset configuration information. Upon receiving this, the protocol stack process forwards it to the application process via the MQTT toApp / requestOffer topic.

[0208] The application process is configured as follows: S1202: Write the first queried media asset configuration information into the memory.

[0209] In some examples, after receiving the first query media asset configuration information, the application process writes it to memory. This allows the browser process to retrieve the first query media asset configuration information from memory to establish a media asset transmission channel.

[0210] In some examples, after the browser process detects that the first query media asset configuration information has been written to memory, it obtains the second communication connection address supported by the display device 200, and then calls the application process and protocol stack process level by level to negotiate communication with the camera to determine the final target communication connection address. Finally, the browser process establishes a media asset data channel based on the target communication connection address and the target media asset configuration information.

[0211] It should be noted that the communication negotiation process is not limited to one negotiation. After multiple negotiations, the final target communication connection address can be determined by the display device 200.

[0212] It is understandable that the technical implementations in the media asset query scenario and the wake-up scenario can be mutually referenced. Therefore, the implementations that appear in the wake-up scenario can also be reused in the media asset query scenario according to the actual scenario requirements. That is, the technical solutions that appear in the wake-up scenario will not be described in detail here.

[0213] Figure 13 The following is an interactive flowchart of a camera acting as a media asset push initiator, as provided in an embodiment of this application.

[0214] like Figure 13 As shown, the browser process is configured as follows: S1300: Upon detecting that the first push media asset configuration information has been written to the memory, the application process and the protocol stack process send the second communication connection address to the camera so that the camera can send back the first communication connection address.

[0215] The second communication connection address is a communication connection address supported by the display device. The first push media asset configuration information is information sent to the display device when the camera acts as the initiator of media asset push.

[0216] In some examples, after detecting a preset push event, the camera proactively initiates the media asset push process. The camera generates session description information, packages the session description information and the media asset capability parameters supported by the camera into the first push media asset configuration information, and sends it to the protocol stack process of the display device through the ProvideOffer command of the smart home interconnection protocol.

[0217] The protocol stack process receives the first push media asset configuration information through the Provider Cluster callback interface and performs stream verification. Since the display device acts as a signaling relay rather than the actual media endpoint, the protocol stack process adopts a pass-through strategy, directly passing the stream verification callback without intercepting or modifying any stream allocation or media asset parameters. The protocol stack process creates a communication session and assigns a session identifier, then forwards the first push media asset configuration information to the application process via the MQTT toApp / provideOffer topic.

[0218] After receiving the first push media asset configuration information forwarded by the protocol stack process, the application process writes the information to a designated location in the memory. Simultaneously, based on the event type identifier carried in the first push media asset configuration information (e.g., doorbell event, motion detection event, package arrival event), the application process launches the corresponding user interface. When the user performs a confirmation operation in the user interface triggered by the push event, such as clicking the view button or automatically opening the preview page, the application process, in response to the user action or automatic policy, generates a session confirmation message and sends it to the protocol stack process via the MQTT fromApp / provideOfferResponse topic. The session confirmation message contains a session identifier assigned by the protocol stack process, used to associate this push session.

[0219] The browser process detects that the first push media asset configuration information has been written by listening for write events in the memory. The browser process creates a peer connection object RTCPeerConnection and sets the session description information from the first push media asset configuration information as the remote description of the peer connection object. Based on the media asset capability parameters supported by the camera in the first push media asset configuration information, the browser process matches the capabilities with the second media asset configuration information supported by the display device itself to determine the target media asset configuration information that both support.

[0220] The browser process packages the generated SDP Answer and the collected second communication connection address into a media asset configuration response message, which is then forwarded to the camera device through the application process and the protocol stack process. The application process sends the media asset configuration response message to the protocol stack process via the MQTT fromApp / provideAnswer topic. Upon receiving the message, the protocol stack process sends an Answer command to the camera via the smart home interconnection protocol, carrying the SDP Answer and the second communication connection address from the display device. Upon receiving the Answer command, the camera completes the peer-to-peer connection negotiation, collects its own ICE candidate list, and returns it as the first communication connection address to the protocol stack process.

[0221] The protocol stack process is configured as follows: S1301: In response to the first communication connection address received from the camera, forward the first communication connection address to the application process.

[0222] In some examples, the protocol stack process receives the first communication connection address returned by the camera through the ICE candidate callback interface, converts it into an inter-process communication message format, and then forwards it to the application process through the MQTT topic toApp / provideICECandidates.

[0223] The application process is configured as follows: S1302: Write the first communication connection address into the memory.

[0224] In some examples, the application process writes the first communication connection address to memory so that the browser process can access it.

[0225] The browser process is configured as follows: S1303: If the first communication connection address is detected to be written to the memory, a target communication connection address is determined based on the first communication connection address and the second communication connection address.

[0226] The target communication connection address is a communication connection address jointly supported by the display device and the camera.

[0227] In some examples, the browser process performs an ICE pairing operation on the first communication connection address and the second communication connection address, performs pairing and connectivity checks on the candidate addresses of both parties, and determines the address pair that can successfully communicate as the target communication connection address.

[0228] The ICE candidate exchange between the two parties adopts the Trickle ICE mechanism, where newly added candidate addresses are forwarded bidirectionally through the protocol stack process and the application process in real time, accelerating the connectivity detection and connection establishment process.

[0229] S1304: Based on the target communication connection address and the target media asset configuration information, establish a media asset transmission channel for receiving media asset data from the camera.

[0230] The target media asset configuration information is the intersection of the first pushed media asset configuration information and the second media asset configuration information, and the second media asset configuration information is the media asset configuration information supported by the display device.

[0231] In some examples, the target media asset configuration information is the intersection of the first pushed media asset configuration information and the second media asset configuration information supported by the display device, selected as the optimal parameter combination according to a preset priority strategy. After ICE pairing is completed, the peer-to-peer connection between the browser process and the camera enters a ready state, and the media asset transmission channel is established. The camera sends audio and video media asset data directly to the browser process through the media asset transmission channel, and the browser process decodes and renders it on the display.

[0232] In the technical solution provided in this application embodiment, the controller uses a protocol stack process as a signaling relay to perform bidirectional protocol conversion and forwarding between the smart home interconnection protocol and the inter-process communication protocol, thereby completing all signaling processing functions even without media asset processing capabilities. Through cross-process collaboration between the protocol stack process, application process, and browser process, the display device can establish an end-to-end media asset transmission channel in three business scenarios: camera wake-up, media asset query, and camera active push. This solves the structural constraint problem of separating protocol capabilities and media asset processing capabilities under a multi-process architecture, enabling the protocol stack process, which lacks media asset processing capabilities, to complete all signaling functions defined by the smart home interconnection protocol specification. Furthermore, media asset data is directly transmitted through a peer-to-peer connection between the browser process and the camera, avoiding additional resource overhead caused by the protocol stack process's lack of media asset processing capabilities.

[0233] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: monitor; The communication device is configured to establish a communication connection with the camera based on a smart home interconnection protocol; The controller is configured with a protocol stack process, an application process, and a browser process. The protocol stack process is configured as follows: In response to a media asset configuration completion message received from the application process, the media asset configuration completion message is sent to the camera to trigger the camera to send back a first communication connection address to the protocol stack process; wherein, the media asset configuration completion message includes target media asset configuration information, which is media asset configuration information jointly supported by the display device and the camera; In response to the first communication connection address received from the camera, the first communication connection address is forwarded to the application process; The application process is also configured to write the first communication connection address into a memory. The browser process is also configured as follows: A target communication connection address is determined based on the first communication connection address and the second communication connection address; wherein, the second communication connection address is a communication connection address supported by the display device, and the target communication connection address is a communication connection address jointly supported by the display device and the camera; Based on the target communication connection address and the target media asset configuration information, a media asset transmission channel is established for receiving media asset data from the camera.

2. The display device according to claim 1, characterized in that, Before the protocol stack process executes the response to the media asset configuration completion message received from the application process and sends the media asset configuration completion message to the camera to trigger the camera to send back the first communication connection address to the protocol stack process, it is also configured as follows: In response to a first signaling message received from the application process, a first wake-up command is sent to the camera; wherein the first signaling message is a signaling message generated by the application process when the display device is the initiator of camera wake-up; In response to the first wake-up media asset configuration information received from the camera, the first wake-up media asset configuration information is forwarded to the application process; The application process is configured to write the first wake-up media configuration information into a memory. The browser process is configured as follows: If the first wake-up media configuration information is detected to be written to the memory, target media configuration information is determined based on the first wake-up media configuration information and the second wake-up media configuration information; wherein, the second wake-up media configuration information is media configuration information supported by the display device; The application process sends a media asset configuration completion message containing the target media asset configuration information to the protocol stack process.

3. The display device according to claim 2, characterized in that, When the browser process detects that the first wake-up media asset configuration information has been written to memory, it determines the target media asset configuration information based on the first wake-up media asset configuration information and the second wake-up media asset configuration information, specifically configured as follows: Obtain the second media asset configuration information supported by the display device; The target media asset configuration information is determined based on the first wake-up media asset configuration information and the second media asset configuration information.

4. The display device according to claim 2, characterized in that, The browser process executes the sending of a media asset configuration completion message containing the target media asset configuration information to the protocol stack process via the application process, specifically configured as follows: A media asset configuration completion message is created based on the target media asset configuration information and the second communication connection address; Send the media asset configuration completion message to the application process; The application process is configured to send the media asset configuration completion message to the protocol stack process.

5. The display device according to claim 2, characterized in that, Before the protocol stack process executes the response to the first wake-up media asset configuration information received from the camera and forwards the first wake-up media asset configuration information to the application process, it is also configured to: In response to a first wake-up negotiation instruction received from the camera, a first communication session is created; wherein the first wake-up negotiation instruction is used to indicate that the camera has entered the wake-up preparation phase; Send a first communication session creation message to the application process; The application process is configured to: in response to a first communication session creation message received from the protocol stack process, send a communication preparation request to the browser process; The browser process is configured to: in response to the communication preparation request received from the application process, obtain a second communication connection address supported by the display device; Write the second communication connection address into the memory.

6. The display device according to claim 1, characterized in that, The protocol stack process, in response to the media asset configuration completion message received from the application process, sends the media asset configuration completion message to the camera, thereby triggering the camera to send back the first communication connection address to the protocol stack process, specifically configured as follows: The first instruction transmission channel between the display device and the camera is detected; wherein, the first instruction transmission channel is a historical instruction transmission channel between the display device and the camera; If the first instruction transmission channel is not detected, a second instruction transmission channel is created; wherein the data capacity of the second instruction transmission channel is greater than the data capacity of the first instruction transmission channel; The media asset configuration completion message is sent to the camera via the second instruction transmission channel.

7. A display device, characterized in that, include: monitor; The communication device is configured to establish a communication connection with the camera based on a smart home interconnection protocol; The controller is configured with a protocol stack process, an application process, and a browser process. The browser process is configured as follows: Upon detecting that the first query media asset configuration information has been written to the memory, the application process and the protocol stack process send a second communication connection address to the camera, so that the camera returns the first communication connection address; wherein, the second communication connection address is a communication connection address supported by the display device; the first query media asset configuration information is the media asset configuration information returned by the camera to the display device when the display device is the initiator of the media asset query; the first query media asset configuration information is media asset configuration information jointly supported by the display device and the camera; The protocol stack process is also configured to: forward the first communication connection address to the application process in response to the first communication connection address received from the camera; The application process is also configured to write the first communication connection address into a memory. The browser process is further configured to: determine a target communication connection address based on the first communication connection address and the second communication connection address; wherein the target communication connection address is a communication connection address jointly supported by the display device and the camera; Based on the target communication connection address and the target media asset configuration information, a media asset transmission channel is established for receiving media asset data from the camera.

8. The display device according to claim 7, characterized in that, The protocol stack process is also configured as follows: In response to a first signaling message received from the application process, a first media asset query instruction is sent to the camera; wherein, the first signaling message is a signaling message generated by the application process when the display device is the initiator of the media asset query; the first signaling message includes second media asset configuration information supported by the display device; In response to the first query media asset configuration information received from the camera, the first query media asset configuration information is forwarded to the application process; wherein, the first query media asset configuration information is media asset configuration information jointly supported by the display device and the camera; The application process is configured to write the first queried media asset configuration information into a memory.

9. The display device according to claim 8, characterized in that, The application process is also configured as follows: In response to a detected media asset query operation, a media asset configuration information request instruction is sent to the browser process; The browser process is configured to: in response to the configuration information request instruction received from the application process, obtain the second media asset configuration information supported by the display device; The first signaling message is created based on the second media asset configuration information; The application process sends the first signaling message to the protocol stack process.

10. A display device, characterized in that, include: monitor; The communication device is configured to establish a communication connection with the camera based on a smart home interconnection protocol; The controller is configured with a protocol stack process, an application process, and a browser process. The browser process is configured as follows: Upon detecting that the first push media asset configuration information has been written to the memory, the application process and the protocol stack process send a second communication connection address to the camera, so that the camera can send back the first communication connection address; wherein, the second communication connection address is a communication connection address supported by the display device; the first push media asset configuration information is the information sent to the display device when the camera is the initiator of media asset push; The protocol stack process is configured to: forward the first communication connection address to the application process in response to the first communication connection address received from the camera; The application process is configured to write the first communication connection address into the memory; The browser process is configured as follows: If the first communication connection address is detected to be written to the memory, a target communication connection address is determined based on the first communication connection address and the second communication connection address; wherein, the target communication connection address is a communication connection address jointly supported by the display device and the camera; Based on the target communication connection address and the target media asset configuration information, a media asset transmission channel is established for receiving media asset data from the camera; wherein, the target media asset configuration information is the intersection of the first push media asset configuration information and the second media asset configuration information, and the second media asset configuration information is the media asset configuration information supported by the display device.

Images