A method for transmitting a code stream, an electronic device, and a computer-readable storage medium

Abstract

This application provides a method for transmitting video streams, an electronic device, and a computer-readable storage medium. The electronic device can be a smart screen or other device that includes video transmission interfaces such as HDMI and DP and is capable of acquiring online live streaming content. Whether the HDMI or other video transmission interface of the smart screen is in a working state or not, the original HDMI video stream can be copied, stored, and decoded to obtain the image data and / or audio data corresponding to the clean HDMI video stream. Then, the image data and / or audio data are standardized and encoded, and the HDMI video standard stream including image data and / or audio data is transmitted to devices such as mobile phones and tablets, enabling the devices to play HDMI video. This process shields the hardware differences between different electronic devices, can obtain customized video streams input from external devices, and, in conjunction with network transmission protocols, extends independent audio and video encoded data to other devices for playback.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a method for transmitting a code stream, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of the digital intelligence era and electronic devices, users are using more and more types of electronic devices in their daily lives, and each electronic device has increasingly rich functions, allowing users to use different functions of an electronic device.

[0003] For example, in a home setting, users may use various electronic devices such as mobile phones, smart screens, set-top boxes, tablets, personal computers (PCs), smart speakers, and routers. Optionally, users can use electronic devices with audio-visual playback capabilities, such as smart screens, tablets, PCs, and mobile phones, to play videos. Smart screens and PCs typically come equipped with high-definition multimedia interfaces (HDMI) and high-definition digital display interfaces (DP) from the factory. During use, users can connect the smart screen or PC to external devices such as set-top boxes through these video transmission interfaces to play streaming media content input from external devices.

[0004] Taking a smart screen as an example, a smart screen can connect to external devices widely used in current home scenarios, such as set-top boxes and Huawei boxes, via HDMI to obtain streaming media content input from these external devices. This streaming media content can include uncompressed high-definition video data and multi-channel audio data. It should be understood that this process primarily relies on the smart screen's decoding capabilities for video and audio streams input through video transmission interfaces such as HDMI and DP. The smart screen can decode the acquired HDMI video stream, then draw, render, and synthesize the image data based on the decoded data, presenting the final video image to the user on the display screen. Simultaneously, the smart screen can also acquire the audio data obtained after decoding the HDMI video stream and synchronously play the audio of the HDMI video through speakers or other audio playback devices.

[0005] Regarding the above implementation process, if the smart screen is playing video content input through video transmission interfaces such as HDMI or DP, the user cannot use other functions of the smart screen simultaneously. For example, the user cannot simultaneously use applications (APPs) such as Huawei Video installed on the smart screen to access online video resources, meaning the user cannot simultaneously use the smart screen to access or play video content from multiple different sources.

[0006] Furthermore, in the above implementation process, the current usage state of the smart screen and its other functions are mutually exclusive. For example, while the smart screen is playing a video, the user cannot simultaneously use it for video calls, video conferencing, or other similar operations. In other words, the above implementation process only supports the realization of a single function of the smart screen; users cannot use multiple different functions of the smart screen simultaneously, resulting in poor utilization of the smart screen's capabilities and a reduced user experience.

[0007] Furthermore, smart screens can play streaming media content input from external devices via video transmission interfaces such as HDMI. However, mobile phones, tablets, and other electronic devices do not have HDMI or similar video transmission interfaces, meaning they lack the ability to acquire HDMI video streams and therefore cannot play streaming media content such as HDMI video from external devices. Users cannot play streaming media content such as HDMI video input from external devices via their mobile phones or tablets, thus degrading their video viewing experience. Summary of the Invention

[0008] This application provides a method for transmitting bitstreams, an electronic device, and a computer-readable storage medium. The method enables the playback of online live streaming content such as HDMI video and DP video on devices such as mobile phones and tablets. This process shields the hardware differences between different electronic devices, can obtain customized video bitstreams input from external devices, and, in conjunction with network transmission protocols, extends independent audio and video encoded data to other electronic devices for playback.

[0009] A first aspect provides a method for transmitting a bitstream, applied to a source device, wherein the source device and a target device establish a communication connection, and the source device includes one or more physical transmission interfaces for acquiring media resources for online live streaming. The method includes: receiving a media resource request sent by the target device, the media resource request being used to request image data and / or audio data corresponding to a first media resource of the online live streaming; responding to the media resource request, detecting whether a first preset interface is in a working state, the first preset interface being an interface for the source device to acquire image data and / or audio data corresponding to the first media resource, the first preset interface being any one of the one or more physical transmission interfaces; when the first preset interface is in a working state, acquiring image data and / or audio data corresponding to the first media resource through the first preset interface; when the first preset interface is in a non-working state, the source device generating a driving instruction, the driving instruction being used to drive the first preset interface to switch from the non-working state to the working state, acquiring image data and / or audio data corresponding to the first media resource through the first preset interface; encoding the image data and / or audio data corresponding to the first media resource to obtain a bitstream corresponding to the first media resource to be sent; and sending the bitstream corresponding to the first media resource to the target device.

[0010] It should be understood that the "source device" in this application embodiment can be understood as a device that provides streaming media content or bitstream. For example, a smart screen serves as a source device. This smart screen is equipped with one or more physical transmission interfaces at the factory. These physical transmission interfaces are used to acquire online live video resources. For example, the aforementioned preset interfaces such as HDMI and DP are the "physical transmission interfaces" or "video transmission interfaces" described in this application embodiment. Different video transmission interfaces can connect to different external devices to acquire streaming media content, i.e., media resources, input from the external devices.

[0011] The “target device” in this application embodiment can be understood as a device that acquires streaming media content or bitstream and plays it, such as a mobile phone, tablet or other portable electronic device. This application embodiment does not limit this.

[0012] It should also be understood that the "stream" in the embodiments of this application may also be referred to as "media stream," "media bitstream," "media data stream," etc., and the "stream" may include video bitstream and / or audio bitstream, or in other words, the "stream" may include image data and / or audio data. For ease of description, the bitstream corresponding to HDMI video that includes image data and / or audio data will be uniformly referred to as HDMI bitstream (or HDMI video bitstream) thereafter.

[0013] Optionally, "first media resource" may include HDMI video resources obtained by the smart screen via HDMI from an external device for online live streaming, or DP video resources obtained by the smart screen via DP from an external device, etc., and this application embodiment does not limit this.

[0014] Optionally, the "first preset interface" can be any of the video transmission interfaces such as HDMI and DP.

[0015] It should be understood that portable electronic devices such as mobile phones and tablets generally do not have video transmission interfaces such as HDMI and DP, and therefore do not have the ability to play HDMI or DP videos.

[0016] Through the above process, the smart screen can transmit streaming media content input from external devices via video transmission interfaces such as HDMI and DP to portable electronic devices such as mobile phones and tablets for playback. Users can watch online live media content such as HDMI and DP videos on portable electronic devices such as mobile phones and tablets, reducing the constraints of the type and number of interfaces of electronic devices on users' access to media content, expanding the video sources of portable electronic devices such as mobile phones and tablets, and improving the user experience.

[0017] In one possible implementation, a media data stream processing module can be integrated into the HAL layer of the smart screen. During the aforementioned process of acquiring and transmitting the bitstream, this module can perform bitstream interception, acquiring the bitstream input from external devices via video transmission interfaces such as HDMI and DP. The module can also decode the bitstream, acquiring and caching the image and / or audio data included within it. Furthermore, when receiving resource acquisition requests from electronic devices such as mobile phones and tablets, the module can encode the image and / or audio data included in the bitstream according to the encoding and decoding capabilities of these devices, obtaining a bitstream that can be decoded by the different devices. This encoded bitstream can then be transmitted to the mobile phones, tablets, and other electronic devices, enabling the playback of different streaming media content such as HDMI and DP videos on these devices.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, detecting whether the first preset interface is in a working state includes: determining that the first preset interface is in a working state when the driver of the first preset interface is detected to be in a working state; or, determining that the first preset interface is in a working state when the source device plays the first media resource in the foreground.

[0019] For example, when the first preset interface is HDMI, the smart screen can detect its own operating status after receiving an HDMI resource request from the mobile phone. It should also be understood that the smart screen can automatically identify its own HDMI input driver and HDMI operating status.

[0020] It should be understood that when a user is watching HDMI video on a smart screen, an external HDMI device needs to input the raw HDMI video stream through the smart screen's HDMI port. When there is a video stream input through the smart screen's HDMI port, the HDMI input driver is guaranteed to be active. Therefore, the smart screen can detect the active status of the HDMI input driver and determine whether the HDMI port is operational.

[0021] For example, when the flag bit of the HDMI input driver is "1", it indicates that the HDMI input driver is in a working state; when the flag bit of the HDMI input driver is "0", it indicates that the HDMI input driver is in a non-working state. This application embodiment does not limit this.

[0022] When the HDMI input driver and HDMI are in working state, that is, the HDMI video playback channel has been established, the smart screen can directly reuse the current HDMI video playback channel, obtain the original bitstream of the HDMI video from the cache module (e.g., video buffer) and copy it in real time, and have it decoded by the HAL encoding and decoding module, thereby obtaining the image data and / or audio data corresponding to the currently playing HDMI video.

[0023] Alternatively, when the driver of the first preset interface is detected to be in a non-working state, the first preset interface is determined to be in a non-working state; or, when the source device is in a sleep state, the first preset interface is determined to be in a non-working state; or, when the source device is playing a second media resource in the foreground, the first preset interface is determined to be in a non-working state, wherein the image data and / or audio data corresponding to the second media resource are obtained through the second preset interface, and the second preset interface is any one of the one or more physical transmission interfaces that is different from the first preset interface; or, when the source device is playing a third media resource in the foreground, the first preset interface is determined to be in a non-working state, wherein the third media resource is a network media resource.

[0024] It should be understood that when the HDMI input driver and HDMI port of the smart screen are not in working state, the above embodiment can generate driver instructions and run the HDMI input driver in the background, thereby accessing the kernel layer of the smart screen. The HDMI input driver actively drives the HDMI port into working state, obtains the raw bitstream of the HDMI video input from the external device, and directly consumes the raw bitstream of the HDMI video. For example, it copies and decodes the raw bitstream of the HDMI video, obtains the corresponding image data and / or audio data, and then encodes and transmits it to the mobile phone, enabling playback of the HDMI video on the mobile phone. This process does not affect the current playback process and capabilities of the smart screen, obtains a clean raw bitstream of the HDMI video, and the process of obtaining the raw bitstream of the HDMI video is unaffected by the content being played on the smart screen.

[0025] Through the above process, the process of the mobile phone acquiring the raw bitstream of HDMI video can be unaffected by the content played on the smart screen. This allows the smart screen and the mobile phone to play different content without interfering with each other. It also supports users to use different functions of the smart screen at the same time, enriching the utilization of the smart screen's capabilities and improving the user experience.

[0026] In conjunction with the first aspect and the above implementation methods, in some implementation methods of the first aspect, obtaining the image data and / or audio data corresponding to the first media resource through the first preset interface includes: obtaining the original bitstream corresponding to the first media resource through the first preset interface; and decoding the original bitstream corresponding to the first media resource to obtain the image data and / or audio data corresponding to the first media resource.

[0027] In conjunction with the first aspect and the above implementation methods, in some implementation methods of the first aspect, before encoding the image data and / or audio data corresponding to the first media resource, the method further includes: obtaining configuration parameters for the first media resource, the configuration parameters including one or more parameters related to adding identifiers, removing identifiers, content overlay, content cropping, advertising insertion, and display effects for the image data corresponding to the first media resource, and / or one or more parameters related to mixing and playing sound effects for the audio data corresponding to the first media resource; and preprocessing the image data and / or audio data corresponding to the first media resource according to the configuration parameters to obtain preprocessed image data and / or audio data corresponding to the first media resource.

[0028] Through the above process, this application embodiment can also support developers in adding different customized processing schemes for video resources from different sources during the development process. Different customized processing schemes correspond to different configuration parameters. Before encoding the image data and / or audio data included in the original bitstream of HDMI video, the smart screen can also obtain the configuration parameters for the HDMI video resource to achieve preprocessing of the image data corresponding to the HDMI video, such as adding or removing identifiers, content overlay, content cropping, ad insertion, and display effects, and / or preprocessing of the audio data corresponding to the HDMI video, such as mixing and playing sound effects. Finally, the preprocessed image data and / or audio data corresponding to the HDMI video are obtained to meet the playback needs of different users.

[0029] It should be understood that the above process is different from the wireless screen mirroring process between a mobile phone and a smart screen. The media data stream processing module of the smart screen obtains a pure stream through video transmission interfaces such as HDMI and DP, which only includes the corresponding image data and / or audio data. It does not include additional content such as emergency information and advertising clips added when the smart screen plays HDMI video and DP video, thus ensuring the smoothness of users watching HDMI video and DP video through electronic devices such as mobile phones and tablets.

[0030] Similarly, if pop-up windows or other interference appear while the smart screen is playing HDMI or DP videos, this interference will not be displayed simultaneously on mobile phones, tablets, or other electronic devices, reducing the impact of interference on the user's viewing experience and improving the user's viewing experience.

[0031] In conjunction with the first aspect and the above implementations, in some implementations of the first aspect, the media resource request further includes information on a target encoding method, wherein the target encoding method is an encoding method that the target device can support. The step of encoding the image data and / or audio data corresponding to the first media resource to obtain the bitstream corresponding to the first media resource to be sent includes: determining the target encoding method according to the media resource request; and performing the encoding process on the image data and / or audio data corresponding to the first media resource according to the target encoding method to obtain the bitstream corresponding to the first media resource to be sent.

[0032] The above method can shield the encoding and decoding capabilities of the hardware chips of different electronic devices such as mobile phones and tablets, and ensure compatibility with the encoding capabilities of chips in different electronic devices for the raw bitstream of HDMI video. The raw bitstream of HDMI video obtained from HDMI is encoded into a unified standard bitstream such as H264 and / or H265 and / or AAC. Then, according to the type of the target device and the encoding methods supported by the target device, a specific encoding format is specified to generate the encoded bitstream corresponding to the HDMI video to be sent. This process can be applied to different electronic devices, transmitting HDMI video on the smart screen to portable electronic devices such as mobile phones and tablets, expanding the user's video sources and improving the user's viewing experience.

[0033] In conjunction with the first aspect and the above implementation methods, in some implementation methods of the first aspect, after obtaining the image data and / or audio data corresponding to the first media resource through the first preset interface, the method further includes: caching the image data and / or audio data corresponding to the first media resource in a preset order; and before encoding the image data and / or audio data corresponding to the first media resource, the method includes: obtaining the cached image data and / or audio data corresponding to the first media resource in the preset order.

[0034] In combination with the first aspect and the above implementation methods, in some implementation methods of the first aspect, the physical transmission interface includes one or more of the following: High-definition multimedia interface (HDMI) and high-definition digital display interface (DP).

[0035] It should be understood that when a smart screen plays streaming media content input from an external device via HDMI, the process mainly relies on the smart screen's ability to decode the HDMI stream, and then uses the decoded image data and / or audio data to achieve the video playback process.

[0036] Through the above process, the media data stream processing module of the smart screen can not only decode the bitstream, but also, upon receiving resource acquisition requests from electronic devices such as mobile phones and tablets, encode the image and / or audio data included in the bitstream according to the encoding and decoding capabilities of the mobile phones, tablets, and other electronic devices, obtaining bitstreams with different encoding methods, and then transmitting the encoded bitstreams to different electronic devices such as mobile phones and tablets. This process utilizes not only the smart screen's decoding capabilities, but also its encoding capabilities and other various other capabilities, fully leveraging the smart screen's chip capabilities and maximizing the utilization of the device's capabilities.

[0037] For example, in response to an HDMI video resource request sent by a mobile phone, the smart screen can obtain image data and / or audio data from the cache module (buffer) of the application framework layer for consumption. That is, according to the order, the transmission protocol of the mobile phone and the smart screen, etc., the encoded HDMI video corresponding bit stream is sent to the mobile phone.

[0038] Accordingly, after receiving the encoded HDMI video stream, the mobile phone can perform decoding to obtain the image data and / or audio data contained in the HDMI video stream, and play the HDMI video based on the contained image data and / or audio data.

[0039] In one possible scenario, when the smart screen's HDMI port is active—meaning the user is watching HDMI video on the smart screen—the HDMI video played by the phone based on the received HDMI video stream is synchronized with the HDMI video playing on the smart screen. In other words, if the smart screen is currently playing a live broadcast from a TV station, the streaming content received via HDMI and transmitted to the phone will also play the same video content on the smart screen.

[0040] In another possible scenario, when the smart screen can be in sleep mode, or when a user is watching a TV series online through the Huawei Video application on the smart screen and the smart screen is playing the TV series, this application embodiment can also realize the HDMI video stream obtained by the mobile phone from the smart screen for any of the above scenarios, and the user can also play the HDMI video on the mobile phone. In this scenario, the display content of the mobile phone and the smart screen can be displayed without interference.

[0041] Through the above method, the embodiments of this application can shield the encoding and decoding capabilities of the hardware chips of different electronic devices such as mobile phones and tablets, and be compatible with the encoding capabilities of the chips of different electronic devices for the raw bitstream of HDMI video. The raw bitstream of HDMI video obtained from HDMI is encoded into a unified standard bitstream such as H264 and / or H265 and / or AAC. Then, according to the type of the target device and the encoding method supported by the target device, a specific encoding format is specified to generate the encoded bitstream corresponding to the HDMI video to be sent. This process can be applied to different electronic devices, transmitting HDMI video on the smart screen to portable electronic devices such as mobile phones and tablets, expanding the user's video sources and improving the user's viewing experience.

[0042] A second aspect provides a method for transmitting a bitstream, applied to a target device, wherein the target device and a source device establish a communication connection, and the source device includes one or more physical transmission interfaces for acquiring media resources for online live streaming. The method includes: sending a media resource request to the source device, the media resource request being used to request image data and / or audio data corresponding to a first media resource of the online live streaming; receiving the bitstream corresponding to the first media resource sent by the source device; decoding the bitstream corresponding to the first media resource to acquire the image data and / or audio data contained in the bitstream corresponding to the first media resource; and playing the first media resource based on the image data and / or audio data contained in the bitstream corresponding to the first media resource.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the media resource request further includes information about a target encoding method, wherein the target encoding method is an encoding method that the target device can support, the bitstream corresponding to the first media resource sent by the source device is a bitstream encoded based on the target encoding method, and the decoding process of the bitstream corresponding to the first media resource to obtain the image data and / or audio data contained in the bitstream corresponding to the first media resource includes:

[0044] According to the target encoding method, the bitstream corresponding to the first media resource is decoded to obtain the image data and / or audio data contained in the bitstream corresponding to the first media resource.

[0045] Through the above process, after the mobile phone receives the encoded HDMI video stream, it can perform decoding to obtain the image data and / or audio data contained in the HDMI video stream, and play the HDMI video based on the contained image data and / or audio data.

[0046] A third aspect provides an electronic device, which may be a source device, comprising: one or more physical transmission interfaces for acquiring media resources for online live streaming; a display screen; one or more processors; one or more memories; a module having multiple applications installed; the memories storing one or more programs, the one or more programs including instructions that, when executed by the electronic device, cause the electronic device to perform a method as described in the first aspect and any one of the first aspects that can be executed by the source device.

[0047] The fourth aspect provides an electronic device, which may be a target device, comprising: a touch screen; one or more processors; one or more memories; a module having multiple applications installed; the memories storing one or more programs, the one or more programs including instructions that, when executed by the electronic device, cause the electronic device to perform a method as described in the second aspect and any one of the second aspects that can be executed by the target device.

[0048] The fifth aspect provides a system comprising a source device and a target device capable of communicating with each other, the source device being capable of performing a method as described in any one of the first aspect and its implementations, and the target device being capable of performing a method as described in any one of the second aspect and its implementations.

[0049] A sixth aspect provides a graphical user interface system for an electronic device, the electronic device having a touch screen, one or more memories, and one or more processors, the one or more processors being configured to execute one or more computer programs stored in the one or more memories, the graphical user interface system including a graphical user interface displayed when the electronic device performs the method as described above.

[0050] A seventh aspect provides an apparatus included in an electronic device, which may be a source device or a target device. When the electronic device is a source device, the apparatus has the function of implementing the source device behavior described in the first aspect and any possible implementation thereof. Alternatively, when the electronic device is a target device, the apparatus has the function of implementing the target device behavior described in the second aspect and any possible implementation thereof.

[0051] Optionally, the functions of the device can be implemented in hardware or by executing corresponding software within the hardware. The hardware or software includes one or more modules or units corresponding to the aforementioned functions. For example, a display module or unit, a detection module or unit, a processing module or unit, etc.

[0052] The eighth aspect provides a computer-readable storage medium storing computer instructions that, when executed on an electronic device, cause the electronic device to perform the first aspect or any possible method of the first aspect, or cause the electronic device to perform any possible method of the second aspect.

[0053] The ninth aspect provides a computer program product that, when run on an electronic device, causes the electronic device to perform the first aspect or any possible method of the first aspect, or causes the electronic device to perform any possible method of the second aspect. Attached Figure Description

[0054] Figure 1 This is a diagram illustrating how a user wirelessly projects the content displayed on their mobile phone onto the display screen of a smart screen in a home setting.

[0055] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0056] Figure 3 This is a software structure block diagram and implementation process schematic diagram of an example electronic device according to an embodiment of this application.

[0057] Figure 4 This is a schematic flowchart corresponding to an example of a method for transmitting a code stream provided in an embodiment of this application.

[0058] Figure 5 This is a schematic diagram illustrating a process by which a mobile phone requests video resources from a smart screen, as provided in an embodiment of this application.

[0059] Figure 6 This is another example of a mobile phone requesting to obtain video resources from a smart screen, provided in an embodiment of this application.

[0060] Figure 7 This is a schematic diagram illustrating an example of a mobile phone playing an HDMI video from a smart screen, provided in an embodiment of this application. Detailed Implementation

[0061] To facilitate understanding, the following will use the process of a user using various electronic devices such as mobile phones and smart screens in a home setting as an example, and will be illustrated with accompanying diagrams to explain the specific application scenarios.

[0062] Figure 1 This is an illustration of a home scenario where a user wirelessly projects their phone's display onto a smart screen. It assumes the user has already enabled wireless screen mirroring on their phone and set it to project the video playing on the phone onto the smart screen for playback.

[0063] For example, such as Figure 1As shown in Figure (a), the current interface 101 of the mobile phone displays various possible interface elements or content, such as the video currently being played by the user, the video's playback progress bar, playback controls, close controls, and back controls. It should be understood that the interface 101 may display less content than shown in the figure or display more content, and the embodiments of this application do not limit the content displayed on the interface 101.

[0064] Optionally, in this embodiment, each application installed on the mobile phone can have an independently developed wireless screen mirroring function. For example, when a user is using a video application on their mobile phone, they can activate the wireless screen mirroring function of the video application through a shortcut control (or switch, etc.), triggering the mobile phone to search for connectable electronic devices such as smart screens, and then mirroring any running interface of the video application displayed on the mobile phone onto the smart screen.

[0065] Alternatively, as trusted devices in a home setting, the mobile phone and smart screen can enable the wireless screen mirroring function of the mobile phone through system-level applications such as settings applications, thereby projecting any interface of the user's mobile phone to the smart screen. This application embodiment does not limit the implementation method of wireless screen mirroring.

[0066] It should be understood that the above wireless screen mirroring process can be understood as a "screen mirroring" process. Specifically, mobile phones and smart screens can share the screen content of mobile phones to the connected smart screen based on wireless screen mirroring protocols such as Miracast, Airplay, and Digital Living Network Alliance (DLNA), that is, to display all the content of the mobile phone screen on the smart screen in real time and synchronously.

[0067] For example, such as Figure 1 As shown in Figure (a), after the mobile phone and the smart screen establish a wireless screen mirroring connection, the interface 101 displayed on the mobile phone can display the currently playing video on the smart screen's display screen, such as... Figure 1 The interface 102 shown in Figure (a) is used, and the audio corresponding to the video is also played through the smart screen's speakers and other audio playback devices.

[0068] In one possible scenario, during wireless screen mirroring between a mobile phone and a smart screen, other applications on the phone might display message pop-ups with different content, or the phone might receive call requests from others, or the phone might receive messages from... application, This includes voice and video requests from social media apps. In this scenario, the phone may display corresponding message pop-ups or notification interfaces. Users need to handle the message notifications in the pop-ups or perform actions such as answering or hanging up calls, or receiving voice and video requests from social media apps within the notification interfaces.

[0069] For example, such as Figure 1 As shown in Figure (b), when the mobile phone temporarily receives a call from user "Mr. Wang" during wireless screen mirroring, the mobile phone can directly interrupt the current video playback and display the incoming call answering interface 103 as shown in the figure. Optionally, the interface 103 may include an avatar, nickname, phone number (138XXXX0493) and location details related to user "Mr. Wang", hang-up control 103-1, answer control 103-2, SMS icon 103-3 and reminder icon 103-4, etc. The display content of the interface 103 is not limited in this embodiment.

[0070] Accordingly, the smart screen's display will also maintain consistency with the phone's interface; for example, the smart screen can display... Figure 1 As shown in Figure (b), on the interface 104, a window 104-1 can automatically pop up on the video playback screen and display the content of the incoming call answering interface 103 of the mobile phone in the window 104-1. At the same time, the video playback on the smart screen is interrupted, that is, the currently playing video is switched to a paused playback state.

[0071] During the aforementioned wireless screen mirroring process, the phone's screen content can be completely copied to the smart screen for playback, for example... Figure 1 In the scenario shown in Figure (b), the mobile phone mirrors the video screen to the smart screen for playback. If the mobile phone receives a call at this time, the smart screen will simultaneously display the incoming call interface and other distracting information, and enter call answering mode, interrupting the video playback process. This process affects the video playback on the smart screen, and in a home setting, it may affect other family members' viewing experience, thus reducing the user's viewing experience.

[0072] Secondly, during the aforementioned wireless screen mirroring process, users may only want to project the video onto the smart screen for playback, and do not expect to see message pop-ups, phone calls, or other content involving personal privacy on the smart screen in addition to the video from their phone. The wireless screen mirroring process (screen mirroring process) described above does not offer the option to select the content to be displayed, and users cannot filter the content they wish to display on the smart screen, thus failing to meet their privacy needs.

[0073] Furthermore, smart screens, mobile phones, and other electronic devices each have numerous functions, and users may expect to use different functions of a single device simultaneously. However, the current usage state of an electronic device and its other functions are often mutually exclusive. For example, taking a smart screen as an example... Figure 1 As shown in Figure (b), when the smart screen pops up window 104-1 and displays the phone's incoming call interface in window 104-1, the video playback process on the smart screen is interrupted and cannot continue normally. Alternatively, while the smart screen is playing a video, the user cannot simultaneously use the smart screen for video calls, video conferencing, or other operations. In the above usage scenarios, the smart screen only supports a single function, preventing users from simultaneously using multiple different functions, resulting in poor utilization of the smart screen's capabilities and a reduced user experience.

[0074] Furthermore, smart screens can play streaming media content input from external devices via video transmission interfaces such as HDMI and DP, and can also access online video resources through installed applications like Huawei Video. However, portable electronic devices such as mobile phones and tablets generally lack video transmission interfaces like HDMI, meaning they lack the ability to play HDMI streams and cannot access streaming media content from external devices. In other words, users cannot play streaming media content input from external devices through mobile phones, tablets, etc., and the video sources available to users are limited by the type of electronic device.

[0075] Finally, when the smart screen plays streaming media content input from external devices such as set-top boxes via HDMI, this process mainly relies on the smart screen's decoding capability for the HDMI stream. It then uses the decoded image data to draw, render, and composite the image, presenting the final video image to the user on the display. Simultaneously, the smart screen also uses the decoded audio data to achieve synchronized playback through speakers and other audio playback devices. This process relies solely on the decoding capabilities of the smart screen and other electronic devices, without utilizing the smart screen's other capabilities, thus failing to fully leverage its potential.

[0076] In conclusion, relying on wireless screen mirroring between different electronic devices is no longer sufficient to meet user needs, and the presence of interference during wireless screen mirroring reduces the user experience.

[0077] To address the aforementioned issues, this application provides a method for transmitting video streams. This method enables portable electronic devices such as mobile phones and tablets to play video resources originating from electronic devices such as smart screens and PCs. These video resources can be streaming media content input from external devices via video transmission interfaces such as HDMI and DP, thereby meeting the usage needs of different users and improving the user experience of using electronic devices.

[0078] It should be understood that the process of transmitting the bitstream provided in the embodiments of this application can occur between two different electronic devices, wherein the two different electronic devices can be the same type of electronic devices or different types of electronic devices.

[0079] For example, two different electronic devices may include any type of electronic device such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops (PCs), ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0080] It should also be understood that, for ease of description, in the embodiments of this application, the two different electronic devices involved in the process of transmitting the bitstream are referred to as the "source device" and the "target device," respectively. The "source device" can be understood as a device that provides streaming media content or bitstream, and the "target device" can be understood as a device that acquires streaming media content or bitstream and plays it.

[0081] For example, in Figure 1 In the home scenario shown, taking a connected mobile phone and smart screen as an example, the smart screen can be used as the "source device" and the mobile phone can be used as the "target device". The mobile phone can obtain media stream content from external devices on the smart screen and play the obtained streaming media content.

[0082] It should also be understood that the "stream" in the embodiments of this application may also be referred to as "media stream," "media bitstream," "media data stream," etc., and the "stream" may include video bitstream and / or audio bitstream, or in other words, the "stream" may include image data and / or audio data. For ease of description, the bitstream corresponding to HDMI video that includes image data and / or audio data will be uniformly referred to as HDMI bitstream (or HDMI video bitstream) thereafter.

[0083] It should also be understood that the hardware and software structures of the source and target devices are capable of supporting the process of implementing the transmission bitstream of this application.

[0084] Before introducing the implementation process of the transmitted code stream, we will first introduce the hardware and software structures that the source and target devices in the embodiments of this application may have.

[0085] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0086] It should be understood that the source device in the embodiments of this application may include Figure 2 The illustrated electronic device 100 may include part or all of its structure, or may also include other structures not shown; similarly, the target device may also include Figure 2 The illustrated electronic device 100 may include part or all of its structure, or may include other structures not shown, but this application embodiment does not limit this.

[0087] For example, such as Figure 2 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0088] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0089] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0090] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0091] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0092] In this embodiment, the electronic device 100 can function as either a source device or a target device. The processor 110 of the electronic device 100 stores a computer program or instructions corresponding to the method for implementing the transmitted bitstream.

[0093] For example, for source devices such as smart screens, when the processor 110 of the smart screen detects an HDMI video resource request sent by a mobile phone, it can call different software and hardware modules to achieve the following: when it detects that the smart screen is playing streaming media content input from an external device via HDMI, it relies on the decoding capability of the processor 110 for the HDMI bitstream, and then implements the video playback process based on the decoded image data and / or audio data.

[0094] Specifically, when the smart screen detects a request for HDMI video resources sent by a mobile phone, in response to the HDMI video resource request, it detects whether the HDMI is in a working state. When the HDMI is in a working state, it acquires the image data and / or audio data corresponding to the HDMI video through the HDMI. When the HDMI is in a non-working state, the processor 110 can generate a driving instruction to drive the HDMI to switch from the non-working state to the working state, acquire the image data and / or audio data corresponding to the HDMI video through the HDMI, encode the image data and / or audio data corresponding to the HDMI video to obtain the HDMI stream to be sent, and send the HDMI stream to the mobile phone so that the mobile phone can play the HDMI video according to the image data and / or audio data contained in the HDMI stream.

[0095] For example, in this embodiment, in addition to decoding the bitstream, the processor 110 can also, upon receiving a resource acquisition request from a mobile phone, tablet, or other electronic device, encode the image data and / or audio data included in the bitstream according to the encoding and decoding capabilities of the mobile phone, tablet, or other electronic device, obtaining bitstreams with different encoding methods, and then transmitting the encoded bitstreams to different electronic devices such as mobile phones and tablets. The above process not only utilizes the decoding capability of the processor 110's encoding and decoding module, but also utilizes its encoding capabilities and other various other capabilities, fully leveraging the capabilities of the smart screen's processor 110 and maximizing the utilization of the device's capabilities.

[0096] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0097] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0098] The I2S interface can be used for audio communication.

[0099] For example, in some embodiments of this application, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to realize communication between the processor 110 and the audio module 170. The processor 110 can transmit the audio data processed by the audio service module to the audio module 170 via the I2S bus. The audio module 170 can establish an audio playback channel and play the corresponding audio based on the acquired audio data.

[0100] In other embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via an I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0101] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0102] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0103] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. The MIPI interface includes the camera serial interface (CSI) and the display serial interface (DSI).

[0104] The GPIO interface can be configured via software. The GPIO interface can be configured as either control signals or data signals.

[0105] USB interface 130 is an interface that conforms to the USB standard specification, specifically it can be a Mini USB interface, Micro USB interface, USB Type C interface, etc.

[0106] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0107] The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0108] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0109] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0110] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0111] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G / 6G, for use on electronic devices 100.

[0112] A modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal.

[0113] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0114] It should be understood that in the embodiments of this application, the source device and the target device are two different electronic devices, and both can include the wireless communication module 160.

[0115] It should also be understood that in the embodiments of this application, the source device and the target device have established a communication connection, and the source device and the target device may have a variety of different communication methods.

[0116] Optionally, the source device and the target device can be devices connected to the same Wi-Fi network, and the two devices can communicate based on the Wi-Fi link; or, the two devices can establish a Bluetooth connection and communicate based on the Bluetooth channel; or, the two devices can establish a connection based on NFC technology such as tap-to-connect; or, the two devices can establish a connection based on wireless communication methods such as infrared (IR), 2G / 3G / 4G / 5G / 6G. Accordingly, after the two electronic devices establish any possible communication connection, they can communicate in the corresponding manner. This application embodiment does not limit the communication method between the source device and the target device.

[0117] Optionally, in the embodiments of this application, the "source device" and the "target device" can be electronic devices in Huawei's super terminal scenario. The "source device" and the "target device" that are super terminals to each other have a device-level trusted relationship. In other words, electronic devices that are super terminals to each other have the right to access each other.

[0118] For example, when the "source device" and the "target device" detect that the other device is logged into the same Huawei account, they can form a super terminal to achieve collaborative management and resource sharing among multiple devices. Users can coordinate multiple electronic devices such as nearby tablets, PCs, smart screens, and smart speakers with one click, and continue the tasks of other electronic devices on the coordinated electronic devices. Video streams and music streams of one electronic device can be transferred to other electronic devices. This application embodiment will not elaborate on this.

[0119] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with networks and other devices through wireless communication technology.

[0120] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0121] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0122] It should be understood that in the embodiments of this application, both the source device and the target device may include a display screen 194, which can display different interface content and receive user touch operations.

[0123] For example, when the smart screen is used as the source device, the display screen 194 of the smart screen can display a video playback interface. For example, the smart screen can play HDMI video or play network video on demand by the user. This application embodiment does not limit this.

[0124] Accordingly, when the mobile phone is the target device, after receiving the HDMI stream sent by the smart screen, the mobile phone can decode the HDMI stream, obtain the image data and audio data included in the HDMI stream, and draw, render and synthesize the image based on the image data obtained after decoding, and present the final video picture to the user on the mobile phone's display screen. For the sake of simplicity, it will not be described in detail here.

[0125] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0126] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0127] Camera 193 is used to capture still images or videos. An object passes through the lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0128] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0129] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0130] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0131] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0132] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0133] In the embodiments of this application, both the source device and the target device include a storage data area, such as a cache module (buffer) of a smart screen and a mobile phone. The cache module (buffer) may further include an audio data cache module and / or an image data cache module, etc.

[0134] When the smart screen plays HDMI video, the HDMI stream is acquired and decoded to obtain audio data and / or image data. The audio data can be initially stored in an audio data cache module, and the image data can be initially stored in an image data cache module. When the smart screen reaches the corresponding video progress, the corresponding audio data can be retrieved directly from the audio data cache module and the corresponding image data from the image data cache module in sequence. This is the "data consumption process of the cache module (buffer)," or in other words, the data stored in the cache module is in a state of waiting to be consumed. This embodiment of the application will not elaborate on this further.

[0135] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0136] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0137] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0138] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0139] For example, in this embodiment of the application, the smart screen acts as the source device. During the playback of HDMI video, the smart screen can obtain the audio data obtained after decoding the bitstream of the HDMI video and play the audio of the HDMI video synchronously through audio playback devices such as speakers.

[0140] When the mobile phone is the target device, after receiving the HDMI stream sent by the smart screen, the mobile phone can decode the HDMI stream and obtain the image data and audio data included in the HDMI stream. Based on the audio data obtained after decoding the HDMI video stream, the audio of the HDMI video can be played synchronously through audio playback devices such as speakers.

[0141] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0142] The 170D headphone jack is used to connect wired headphones.

[0143] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals.

[0144] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for image stabilization.

[0145] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0146] The magnetic sensor 180D includes a Hall effect sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. The accelerometer 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected.

[0147] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0148] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode.

[0149] An ambient light sensor 180L is used to sense the ambient light intensity. Electronic device 100 can adaptively adjust the brightness of display screen 194 according to the sensed ambient light intensity.

[0150] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprint to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc. The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0151] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0152] In this embodiment of the application, for target devices such as mobile phones, the touch sensor 180K can detect user touch, click and other operations, generate operation events, and transmit the operation events to the processor 110. The processor 110 determines the operation events, including the coordinates of the touch point, touch state and other related parameters, and responds according to the operation events.

[0153] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone fragments of the human vocal cords. The button 190 includes a power button, volume buttons, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0154] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0155] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0156] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0157] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture. Taking the system as an example, the software structure of electronic device 100 is illustrated.

[0158] Figure 3 This is a software structure block diagram and implementation process schematic diagram of an example electronic device according to an embodiment of this application.

[0159] Optionally, both the source device and the target device in the embodiments of this application may have The system, with Taking the system as an example, the source device and the target device can have Figure 3 The layered architecture shown in Figure (a) can include several layers, each with a clear role and division of labor, and the layers communicate with each other through software interfaces.

[0160] Exemplary, in some embodiments, such as Figure 3 As shown in Figure (a), it can be The system is divided into four layers, from top to bottom: the application layer, the application framework layer, the audio hardware abstraction layer (Audio HAL), and the kernel layer.

[0161] The application layer can include a series of application packages. For example, such as... Figure 3 As shown in Figure (a), the application layer may include one or more application packages corresponding to applications such as video applications, music applications, and settings applications. This application embodiment does not limit this.

[0162] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0163] Optionally, in this embodiment of the application, taking a smart screen as an example, during the video playback process of the smart screen, the application framework layer may include at least a view system, an audio service module, a cache module (buffer), etc.

[0164] In this embodiment, "view system" can be understood as a general term for one or more software modules used to support the display of images, interfaces, and other content on a smart screen. For example, during video playback on a smart screen, the view system can acquire image data from the media data stream and perform graphics rendering, drawing, compositing, and image display based on the image data. Optionally, the "view system" may also include visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications, and the interface displayed by an electronic device can consist of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0165] Optionally, "one or more software modules for displaying images, interfaces, and other content on the smart screen" may specifically include modules such as a window manager and a content provider. The window manager manages the window programs of the smart screen. The window manager can obtain the screen size, determine the displayed content and status bar, or participate in operations such as locking the screen and capturing the screen. The content provider stores and retrieves data, such as audio data, and makes this data accessible to applications. The stored data may include video data, image data, audio data, etc., as mentioned in the aforementioned "media data stream," which will not be elaborated further here.

[0166] Optionally, in the embodiments of this application, "image data in media data stream" may include image data from different applications such as video applications and music applications at the application layer. For example, when a user watches a video through the Huawei Video application, they can obtain network video resources through the Huawei Video application, that is, obtain the image data included in the network video resources.

[0167] Alternatively, "image data in the media data stream" may also include image data corresponding to streaming media content input from external devices that the smart screen obtains through video transmission interfaces such as HDMI and DP. For example, the smart screen connects to external devices such as set-top boxes and Huawei boxes via HDMI, and then obtains the HDMI stream. After decoding the HDMI stream, image data, audio data, etc. are obtained. This application embodiment does not limit this.

[0168] The “audio service module” can be understood as the collective term for the software modules included in the audio system used to support the playback of audio for videos on smart screens.

[0169] For example, a smart screen can connect to external devices such as set-top boxes and Huawei boxes via HDMI to obtain HDMI streams. The audio service module can obtain audio data from the HDMI streams, construct audio track instances at the application framework layer, and then write audio data to the corresponding audio track. Each audio track can further output decoded audio data to the audio system, and finally play the corresponding audio through audio playback devices such as speakers. This application embodiment does not limit this.

[0170] The buffer module may further include an audio data buffer module, and / or an image data buffer module, etc.

[0171] For example, when a smart screen plays HDMI video, it acquires the HDMI stream and decodes it to obtain audio data and / or image data. The audio data can be initially stored in an audio data cache module, and the image data can be initially stored in an image data cache module. When the smart screen reaches the corresponding video progress, it can directly extract the corresponding audio data from the audio data cache module and the corresponding image data from the image data cache module in sequence. This is the "data consumption process of the cache module (buffer)," or in other words, the data stored in the cache module is in a state of waiting to be consumed. This embodiment of the application will not elaborate on this.

[0172] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0173] HAL is a layer that interacts directly with the hardware. It can isolate all hardware-related code logic into a dedicated module and provide interfaces to the upper layers, thereby enabling the upper layers to be independent of the hardware platform.

[0174] In this embodiment, a "media data stream processing module" is configured and integrated into the HAL, which can be used to process the acquired media stream data. Optionally, the media data stream processing module may include an encoding / decoding module, a customized processing module, etc.

[0175] For example, the encoding / decoding module can decode the acquired HDMI stream to obtain image data and audio data; it can also encode the image data and audio data separately to generate a standardized stream.

[0176] The customized processing module can add watermarks, emergency information, and advertising pop-ups to the image data before the encoding and decoding module encodes the image data and audio data respectively, according to the developer's configuration. It can also perform mixed encoding processing on audio data from different sources. This application embodiment does not limit the customized processing process configured by the developer.

[0177] It should be understood that the software modules included in HAL and the software modules included in the application framework layer can be in one-to-one correspondence. That is, the application layer can call the processing code of HAL layer through the corresponding interface of application framework layer to achieve the relevant functions. These will not be listed or elaborated here.

[0178] For example, in this embodiment of the application, taking the playback of audio on a smart screen as an example, HAL can be responsible for interacting with audio playback devices such as speakers. The software modules related to the audio system in HAL can select audio playback devices such as speakers that play the current audio data, and access the driver corresponding to the audio playback device in the kernel layer, thereby driving the audio playback device to play the audio corresponding to the current video.

[0179] In addition, HAL can also include multiple functional modules such as a surface manager, media libraries, a three-dimensional (3D) graphics processing library (e.g., OpenGL ES), and a two-dimensional (2D) graphics engine. These multiple functional modules can be used to support electronic devices such as smart screens in the process of playing videos, audio, and displaying images.

[0180] For example, the surface manager manages the display subsystem of the electronic device and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as still image files. The media library can support various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D drawing. For simplicity, this application embodiment only shows the software modules related to the process of transmitting the bitstream; other software modules included in the electronic device are not shown one by one.

[0181] The kernel layer is the layer between hardware and software.

[0182] For example, the kernel layer may include at least a display driver, an audio driver, an HDMI input driver, etc., and the kernel layer can connect to various controlled hardware devices through different hardware drivers.

[0183] For example, taking the playback of live video from an external HDMI device via HDMI on a smart screen as an example, the process may include, for instance, the following: Figure 3 The multiple steps shown in Figure (a) include, for example:

[0184] Step 1: When the HDMI input driver is in working state, the HDMI external device inputs the HDMI stream to the smart screen via HDMI.

[0185] It should be understood that "HDMI stream" here includes the sum of uncompressed audio and / or image data, and the audio and / or image data are encoded, and the audio and image data can be transmitted from an external HDMI device to the smart screen through the same transmission channel. For example, the smart screen integrates HDMI driving capabilities to acquire the raw HDMI stream.

[0186] It should also be understood that, in this embodiment, different bitstreams can be further divided into video bitstreams and audio bitstreams. A video bitstream may include image data and audio data, and an audio bitstream may include audio data. Therefore, for the sake of simplicity, the data contained in the bitstream will be uniformly referred to as "audio data and / or image data" in this embodiment, and will not be elaborated further thereafter.

[0187] For example, the original HDMI stream can be a BT1120 data stream with 4 inputs supporting a maximum resolution of 1920×1080 and a refresh rate of 60 frames per second (denoted as: 1080p@60fps), but this application embodiment does not limit it.

[0188] Step 2: After receiving the HDMI stream, the smart screen transmits the received HDMI stream to HAL, where HAL's media data stream processing module processes the stream, such as the encoding / decoding module decoding the HDMI stream to obtain the decoded audio and image data.

[0189] Step 3: HAL's media data stream processing module can transmit the decoded audio data to the audio service module in the application framework layer. The media data stream processing module can also transmit the decoded image data to the image processing modules such as the view system in the application framework layer.

[0190] For example, the audio service module in the application framework layer can process the audio data accordingly, and the processed audio data is stored in the corresponding audio data cache module.

[0191] Image processing modules, such as the view system in the application framework layer, can process image data accordingly, and the processed image data can be stored in the corresponding image data cache module.

[0192] In steps ④ and ⑤, during the playback of HDMI video on the smart screen, corresponding audio data can be obtained from the audio data cache module. The audio data can then be transmitted to audio playback devices such as speakers via the audio driver. Simultaneously, corresponding image data can be obtained from the image data cache module. This image data can undergo image processing by image processing modules such as the view system, and then be displayed via the display driver, i.e., image rendering, drawing, and compositing, ultimately displaying the video image on the smart screen's display screen.

[0193] By following the steps ①②③④⑤ above, the smart screen can play HDMI video input via HDMI, allowing users to watch live HDMI video on the smart screen.

[0194] It should be understood that the above processes may involve the pass-through of image data and / or audio data between multiple layers, or other processing of image data and / or audio data, thereby enabling HDMI video playback. For simplicity, this application is an embodiment and will not elaborate on this.

[0195] Optionally, the above example of playing HDMI video stream on a smart screen illustrates a possible implementation process. It should be understood that the "media stream content (stream)" in this application embodiment can be an audio stream (or audio stream), or a video stream (or video stream), etc. This application embodiment does not limit this.

[0196] For example, in this embodiment, when the "media stream content (stream)" is an HDMI video stream, with the smart screen as the source device and the mobile phone as the target device, the mobile phone can also obtain the HDMI video stream from the smart screen. Specifically, compared to the process of playing HDMI video on the smart screen described above, this embodiment can also transmit the HDMI video stream from the smart screen to the mobile phone. This process can be as follows: Figure 3 As shown in Figure (b), assuming the raw bitstream of the HDMI video may include image and audio data, the process includes the following steps:

[0197] Step 1: When the smart screen's HDMI input driver is in working condition, external HDMI devices input the raw bitstream of HDMI video to the smart screen via HDMI.

[0198] Step 2: After receiving the HDMI stream, the smart screen transmits the received HDMI stream to HAL, where HAL's media data stream processing module decodes the HDMI stream.

[0199] For example, the codec module can decode the HDMI stream to obtain the audio and image data corresponding to the decoded HDMI stream, and cache the decoded image data in the image data cache module, or video data cache module; and cache the decoded audio data in the audio data cache module, or pulse code modulation (PCM) buffer module.

[0200] Step 3: The decoded image data is transmitted to the image processing module in the application framework layer to complete the image data processing process.

[0201] It should be understood that the image processing module may include, for example: Figure 3 The view system described in Figure (a) then transmits the image data via the display driver, which involves the image drawing, rendering, and compositing process, and finally displays the video on the smart screen.

[0202] It should also be understood that the software modules involved in this process can be collectively referred to as the "video processing sub-system (VPSS)". The above process is the video display process of the video processing sub-system on the smart screen (VPSS DISP).

[0203] Step 4: The decoded audio data is transmitted to the audio service module in the application framework layer to complete the audio data processing. For example, during audio playback, the audio system determines the audio playback medium—audio track and audio channels, etc.—and finally, via the audio driver, transmits the data to audio playback devices such as speakers and amplifiers (AMPs) for playback.

[0204] It should be understood that the above steps ①②③④ and Figure 3 The process described in Figure (a) is similar. The smart screen can play HDMI videos input from HDMI, and users can watch live HDMI videos on the smart screen.

[0205] It should also be understood that, in this embodiment of the application, when the smart screen receives an HDMI video resource request sent by the mobile phone, the following steps are also required in response to the HDMI video resource request.

[0206] Step 5: When the smart screen receives the HDMI video resource request sent by the mobile phone, it can obtain the corresponding image data in response to the HDMI video resource request, encode it according to the encoding method supported by the mobile phone, store the encoded bitstream in the image data cache module (buffer) of the application framework layer, and finally transmit the corresponding image data to the mobile phone from the image data cache module (buffer).

[0207] Step 6: The smart screen can also acquire the corresponding audio data, encode it according to the encoding method supported by the mobile phone, store the encoded bitstream in the image data cache module (buffer) of the application framework layer, and finally transmit the corresponding audio data to the mobile phone from the audio data cache module (buffer).

[0208] Through the above process, the mobile phone can receive the encoded HDMI video stream, perform decoding processing, obtain the image data and / or audio data contained in the HDMI video stream, and play the HDMI video based on the contained image data and / or audio data.

[0209] It should be noted that in actual implementation, when the stream to be transmitted is a video stream, the video stream includes both image data and audio data. This process can include all the steps described above. The smart screen can copy, decode, customize, and encode the image data and audio data separately, and then transmit the image data stream and audio data stream to the mobile phone side, or treat the image data and audio data as a whole video data and perform the above processing. This application embodiment does not limit the specific processing procedures and rules.

[0210] When the bitstream to be transmitted is an audio bitstream, the audio bitstream may only include audio data. This process may not involve the image data processing processes described in steps ③ and ⑤ above. The smart screen can copy, decode, customize, and encode the audio data, and then transmit the image data bitstream to the mobile phone. This application embodiment does not limit the specific processing process and rules.

[0211] Below, we will take the process of transmitting video streams (such as HDMI video streams) as an example in a specific scenario, and combine it with... Figure 3 Figures (a) and (b) in this application detail the process of transmitting the code stream provided in the embodiments of this application.

[0212] Figure 4 This is a schematic flowchart corresponding to an example of a method for transmitting a code stream provided in an embodiment of this application.

[0213] For example, the following embodiments of this application will be used as... Figure 1Taking the smart screen and mobile phone in the home scenario shown as an example, let's assume that both the smart screen and the mobile phone can have Figure 2 and Figure 3 The structure shown in Figure (a) uses a mobile phone as the "target device" described above and a smart screen as the "source device" that provides the HDMI stream. The process of transmitting the stream is explained in detail with reference to the accompanying figures and application scenarios.

[0214] Specifically, the method 400 may include the following steps:

[0215] 401, the source device receives a media resource request sent by the target device, the media resource request being used to request image data and / or audio data corresponding to the first media resource of the online live broadcast.

[0216] It should be understood that, as a source device, the smart screen is equipped with one or more physical transmission interfaces at the factory. These physical transmission interfaces are used to acquire media resources from online live streaming. For example, the aforementioned preset interfaces such as HDMI and DP are the "physical transmission interfaces" or "video transmission interfaces" described in this application embodiment. Different video transmission interfaces can connect to different external devices to acquire streaming media content, i.e., media resources, input from external devices.

[0217] It should also be understood that, in the embodiments of this application, streaming media content (media resources) is not limited to video resources, audio resources, etc.

[0218] Optionally, in the embodiments of this application, the "first media resource" may include HDMI video resources obtained by the smart screen via HDMI from an external device for online live streaming, or DP video resources obtained via DP from an external device, etc., and the embodiments of this application do not limit this.

[0219] It should also be understood that video transmission interfaces such as HDMI and DP rely on external devices such as set-top boxes and Huawei boxes. The external devices transmit video resources to the smart screen through the video transmission interface. This process is different from the network video resources that users can access through the smart screen's on-demand function.

[0220] For example, when a user plays a TV series online through an application such as Huawei Video installed on a smart screen, it only requires the communication capability between the Huawei Video application and its corresponding server. The Huawei Video application requests the video resource of the TV series from the corresponding server, and the server returns the image data and / or audio data of the TV series to the smart screen, which then completes the playback process.

[0221] Therefore, in this embodiment of the application, the image data and / or audio data (i.e., the original bitstream) corresponding to the "first media resource" are obtained by external devices and transmitted to the smart screen through video transmission interfaces such as HDMI and DP. The image data and / or audio data corresponding to the network video resources that users watch on demand through applications such as Huawei Video are obtained directly from the server by the smart screen side, without needing to go through video transmission interfaces such as HDMI and DP, which will not be described in detail later.

[0222] Optionally, in step 401, taking the "media resource request" sent by the mobile phone to request the HDMI video resources of the smart screen as an example, the user can trigger the mobile phone (target device) to send an HDMI video resource request (i.e., media resource request) to the smart screen in a variety of possible ways to obtain the HDMI video resources (first media resources) of the smart screen.

[0223] Figure 5 This is a schematic diagram illustrating a process by which a mobile phone requests video resources from a smart screen, as provided in an embodiment of this application.

[0224] It should be understood that in a home setting, multiple electronic devices such as mobile phones, smart screens, and tablets can form a group of super terminal devices. Within this group, the electronic devices are mutually trusted with each other. It should be understood that these mutually trusted electronic devices can access and obtain data from each other.

[0225] In one possible implementation, users can trigger a request from their mobile phone to obtain HDMI video resources from the smart screen via the HyperTerminal function option on their phone.

[0226] For example, Figure 5 Figure (a) shows the interface 501 displayed on the mobile phone in unlocked mode. Interface 501 displays a weather clock component and multiple applications (Apps). These applications may include video, camera, gallery, WeChat, phone, messages, music, settings, etc. It should be understood that interface 501 may also include other applications or display more content; this application embodiment does not limit the content of the mobile phone's interface.

[0227] like Figure 5 As shown in Figure (a), when a user clicks the settings app icon, the phone displays the following in response to the user's click: Figure 5Figure (b) shows the main settings interface 502. This main settings interface 502 may include multiple options or menus such as the user account logged in on the mobile phone, WLAN options, Bluetooth options, desktop and wallpaper options, display and brightness options, sound and vibration options, HyperTerminal options, and accessibility options. Users can access the corresponding settings interface through different options or menus according to their different usage or setting needs. For simplicity, this embodiment will not elaborate on these details.

[0228] User execution such as Figure 5 As shown in Figure (b), clicking the HyperTerminal option will cause the phone to display the following in response to the user's click: Figure 5 The interface 503 shown in Figure (c) can display one or more electronic devices that currently have a mutual trust relationship with the mobile phone. For example, as... Figure 5 As shown in Figure (c), the interface 503 displays the implementation details of HyperTerminal: it can automatically discover nearby devices with the same account and form a HyperTerminal.

[0229] In addition, interface 503 also displays the icons and names of nearby devices that have been discovered and connected, such as the local device (Mate 40 Pro), the smart screen (V75), and the tablet (Mate Pad). Interface 503 may also include a "More HyperTerminal Devices" option, which allows users to discover more nearby electronic devices, such as those logged into different user accounts in a home setting. Users can further add these devices to the HyperTerminal group, establishing mutual trust between them; details will not be elaborated further here.

[0230] User execution such as Figure 5 As shown in Figure (c), clicking the icon or name of the Smart Screen (V75) will trigger a response from the user's click, and the phone will display the following: Figure 5 The interface 504 shown in Figure (d) can display the details interface of the smart screen.

[0231] For example, such as Figure 5As shown in Figure (d), interface 504 displays multiple options, such as the "Allow Discovery" option, indicating that the user has set the Smart Screen (V75) to be discoverable by nearby devices. Furthermore, interface 504 also displays a list of access permissions granted to the Smart Screen (V75) by the mobile phone. This list includes one or more video resources from different sources that the mobile phone can access and obtain from the Smart Screen (V75). For example, this access permission list includes HDMI video resources, DP video resources, and video resources from Huawei Box, indicating that the mobile phone can access and obtain the HDMI video resources, DP video resources, and video resources from Huawei Box of the Smart Screen (V75). This embodiment of the application does not limit this.

[0232] It should be noted that the access permission list of the 504 error on this interface may contain one or more video resources from different sources that cannot be accessed on the mobile phone. For example, for HDMI video resources, since the mobile phone does not have a video transmission interface such as HDMI, HDMI video resources are content that the mobile phone cannot access when used independently.

[0233] In addition, the access permission list of the interface 504 can also display one or more applications installed on the smart screen (V75). That is, the mobile phone can access and obtain video resources from one or more installed applications on the smart screen (V75), such as Huawei Video application, Tencent Video application, and other applications. These multiple applications can be applications that are not installed on the mobile phone and are only installed on the smart screen (V75), or applications that are installed on both the mobile phone and the smart screen (V75). This application embodiment does not limit this.

[0234] When the user executes such Figure 5 As shown in Figure (d), clicking the "HDMI Video Resources" option in the access permission list of the 504 error on this interface will cause the phone to display the following in response to the user's click: Figure 5 The interface 505 shown in Figure (e) may include a switch for "Get HDMI Video Resources," etc. The user executes... Figure 5 As shown in Figure (e), clicking the "Get HDMI Video Resources" switch on interface 505 responds to the user's click, allowing the mobile phone to send an HDMI video resource request (i.e., a media resource request) to the smart screen (V75). The mobile phone can then display the content shown in interface 505, including a "Resource Loading" message or a loading indicator. Correspondingly, the smart screen (V75) can receive and respond to the HDMI video resource request (i.e., the media resource request) sent by the mobile phone; however, this embodiment does not limit the specific implementation details.

[0235] Figure 6 This is another example of a mobile phone requesting to obtain video resources from a smart screen, provided in an embodiment of this application.

[0236] In another possible implementation, users can use a shortcut to access the phone's HyperTerminal function options, triggering the phone to request the HDMI video resources from the smart screen.

[0237] For example, the user performs such Figure 6 As shown in Figure (a), the operation involves swiping down from the top of the phone. In response to the user's swipe, the phone displays the following... Figure 6 Figure (b) shows the control center interface 602. This control center interface 602 can display various quick settings options or content for the mobile phone, such as WLAN details, Bluetooth details, WLAN switch, Bluetooth switch, mobile data switch, mute switch, auto-rotate switch, display brightness adjustment progress bar, etc., as well as the HyperTerminal window involved in this application embodiment.

[0238] For example, in Figure 6 In the HyperTerminal window shown in Figure (b), icons and names of one or more electronic devices that are part of a HyperTerminal group with the mobile phone can be displayed, such as icons and names of multiple electronic devices including the Smart Screen (V75), Tablet (Mate Pad), and Speaker. When a user clicks on the icon or name display area of ​​the Smart Screen (V75), the mobile phone displays the following in response to the user's click: Figure 6 Interface 603 shown in Figure (c) is followed in sequence. Figure 6 The process shown in Figures (c), (d), (e), and (f) illustrates how, in response to a user's click, the mobile phone can send an HDMI video resource request (i.e., a media resource request) to the smart screen (V75). This process can be referenced... Figure 5 For the sake of simplicity, the relevant descriptions of Figures (c), (d), (e), and (f) will not be repeated here.

[0239] The above describes the possible ways in which a user can trigger an HDMI video resource request (i.e., media resource request) from their mobile phone to the Smart Screen (V75). The Smart Screen (V75) can receive and respond accordingly to the HDMI video resource request (i.e., media resource request) sent by the mobile phone.

[0240] Optionally, in this embodiment, the user can also perform possible operations on the smart screen (V75). The smart screen (V75) does not need to receive HDMI video resource requests (i.e., media resource requests) sent by the mobile phone and can directly respond accordingly. For example, the user's possible operation on the smart screen (V75) triggers the smart screen (V75) to determine the HDMI video resource content to be sent and sends the image data and / or audio data corresponding to the HDMI video to the mobile phone. Then, the user can agree to perform the operation of receiving the image data and / or audio data corresponding to the HDMI video on the mobile phone to realize the playback of the HDMI video resources of the smart screen (V75) on the mobile phone. This embodiment does not limit this.

[0241] 402. In response to the media resource request, the source device detects whether the first preset interface is in a working state. The first preset interface is used to obtain image data and / or audio data corresponding to the first media resource. The first preset interface is any one of one or more of the physical transmission interfaces.

[0242] 403. When the first preset interface is in working state, the source device obtains the image data and / or audio data corresponding to the first media resource through the first preset interface.

[0243] Optionally, the "first preset interface" can be any of the video transmission interfaces such as HDMI and DP described above.

[0244] In one possible implementation, when the driver of the first preset interface is detected to be in a working state, the source device can determine that the first preset interface is in a working state; or, when the source device plays the first media resource in the foreground, the source device can determine that the first preset interface is in a working state.

[0245] For example, when the first preset interface is HDMI, the smart screen can detect its own operating status after receiving the HDMI resource request from the mobile phone. It should also be understood that the smart screen can automatically identify its own HDMI input driver and the working status of the HDMI. Specifically, the smart screen can determine that the HDMI is in a working state in the following ways:

[0246] (1) The smart screen can detect the working status of the HDMI input driver and determine the working status of the HDMI based on the working status of the HDMI input driver.

[0247] It should be understood that when a user is watching HDMI video on a smart screen, an external HDMI device needs to input the raw HDMI video stream through the smart screen's HDMI port. When there is a video stream input through the smart screen's HDMI port, the HDMI input driver is guaranteed to be active. Therefore, the smart screen can detect the active status of the HDMI input driver and determine whether the HDMI port is operational.

[0248] For example, when the flag bit of the HDMI input driver is "1", it indicates that the HDMI input driver is in a working state; when the flag bit of the HDMI input driver is "0", it indicates that the HDMI input driver is in a non-working state. This application embodiment does not limit this.

[0249] (2) The smart screen can detect the currently playing video content.

[0250] It should be understood that when a user is playing a video on the smart screen, the smart screen can determine that the currently playing video originates from HDMI. That is, HDMI video resources must be transmitted to the smart screen via HDMI. Therefore, the smart screen can indirectly determine whether the HDMI is in working condition based on the video content.

[0251] For example, when a user is using a smart screen to play online live video content such as CCTV-1, it is determined that the HDMI is in a working state; when a user is using the Huawei Video application on the smart screen to watch online video resources of a TV series, it is determined that the HDMI is in a non-working state. This application embodiment does not limit this.

[0252] It should be understood that in the above situations (1) and (2), the HDMI input driver and HDMI are in working state, that is, the HDMI video playback channel has been established. Therefore, the process may also include the following steps:

[0253] 404-1 and 404-2: When an HDMI external device inputs the raw bitstream of an HDMI video via HDMI, the smart screen receives and caches the raw bitstream of the HDMI video into a cache module (e.g., a video buffer).

[0254] The smart screen can directly reuse the current HDMI video playback channel, obtain the original bitstream of the HDMI video from the cache module (such as videobuffer) and copy it in real time. The HAL encoding and decoding module then decodes the bitstream to obtain the image data and / or audio data corresponding to the currently playing HDMI video.

[0255] It should be noted that the "caching module (e.g., video buffer)" in steps 404-1 and 404-2 can be HAL or other hardware storage units. When the smart screen plays HDMI video, after decoding the original HDMI bitstream, the smart screen caches the decoded image data and / or audio data according to the data order in the original bitstream. During playback, data is consumed from the cache according to this data order. That is, the image data in the cache module (e.g., video buffer) is sent to the display to ensure the display of the image; the audio data in the cache module (e.g., video buffer) is used to play the sound, thereby ensuring smooth video playback. This will not be elaborated further here.

[0256] Furthermore, in the embodiments of this application, in the above scenarios 1 and 2, when the HDMI input driver and HDMI of the smart screen are in working state, the raw bitstream of the HDMI video can be directly consumed. The "media data stream processing module" also needs to obtain the raw bitstream of the HDMI video from the "caching module (e.g., video buffer)" and perform real-time copying and decoding to obtain the image data and / or audio data corresponding to the HDMI video. This process does not affect the current progress and playback capability of the smart screen playing HDMI video.

[0257] In other words, combining Figure 3 As shown in Figure (a), after the smart screen obtains the raw HDMI video stream in step ①, it decodes the raw HDMI video stream and obtains the corresponding image and / or audio data for normal HDMI video playback. Simultaneously, the raw HDMI video stream is also processed by… Figure 3 The "Media Data Stream Processing Module" shown in Figure (a) copies and decodes the data, and also obtains the image data and / or audio data corresponding to the original bitstream of the HDMI video. Based on the HDMI video resource request (i.e., media resource request) sent by the mobile phone, it continues to execute the following steps 405.

[0258] 405, Encode the image data and / or audio data corresponding to the first media resource to obtain the bitstream corresponding to the first media resource to be sent.

[0259] In one possible implementation, the media resource request also includes information about the target encoding method, which is an encoding method that the target device can support.

[0260] For example, when a smart screen receives an HDMI video resource request from a mobile phone, the HDMI video resource request can indicate not only the source of the requested video resource, but also the bitstream format that the mobile phone can decode, i.e., the mobile phone's encoding and decoding capabilities.

[0261] Optionally, the mobile phone has the ability to decode standard bitstreams such as H264 and / or H265 and / or advanced audio codec (AAC). The target encoding method can be any encoding method that can obtain one or more of the standard bitstreams listed above, and this application embodiment does not limit this.

[0262] Accordingly, the smart screen can determine the data encoding formats that the phone supports based on the target encoding method information, and then... Figure 3 The "encoding and decoding module" of the "media data stream processing module" in Figure (a) encodes the image data and / or audio data corresponding to the HDMI video according to the encoding method to obtain H264 and / or H265 and / or AAC bitstreams, that is, to obtain the bitstream corresponding to the HDMI video to be sent.

[0263] It should be understood that in this embodiment, the "bitstream corresponding to the HDMI video" is a clean video bitstream. For example, when a user watches HDMI video on a smart screen, some advertisements, emergency information, etc., may be added to the smart screen. In specific implementations, an advertisement bitstream is added to the original bitstream of the HDMI video, thus adding the advertisement during the playback of the HDMI video. However, in this embodiment, Figure 3 The "Media Data Stream Processing Module" in Figure (a) obtains the pure raw bitstream of HDMI video from HDMI, excluding added content such as advertisements and emergency information.

[0264] In another possible implementation, developers can add different customized processing schemes for video resources from different sources during the development process, with different customized processing schemes corresponding to different configuration parameters.

[0265] Optionally, before the encoding process in step 405, the smart screen can obtain configuration parameters for the HDMI video resource. These configuration parameters include one or more parameters related to adding or removing identifiers, content overlay, content cropping, ad insertion, and display effects for the image data corresponding to the HDMI video, and / or one or more parameters related to mixing and playing sound effects for the audio data corresponding to the HDMI video. Then, based on the configuration parameters, the image data and / or audio data corresponding to the HDMI video are preprocessed to obtain preprocessed image data and / or audio data corresponding to the HDMI video.

[0266] For example, in the specific implementation process, for the image data in the raw bitstream of HDMI video, Figure 3 The "Customized Processing Module" of the "Media Data Stream Processing Module" in Figure (a) can also perform one or more processing operations, such as adding watermarks, removing watermarks, inserting advertisements, overlaying scrolling emergency information, and image display effects. Regarding the audio data in the raw bitstream of HDMI video, Figure 3 The "Customized Processing Module" of the "Media Data Stream Processing Module" in Figure (a) can also perform one or more processing operations such as audio mixing and adding playback sound effects.

[0267] Then, step 405 is executed, and the image data and / or audio data corresponding to the customized HDMI video are subjected to the encoding processing described above, thereby obtaining the bitstream corresponding to the HDMI video with the target encoding method that the mobile phone can support.

[0268] 406, The source device stores the bitstream corresponding to the first media resource to be sent in the cache module.

[0269] It should be understood that the buffer module in step 406 may be different from the "buffer module (e.g., video buffer)" described in steps 404-1 and 404-2 above.

[0270] For example, the "caching module (e.g., video buffer)" described in steps 404-1 and 404-2 can be a HAL or other hardware storage unit, etc., and the cache module in step 406 can be as follows: Figure 3 As shown in Figure (a), the buffer module, located in the application framework layer, may further include an audio data buffer module and / or an image data buffer module. After the media data stream processing module obtains the image data and / or audio data corresponding to the HDMI video from the raw bitstream copied and decoded from the HDMI video, it can cache the image data in the image data buffer module of the application framework layer and the audio data in the audio data buffer module of the application framework layer, respectively.

[0271] Optionally, step 406 above can be executed automatically during the playback of HDMI video on the smart screen. The buffer module can periodically update the stored image data and / or audio data according to the progress of the HDMI video playback on the smart screen. Upon receiving a media resource request from the mobile phone, in response to the media resource request, image data and / or audio data are obtained from the buffer module, and then encoded according to the process in step 405, which will not be elaborated here.

[0272] Optionally, the above step 406 can occur after the smart screen receives the HDMI video resource request sent by the mobile phone. In response to the HDMI video resource request, the screen obtains the original bitstream of the HDMI video according to its own HDMI video playback process, and performs decoding, encoding, and caching processes. This application embodiment does not limit this.

[0273] 407. In response to the media resource request, the source device obtains the encoded bitstream corresponding to the first media resource from the cache module and sends the bitstream corresponding to the first media resource to the target device, so that the target device can play the first media resource according to the image data and / or audio data contained in the bitstream corresponding to the first media resource.

[0274] For example, in response to an HDMI video resource request sent by a mobile phone (i.e., a media resource request sent by the mobile phone in step 401), the smart screen can obtain image data and / or audio data from the cache module (buffer) of the application framework layer for consumption. That is, according to the order, the transmission protocol of the mobile phone and the smart screen, etc., the encoded HDMI video corresponding bit stream is sent to the mobile phone.

[0275] Accordingly, after receiving the encoded HDMI video stream, the mobile phone can perform decoding to obtain the image data and / or audio data contained in the HDMI video stream, and play the HDMI video based on the contained image data and / or audio data.

[0276] In one possible scenario, during steps 403-407 described above, the smart screen's HDMI port is active, meaning the user is watching HDMI video on the smart screen. In this scenario, the HDMI video played by the mobile phone based on the received HDMI video stream is synchronized with the HDMI video being played on the smart screen. In other words, if the smart screen is currently playing a live broadcast from a TV station, after the streaming media content acquired via HDMI is transmitted to the mobile phone, the mobile phone will also play the same video content as on the smart screen.

[0277] Through the above method, the embodiments of this application can shield the encoding and decoding capabilities of the hardware chips of different electronic devices such as mobile phones and tablets, and be compatible with the encoding capabilities of the chips of different electronic devices for the raw bitstream of HDMI video. The raw bitstream of HDMI video obtained from HDMI is encoded into a unified standard bitstream such as H264 and / or H265 and / or AAC. Then, according to the type of the target device and the encoding method supported by the target device, a specific encoding format is specified to generate the encoded bitstream corresponding to the HDMI video to be sent. This process can be applied to different electronic devices, transmitting HDMI video on the smart screen to portable electronic devices such as mobile phones and tablets, expanding the user's video sources and improving the user's viewing experience.

[0278] 408. When the first preset interface is in a non-working state, the source device generates a driver instruction, which is used to call the driver module of the first preset interface. It should be understood that the "non-working state" in this embodiment can also be called the "non-running state".

[0279] 409. According to the driving instruction, drive the first preset interface to switch from the non-working state to the working state.

[0280] In another possible implementation, the first preset interface is determined to be in a non-working state when the driver of the first preset interface is detected to be in a non-working state; or, the first preset interface is determined to be in a non-working state when the source device is detected to be in a sleep state; or, the first preset interface is determined to be in a non-working state when the source device is playing a second media resource in the foreground, wherein the image data and / or audio data corresponding to the second media resource are obtained through a second preset interface, and the second preset interface is any one of one or more physical transmission interfaces that is different from the first preset interface; or, the first preset interface is determined to be in a non-working state when the source device is playing a third media resource in the foreground, wherein the third media resource is a network media resource.

[0281] For example, when the first preset interface is HDMI, after receiving the HDMI resource request sent by the mobile phone, the smart screen can detect its own operating status. It should also be understood that the smart screen can automatically identify its own HDMI input driver and the working status of the HDMI. Specifically, the smart screen can determine that the HDMI is in a non-working state in the following ways:

[0282] (3) When it is detected that the driver of the first preset interface is in a non-working state, it is determined that the HDMI is in a non-working state.

[0283] For example, referring to the foregoing description, the smart screen can detect the working status of the HDMI input driver and determine whether the HDMI is in a working state. For instance, when the flag bit of the HDMI input driver is "0", it indicates that the HDMI input driver is not in a working state.

[0284] (4) The source device is detected to be in a sleep state, and the HDMI is determined to be in a non-working state.

[0285] Optionally, this sleep state can be understood as a screen-off state where the smart screen is powered on but not turned off and no media content such as video and / or audio is being played.

[0286] For example, when the smart screen does not receive any operation instructions within a preset time, it can enter a sleep state. At this time, the smart screen can display multiple screensaver images in turn, or only display an advertisement image. This application embodiment does not limit this.

[0287] (5) The smart screen can detect the currently playing video content. When the source device is playing any content other than HDMI video, such as DP video or online video of a TV series played on Huawei App, it is determined that the HDMI is in a non-working state.

[0288] For example, the image and / or audio data corresponding to DP video resources are obtained through a physical transmission interface independent of HDMI—DP. When the smart screen's DP input driver obtains DP video resources through DP, it can be determined that both the HDMI input driver and HDMI are in a non-working state. Similarly, when a user plays a TV series' online video resource through the Huawei Video application, the smart screen requests the video resource from the server corresponding to the Huawei Video application. In this case, the smart screen's DP input driver, DP, HDMI input driver, and HDMI are all in a non-working state, which will not be elaborated further here.

[0289] In steps 408 and 409, when the smart screen is not playing HDMI video, the HDMI video playback channel is not established, meaning both the HDMI input driver and HDMI are inactive. After automatically recognizing that HDMI is inactive, the smart screen can generate a driver command to activate the HDMI input driver, enabling it to run in the background and integrate HDMI audio and video driving capabilities to actively drive HDMI and obtain the raw HDMI video stream.

[0290] Once the smart screen's backend obtains the original bitstream of the HDMI video via HDMI, it continues to execute the corresponding processes according to steps 403-405-406-407 described above. For simplicity, these steps will not be repeated here.

[0291] In scenarios (3), (4), and (5) above, when the HDMI input driver and HDMI of the smart screen are not in working state, the above embodiments can generate driver instructions and run the HDMI input driver in the background, thereby accessing the kernel layer of the smart screen. The HDMI input driver actively drives the HDMI to enter working state, obtains the raw bitstream of the HDMI video input from the external device, and directly consumes the raw bitstream of the HDMI video. For example, the raw bitstream of the HDMI video is copied and decoded. After obtaining the image data and / or audio data corresponding to the HDMI video, it is encoded and transmitted to the mobile phone to realize the playback of HDMI video on the mobile phone. This process does not affect the current playback process and playback capability of the smart screen, can obtain a clean raw bitstream of HDMI video, and the process of obtaining the raw bitstream of HDMI video is not affected by the content played on the screen of the smart screen.

[0292] Figure 7 This is a schematic diagram illustrating an example of a mobile phone playing an HDMI video from a smart screen, provided in an embodiment of this application.

[0293] In one possible implementation, when the user follows... Figure 5 The process shown in Figures (a), (b), (c), and (d), or according to... Figure 6 The process shown in Figures (a), (b), (c), and (d) triggers the mobile phone to send an HDMI video resource request to the smart screen. In response to this HDMI video resource request, if the smart screen returns the bitstream containing the image data and / or audio data corresponding to the HDMI video to the mobile phone, the mobile phone can further display... Figure 5 The interface 506 shown in Figure (f) is displayed, or shows Figure 6 The interface 606 shown in Figure (f) is an HDMI video playback interface. Assuming that the current user is watching CCTV-1 live content through the HDMI of the smart screen, this interface 506 (or interface 606) is the CCTV-1 live broadcast interface. The interface content will not be described in detail in this embodiment.

[0294] In the scenarios described above, when a user acquires and plays HDMI video content from a smart screen via their mobile phone, the smart screen can be in a state where HDMI video is being played; or, the smart screen can be in a sleep state; or, the user can use the Huawei Video app on the smart screen to watch a TV series online, and the smart screen is playing that TV series. For any of these scenarios, the method provided in this application embodiment ensures that the process of the mobile phone acquiring the raw bitstream of the HDMI video is unaffected by the content being played on the smart screen, thus enabling the playback of HDMI video on the mobile phone.

[0295] In one possible scenario, when the smart screen is playing HDMI video, the mobile phone, after obtaining the HDMI video stream from the smart screen, can maintain synchronized playback with the smart screen. For example, if the smart screen is playing live CCTV-1 content, the mobile phone will simultaneously play the same live CCTV-1 content.

[0296] For example, in this scenario, if an emergency message scrolls across the video interface or an ad pop-up appears in the lower right corner while the smart screen is playing a live CCTV-1 broadcast, the emergency message will not be displayed on the phone and no ad pop-up will appear because the HDMI video stream obtained by the mobile phone is a clean stream containing only the image and audio data corresponding to the HDMI video.

[0297] Through the above process, users can watch online live media content such as HDMI video and DP video through portable electronic devices such as mobile phones and tablets. This reduces the constraints imposed by the type and number of interfaces of electronic devices on users' access to media content, expands the video sources of portable electronic devices such as mobile phones and tablets, and improves the user experience.

[0298] In addition, the above implementation method can reduce the impact of interfering information such as emergency information and pop-up ads on users' video viewing experience, thus ensuring the user's viewing experience on mobile devices.

[0299] In another possible scenario, when the smart screen can be in sleep mode, or when a user is watching a TV series online through the Huawei Video application on the smart screen and the smart screen is playing the TV series, this application embodiment can also realize the HDMI video stream obtained by the mobile phone from the smart screen for any of the above scenarios, and the user can also play the HDMI video on the mobile phone. In this scenario, the display content of the mobile phone and the smart screen can be displayed without interference.

[0300] For example, such as Figure 7 As shown in Figure (a), suppose a user is watching a video about food preparation online on a smart screen using the Huawei Video app. For example, the smart screen is currently playing the video "Food Tutorial Class Episode 1 - Pizza Making," and the user is simultaneously clicking on their phone... Figure 5 or Figure 6 During the introduction, the HDMI video stream was obtained from the smart screen, and the mobile phone can display the CCTV-1 live broadcast interface 701 based on the obtained HDMI video stream.

[0301] For example, such as Figure 7As shown in Figure (b), assuming that during the playback of the video "Food Tutorial Classroom Episode 1 - Pizza Making" on the smart screen, a call from a family member is received in the home scene, the user sets the call to be transferred to the display screen of the smart screen, as shown in interface 704. The smart screen displays a floating window 704-1 on the interface of playing the video "Food Tutorial Classroom Episode 1 - Pizza Making". The floating window 704-1 displays the call notification content from the user "Mr Wang", such as the avatar, nickname, phone number (138XXXX0493) and location details related to the user "Mr Wang", hang-up control, answer control, etc. The content of the floating window 704-1 is not limited in this embodiment of the application.

[0302] It should be understood that, Figure 7 In the scenarios shown in Figures (a) and (b), the smart screen does not affect the playback of HDMI video on the mobile phone, whether it is used to play videos on demand or to answer calls from other family members.

[0303] Through the above process, the process of the mobile phone acquiring the raw bitstream of HDMI video can be unaffected by the content played on the smart screen. This allows the smart screen and the mobile phone to play different content without interfering with each other. It also supports users to use different functions of the smart screen at the same time, enriching the utilization of the smart screen's capabilities and improving the user experience.

[0304] In summary, the process of acquiring and transmitting the bitstream provided in this application embodiment integrates a media data stream processing module at the HAL layer. This module can intercept the bitstream input from external devices via video transmission interfaces such as HDMI and DP. It can also decode the bitstream, acquiring and caching the image and / or audio data included within it. Furthermore, when a resource acquisition request is received from a mobile phone, tablet, or other electronic device, the module can encode the image and / or audio data included in the bitstream according to the encoding / decoding capabilities of the device, obtaining a bitstream that can be decoded by the device. The encoded bitstream is then transmitted to these devices, enabling the playback of different streaming media content such as HDMI and DP videos on various electronic devices.

[0305] Firstly, portable electronic devices such as mobile phones and tablets generally lack video transmission interfaces such as HDMI and DP, and therefore cannot play HDMI or DP videos. Through the method of this application embodiment, the smart screen can transmit streaming media content input from external devices via video transmission interfaces such as HDMI and DP to portable electronic devices such as mobile phones and tablets for playback. Users can then watch online live media content such as HDMI and DP videos on these devices, reducing the constraints imposed by the type and number of interfaces of electronic devices on users' access to media content, expanding the video sources for portable electronic devices such as mobile phones and tablets, and improving the user experience.

[0306] Secondly, this process is different. Figure 1 The wireless screen mirroring process between the mobile phone and the smart screen shown demonstrates that the media data stream processing module of the smart screen acquires a clean stream containing only the corresponding image and / or audio data through video transmission interfaces such as HDMI and DP. This stream excludes additional content such as emergency information and advertising clips added when the smart screen plays HDMI or DP videos, ensuring smooth playback of HDMI and DP videos on mobile phones, tablets, and other electronic devices. Similarly, if pop-up windows or other interfering information appear during HDMI or DP video playback on the smart screen, this interference information will not be simultaneously displayed on mobile phones, tablets, or other electronic devices, reducing the impact of interference information on the user's viewing experience and improving overall viewing enjoyment.

[0307] Furthermore, the process of obtaining the raw bitstream of HDMI video in this embodiment of the application is unaffected by the content being played on the smart screen. Optionally, the smart screen can be in a state of playing HDMI video, i.e., the HDMI video playback channel has been established. This method can support reusing the current HDMI video playback channel, copying the raw bitstream of HDMI video from the smart screen's cache module in real time, and performing decoding processing to obtain the image data and / or audio data included in the raw bitstream of the HDMI video. This process does not require driving HDMI again, avoiding secondary overhead for the smart screen, and does not affect the original playback capability of the smart screen. When the smart screen is not playing HDMI video, i.e., the HDMI video playback channel has not been established, this method can also support background driving of the HDMI input driver, thereby driving HDMI into a background running state, and obtaining the raw bitstream of HDMI video in real time through HDMI, performing decoding processing to obtain the image data and / or audio data included in the raw bitstream of the HDMI video. The above process can support users to use different functions of the smart screen simultaneously, improving the functional utilization rate of the smart screen.

[0308] For example, targeting Figure 1As shown in Figure (b), in this scenario, users can make video calls with family members through the smart screen, while simultaneously playing HDMI videos received from the smart screen using mobile phones, tablets, and other electronic devices. This scenario supports the implementation of multiple functions of the smart screen, allowing users to use various different functions simultaneously, enriching the utilization of the smart screen's capabilities and enhancing the user experience.

[0309] Furthermore, this application embodiment also supports developers in adding different customized processing schemes for video resources from different sources during the development process. Different customized processing schemes correspond to different configuration parameters. Before encoding the image data and / or audio data included in the original bitstream of HDMI video, the smart screen can also obtain the configuration parameters for the HDMI video resource to achieve preprocessing of the image data corresponding to the HDMI video, such as adding or removing identifiers, overlaying content, cropping content, inserting advertisements, and displaying effects, and / or preprocessing of the audio data corresponding to the HDMI video, such as mixing and playing sound effects. Finally, the preprocessed image data and / or audio data corresponding to the HDMI video are obtained to meet the playback needs of different users.

[0310] Finally, when the smart screen plays streaming media content input from external devices via HDMI, this process mainly relies on the smart screen's decoding capability for the HDMI stream, and then implements the video playback process based on the decoded image data and / or audio data. In contrast, in this embodiment, the media data stream processing module of the smart screen, in addition to decoding the stream, can also encode the image data and / or audio data included in the stream according to the encoding and decoding capabilities of the mobile phone, tablet, or other electronic devices when it receives a resource acquisition request from such devices, obtaining streams with different encoding methods, and then transmitting the encoded streams to the mobile phone, tablet, or other electronic devices. The above process not only utilizes the smart screen's decoding capability but also its encoding capability and other various other capabilities, making full use of the smart screen's chip capabilities and maximizing the utilization of device capabilities.

[0311] It should be noted that, in the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0312] It should be understood that the methods, situations, categories, and classifications of embodiments in this application are for the convenience of description only and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined without contradiction.

[0313] It should also be understood that the terms "first," "second," and "third" in the embodiments of this application are for distinction only and should not constitute any limitation on this application.

[0314] For example, in the embodiments of this application, "first preset interface", "second preset interface", etc., represent different video transmission interfaces of the smart screen, that is, physical transmission interfaces set at the factory by the smart screen for obtaining media resources for online live broadcast.

[0315] For example, in the embodiments of this application, "first media resource", "second media resource", "third media resource", etc., represent video resources from different sources obtained by the smart screen. For example, "first media resource" can represent HDMI video, "second media resource" can represent DP video, and "third media resource" can represent network video that the user requests through the Huawei video application.

[0316] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0317] It should also be noted that in the embodiments of this application, "preset", "fixed value", etc., can be implemented by pre-saving corresponding codes, tables, or other methods that can be used to indicate relevant information in the electronic device (e.g., source device and target device, etc.), and this application does not limit the specific implementation method. For example, "preset order" and "preset interface" in the embodiments of this application.

[0318] It is understood that, in order to achieve the above-mentioned functions, electronic devices include hardware and / or software modules that perform the respective functions. Based on the algorithmic steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0319] This embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0320] With each functional module corresponding to its respective function, the above embodiments illustrate a possible configuration of the electronic device, which may include a display unit, a detection unit, and a processing unit. The display unit, detection unit, and processing unit cooperate to support the electronic device in performing the aforementioned steps and / or other processes related to the technology described herein.

[0321] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0322] The electronic device provided in this embodiment is used to execute the above-described video playback method, and therefore can achieve the same effect as the above-described implementation method.

[0323] When using integrated units, the electronic device may include a processing module, a storage module, and a communication module. The processing module can be used to control and manage the operation of the electronic device; for example, it can support the electronic device in executing the steps performed by the display unit, detection unit, and processing unit. The storage module can support the electronic device in executing stored program code and data. The communication module can support communication between the electronic device and other devices.

[0324] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0325] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment can be a device having... Figure 2 The device with the structure shown.

[0326] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the method for capturing long-exposure images in the above embodiment.

[0327] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the method for capturing long-exposure images in the above embodiment.

[0328] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the method of capturing long exposure images in the above method embodiments.

[0329] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0330] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0331] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0332] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0333] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0334] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0335] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for transmitting a code stream, the method comprising: The method is applied to a source device, the source device and a target device establish a communication connection, the source device comprises one or more physical transmission interfaces for acquiring media resources of online live broadcast, and the method comprises the following steps: Receiving a media resource request sent by the target device, the media resource request is used to request image data and / or audio data corresponding to a first media resource of online live broadcast; In response to the media resource request, detecting whether a first preset interface is in a working state, the first preset interface is an interface for the source device to acquire image data and / or audio data corresponding to the first media resource, and the first preset interface is any one of the one or more physical transmission interfaces; the physical transmission interface is a video transmission interface, and the video transmission interface is used to connect an external device and acquire media resources input by the external device; When the first preset interface is in the working state, acquiring the image data and / or audio data corresponding to the first media resource through the first preset interface; When the first preset interface is in a non-working state, the source device generates a driving instruction, the driving instruction is used to drive the first preset interface to switch from the non-working state to the working state, and the image data and / or audio data corresponding to the first media resource are acquired through the first preset interface; the driving instruction is used to run the first preset interface in the background; Encoding the image data and / or audio data corresponding to the first media resource to obtain a code stream corresponding to the first media resource to be sent; Sending the code stream corresponding to the first media resource to the target device; Before the image data and / or audio data corresponding to the first media resource are encoded, the method further comprises the following steps: Acquiring configuration parameters for the first media resource, the configuration parameters comprise one or more parameters related to the increase, removal, content superposition, content interception, advertisement insertion and display effect of the image data corresponding to the first media resource, and / or one or more parameters related to the mixing and playing sound effect of the audio data corresponding to the first media resource; According to the configuration parameters, pre-processing the image data and / or audio data corresponding to the first media resource to obtain pre-processed image data and / or audio data corresponding to the first media resource.

2. The method of claim 1, wherein, The detection of whether the first preset interface is in the working state comprises: When it is detected that the driving of the first preset interface is in the working state, it is determined that the first preset interface is in the working state; or, When the source device plays the first media resource in the foreground, it is determined that the first preset interface is in the working state; or, When it is detected that the driving of the first preset interface is in the non-working state, it is determined that the first preset interface is in the non-working state; or, When the source device is in a hibernation state, it is determined that the first preset interface is in the non-working state; or, determining that the first preset interface is in a non-working state when the source device foreground plays a second media resource, wherein image data and / or audio data corresponding to the second media resource is acquired through a second preset interface, the second preset interface being any one of the one or more physical transmission interfaces different from the first preset interface; or determining that the first preset interface is in a non-working state when the source device foreground plays a third media resource, wherein the third media resource is a network media resource.

3. The method according to claim 1 or 2, characterized in that, the acquiring of the image data and / or audio data corresponding to the first media resource through the first preset interface comprises: acquiring an original code stream corresponding to the first media resource through the first preset interface; decoding the original code stream corresponding to the first media resource to obtain the image data and / or audio data corresponding to the first media resource.

4. The method according to any one of claims 1 to 3, characterized in that, the media resource request further comprises information of a target encoding mode, the target encoding mode being an encoding mode that can be supported by the target device, and the encoding processing of the image data and / or audio data corresponding to the first media resource to obtain a code stream corresponding to the first media resource to be sent comprises: determining the target encoding mode according to the media resource request; performing the encoding processing on the image data and / or audio data corresponding to the first media resource according to the target encoding mode to obtain the code stream corresponding to the first media resource to be sent.

5. The method according to any one of claims 1 to 4, characterized in that, after the acquiring of the image data and / or audio data corresponding to the first media resource through the first preset interface, the method further comprises: buffering the image data and / or audio data corresponding to the first media resource according to a preset order; and before the encoding processing of the image data and / or audio data corresponding to the first media resource, comprising: acquiring the buffered image data and / or audio data corresponding to the first media resource according to the preset order.

6. The method according to any one of claims 1 to 5, characterized in that, the physical transmission interface comprises one or more of a high-definition multimedia interface (HDMI) and a high-definition digital display interface (DP).

7. An electronic device, comprising: comprising: a display screen; one or more physical transmission interfaces for acquiring online live media resources; one or more processors; one or more memories; a module installed with a plurality of application programs; the memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device performs the method of any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, the computer readable storage medium stores computer instructions, and when the computer instructions run on an electronic device, the electronic device performs the method of any one of claims 1 to 6.

9. A computer program product, characterised in that, when the computer program product runs on a computer, the computer performs the method of any one of claims 1 to 6.

Images