Listening method and device, electronic equipment, storage medium and computer program product

Abstract

The present disclosure relates to a monitoring method and device, electronic equipment, storage medium and computer program product. The method comprises: in response to a trigger event for media playing on a display interface, determining a target cache component to be configured for a target application interface; in response to configuring the target cache component for the target interface, transmitting configuration parameters of the target application interface to the target multimedia control based on a configuration response mechanism between the target cache component and the target multimedia control; wherein the target cache component is created by the target multimedia control in the system framework and is used to provide a cache channel between the target application interface and the composite display service; and in response to the configuration parameters in the target multimedia control being monitored by a monitoring service in the system framework, determining the type of media data displayed on the display interface by the target multimedia control. In this way, by transmitting configuration parameters to the multimedia control, the system side can monitor the display interface and make targeted optimization.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a monitoring method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] With the improvement of hardware and software performance of electronic devices and the continuous improvement of system framework, for electronic devices with display functions, some display interfaces are implemented using specified controls in the system framework; for example, video playback, camera preview and game display screens are all implemented using some multimedia controls on the system side (such as SurfaceView and Texture View).

[0003] In the original logic of the system framework, the multimedia controls and the media data producer are decoupled and have no information binding. Therefore, the system cannot know the type of media data that such multimedia controls will ultimately play on the display interface, which limits the system's ability to optimize and extend the display interface. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a monitoring method, apparatus, electronic device, storage medium, and computer program product. By transmitting configuration parameters to a multimedia control, this disclosure allows the system to monitor the display interface and perform targeted optimizations.

[0005] According to a first aspect of the present disclosure, a monitoring method is provided. The monitoring method includes:

[0006] In response to a media playback trigger event on the display interface, a target cache component to be configured for the target application interface is determined; the target application interface is used to output media data obtained in response to the media playback trigger event.

[0007] In response to configuring the target cache component for the target interface, the configuration parameters of the target application interface are transmitted to the target multimedia control based on the configuration response mechanism between the target cache component and the target multimedia control; wherein, the target cache component is created by the target multimedia control in the system framework and is used to provide a cache channel between the target application interface and the composite display service;

[0008] In response to the listening service in the system framework detecting the configuration parameters in the target multimedia control, the media data type displayed by the target multimedia control on the display interface is determined.

[0009] Thus, this disclosure configures a response mechanism that, when the target application interface configures the target cache component, directly transmits the configuration parameters to the target multimedia control, enabling the system to obtain the media data type played by the multimedia control on the display screen. This allows the system to further optimize the screen display, expanding its ability to optimize display effects.

[0010] In some optional embodiments, the step of transmitting configuration parameters of the target application interface to the target multimedia control in response to configuring the target cache component for the target interface, based on the configuration response mechanism between the target cache component and the target multimedia control, includes:

[0011] In response to the target application interface calling the first interface function of the target cache component to transmit the configuration parameters, the second interface function of the target multimedia component is called back based on a preset callback mechanism;

[0012] Based on the second interface function, the configuration parameters are returned to the target multimedia control.

[0013] The embodiments disclosed herein utilize a callback mechanism between interface functions to promptly and effectively send configuration data back to the multimedia control when the application interface configures the cache component.

[0014] In some optional embodiments, the step of transmitting the configuration parameters in response to the target application interface calling the first interface function of the target cache component, and calling back the second interface function of the target multimedia component based on a preset callback mechanism, includes:

[0015] In response to the target application interface calling the first interface function to transmit the configuration parameters, the second interface function is called back based on the registered object in the target cache component;

[0016] The registered object is registered in the target cache component when the multimedia control creates the target cache component, and is used to implement a reference to the second interface function.

[0017] In this way, by registering objects in the cache component, the multimedia control can implement a callback reference to the second interface function when the first interface function is called. Compared with directly calling back the second interface function in the first interface function, the functions have higher decoupling and the configuration parameters can be transmitted outward more flexibly.

[0018] In some optional embodiments, the configuration parameters include an application interface identifier;

[0019] The system framework's listening service detects the configuration parameters in the target multimedia control and determines the media data type displayed by the target multimedia control on the display interface, including:

[0020] In response to the listening service detecting the application interface identifier, the media data type of the target multimedia control displayed on the display interface is determined based on the application interface identifier;

[0021] The media data types include video types or camera preview image types.

[0022] Thus, because the configuration parameters include the application interface identifier, the system can directly determine the application interface bound to the surface created by the target multimedia control when it obtains the application interface identifier from the target multimedia control, thereby improving the accuracy and efficiency of monitoring and recognizing the display interface.

[0023] In some optional embodiments, the method further includes:

[0024] In response to the monitoring service determining the media data type played by at least one of the target multimedia controls on the display interface, the display scene of the display interface is determined;

[0025] Based on the display scenario, determine the adjustment strategy for the display interface;

[0026] The adjustment strategy includes at least one of the following: display frame rate adjustment, screen refresh rate adjustment, and display window adjustment.

[0027] Thus, by monitoring the configuration parameters in at least one target multimedia control through the system framework, the display scenario of the display interface can be determined, enabling the system to flexibly make targeted display adjustment strategies, thereby further enhancing the comprehensiveness and functional diversity of the system framework.

[0028] In some optional embodiments, determining the adjustment strategy for the display interface based on the display scenario includes:

[0029] In response to the listening service detecting that the configuration parameters in the target multimedia control include description parameters of media data, an adjustment strategy for the display interface is determined based on the display scenario and the description parameters;

[0030] The description parameters include frame rate and / or resolution.

[0031] In this way, by listening to descriptive parameters such as frame rate and resolution that describe the specific display effect of media data, the system can make more detailed display adjustment strategies, thereby further improving the system's optimization capabilities and display effect.

[0032] In some optional embodiments, the method further includes:

[0033] In response to the unbinding of the target application interface from the target cache component, the third interface function of the target multimedia control is called back;

[0034] Based on the third interface function, the configuration parameters in the target multimedia control are cleared.

[0035] Thus, by calling back a third interface function when unbinding the target application interface and the target cache component, the embodiments of this disclosure can improve the real-time performance and accuracy of the monitoring results obtained by the system-side monitoring service.

[0036] According to a second aspect of the present disclosure, a listening device is provided, the device comprising:

[0037] The module is configured to determine the target cache component to be configured for the target application interface in response to a trigger event for media playback on the display interface; the target application interface is used to output the media data obtained in response to the trigger event for media playback.

[0038] The transmission module is configured to, in response to configuring the target cache component for the target interface, transmit the configuration parameters of the target application interface to the target multimedia control based on the configuration response mechanism between the target cache component and the target multimedia control; wherein, the target cache component is created by the target multimedia control in the system framework and is used to provide a cache channel between the target application interface and the composite display service;

[0039] The listening module is configured to respond to the listening service in the system framework listening for the configuration parameters in the target multimedia control, and to determine the media data type displayed by the target multimedia control on the display interface.

[0040] In some optional embodiments, the transmission module is further configured to transmit the configuration parameters in response to the target application interface calling the first interface function of the target cache component, and to call back the second interface function of the target multimedia component based on a preset callback mechanism; and to pass the configuration parameters back to the target multimedia control based on the second interface function.

[0041] In some optional embodiments, the transmission module is further configured to, in response to the target application interface calling the first interface function to transmit the configuration parameters, call back the second interface function based on the registered object in the target cache component; wherein the registered object is registered in the target cache component when the multimedia control creates the target cache component, and is used to implement a reference to the second interface function.

[0042] In some optional embodiments, the configuration parameters include an application interface identifier; the monitoring module is further configured to, in response to the monitoring service detecting the application interface identifier, determine the media data type displayed by the target multimedia control on the display interface based on the application interface identifier; wherein, the media data type includes a video type or a camera preview image type.

[0043] In some optional embodiments, the monitoring module is further configured to, in response to the monitoring service determining the media data type played by at least one of the target multimedia controls on the display interface, determine the display scene of the display interface; and, based on the display scene, determine an adjustment strategy for the display interface; wherein the adjustment strategy includes at least one of the following: display frame rate adjustment, screen refresh rate adjustment, and display window adjustment.

[0044] In some optional embodiments, the listening module is further configured to, in response to listening to the configuration parameters in the target multimedia control, wherein the configuration parameters include description parameters of media data, determine an adjustment strategy for the display interface based on the display scene and the description parameters; wherein the description parameters include frame rate and / or resolution.

[0045] In some optional embodiments, the transmission module is further configured to, in response to the unbinding of the target application interface from the target cache component, call back a third interface function of the target multimedia control; and based on the third interface function, clear the configuration parameters in the target multimedia control.

[0046] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0047] processor;

[0048] Memory used to store computer programs or instructions;

[0049] The processor executes the computer program or instructions to implement the steps of the method in the first aspect described above.

[0050] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing a computer program or instructions, wherein when the computer program or instructions in the storage medium are executed by a processor, the steps of the method in the first aspect described above are implemented.

[0051] According to a fifth aspect of the embodiments of this disclosure, a computer program product is provided.

[0052] When the computer program or instructions are executed by the processor, they implement the steps of the method in the first aspect described above.

[0053] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0054] In the monitoring method proposed in this disclosure, a configuration response mechanism between the target multimedia control and the corresponding target cache component enables the transmission of configuration parameters from the target application interface to the target multimedia control when the target application interface configures the target cache component for writing media data. This improves the information decoupling problem between the multimedia control and the media data producer in the native logic of the system framework, allowing the system to clearly identify the type of media played by the multimedia control on the display interface and thus obtain the image monitoring results for the display interface. Based on these results, the system can then optimize the display, expanding its ability to improve display effects. Furthermore, compared to using a whitelist to monitor the display interface, this disclosure employs a configuration response mechanism to directly transmit configuration parameters to the multimedia control when configuration occurs. This results in more accurate and reliable data, mitigating the problem of monitoring failures or errors caused by whitelist invalidation due to page changes or other factors. Additionally, the system does not need to maintain the whitelist, thus reducing system costs.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0057] Figure 1 This is a flowchart illustrating a listening method according to an exemplary embodiment.

[0058] Figure 2 This is a schematic diagram illustrating the workflow of a media codec according to an exemplary embodiment;

[0059] Figure 3 This is a schematic diagram illustrating a process for displaying media data on a screen, based on relevant technologies.

[0060] Figure 4 This is a schematic diagram illustrating a process for displaying media data on a screen according to an exemplary embodiment.

[0061] Figure 5 This is a block diagram illustrating a listening device according to an exemplary embodiment.

[0062] Figure 6 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0064] Figure 1 This is a flowchart illustrating a listening method according to an exemplary embodiment. For example... Figure 1 As shown, this monitoring method is applied in electronic devices, and the method mainly includes the following steps:

[0065] Step 101: In response to a media playback trigger event on the display interface, determine the target cache component to be configured for the target application interface;

[0066] The target application interface is used to output media data obtained in response to media playback trigger events;

[0067] Step 102: In response to configuring the target cache component for the target interface, the configuration parameters of the target application interface are transmitted to the target multimedia control based on the configuration response mechanism between the target cache component and the target multimedia control.

[0068] The target cache component is created by the target multimedia control in the system framework and is used to provide a cache channel between the target application interface and the composite display service.

[0069] Step 103: In response to the listening service in the system framework listening to the configuration parameters in the target multimedia control, determine the media data type displayed by the target multimedia control on the display interface.

[0070] Here, the monitoring method proposed in this disclosure is applied to an electronic device with display functionality and implemented by a system framework in the operating system (such as the Android system framework). The electronic device to which this method is applied includes a fixed terminal, a mobile terminal, or a portable device; fixed terminals include, but are not limited to, in-vehicle terminals, televisions, etc.; mobile terminals include, but are not limited to, mobile phones, tablets, etc.; portable devices include, but are not limited to, smartwatches, etc., and this disclosure does not impose further limitations in this regard. The aforementioned system framework can be the Android system framework, the HarmonyOS system framework, or the iOS system framework, etc., and this disclosure does not impose further limitations in this regard either.

[0071] Taking the Android system framework as an example, as proposed in this disclosure, the Android system framework includes: Application Layer, Application Framework Layer, Native Libraries Layer, System Services Layer, Hardware Abstraction Layer (HAL), and Linux Kernel Layer. The Application Layer provides applications that users can directly interact with, such as pre-installed system applications or third-party applications (browsers, communication software, etc.). The Application Framework Layer provides various Application Programming Interfaces (APIs) and services required for developing Android applications, managing the application lifecycle and implementing core functions such as resource access. The Native Libraries Layer consists of a series of native libraries that provide core functional interfaces for the operating system, such as graphics rendering (OpenGL ES, Skia), multimedia processing (Stagefright, OpenSL ES), and network communication (libnetutils). These libraries are typically written in C or C++, providing high-performance underlying support for upper-layer applications and frameworks. The System Services Layer provides system-level services running in the background, providing underlying support for the Application Framework Layer. The Hardware Abstraction Layer (HAL) provides abstract interfaces for hardware devices (such as cameras, displays, and sensors), allowing upper-layer applications and frameworks to access hardware functions through these interfaces. The Linux kernel layer provides core system functions, such as process management, memory management, file systems, network stacks, and drivers, providing stable foundational support for the entire system.

[0072] It should be noted that, for the native system logic, the multimedia control (such as SurfaceView or TextureView) in the system's framework layer creates a cache component (surface). The surface provides a cache queue BufferQueue, which serves as a cache channel between producers and consumers. Media data producers write data to the BufferQueue, and media data consumers (such as the composite display service, SurfaceFlinger, as proposed in this disclosure) read data from the BufferQueue, and then, based on the WindowManager Service (WMS), composite the data from each surface and send it to the display screen after composite.

[0073] However, in this original logic, the multimedia control is only responsible for creating and managing the surface, thereby enabling the media data within the surface to be read and played on the display interface. The multimedia control is unaware of the specific data content within the surface, nor does it know which media data producer the data comes from. Furthermore, the surface only provides the function of media data consumers reading media data from the BufferQueue, hiding the data source from the outside world. Therefore, the various services in the framework layer that interact with the multimedia control, as well as the services in the system service layer, cannot know the type of media data displayed in the specified window of the display interface corresponding to the multimedia control's surface. In other words, when this multimedia control is used for media data playback, the system side (referring to the application framework layer and system service layer in the system framework) cannot know the actual display status of the current display interface.

[0074] In view of this, the embodiments of this disclosure, through the scheme described in steps 101 to 103, enable multimedia controls (such as SurfaceView or TextureView) to obtain the configuration parameters of the application interface corresponding to the surface, thereby determining which media data producer the media data in the surface held by SurfaceView or TextureView comes from. In this way, the listening service located below the application layer in the system framework (specifically set in the application framework layer or system service layer) can determine the current display status of the display interface. By opening this listening capability, the system side can perform targeted optimization of the current display interface.

[0075] In step 101 above, the media playback trigger event occurring on the display interface can be: a click event on a media application (such as a camera application, live streaming application, or short video application) or a media playback control of a non-media application (such as a video call control in a social application, or a video playback control in a map / food delivery application), or a voice selection event on the aforementioned program or control. At this time, based on the media playback trigger event, the application layer sends a media playback task to the framework layer. The application layer will choose to send the task to the multimedia control (either a SurfaceView or a TextureView) in the framework layer, instructing the SurfaceView or TextureView to create a cache component (surface); and will send a task to the media data producer, instructing the media data producer in the framework layer to determine the target surface to be configured based on the connection method provided by the libgui library in the native library layer.

[0076] It should be noted that the media data producer end in this disclosure includes a camera application interface (Camera API) related to camera applications and a media codec application interface (MediaCodecAPI) related to video playback functions.

[0077] In step 102 above, the media data producer (Camera API or MediaCodec API) of the framework layer configures the target surface and outputs media data to the buffer queue of the surface according to the frame rate, resolution, rendering method or media data format indicated by the application layer.

[0078] Here, the Camera API is a set of system interfaces provided by the system side to the application side for using camera applications. In camera preview and camera capture scenarios, the Camera API interface CameraCaptureSession can be used to configure media data parameters (such as frame rate, resolution, rendering method, and data format). These configurations are then transmitted to the hardware layer (camera and image sensor, etc.) through the camera service of the framework layer, and the media data generated according to these parameter configurations is obtained from the hardware layer.

[0079] The MediaCodec API is an application interface corresponding to a media codec (MediaCodec), which implements the encoding and decoding of media data. A codec is a tool that processes input data and generates new data. It can perform data processing asynchronously, using a set of input and output buffers to implement the encoding and decoding process. In the video playback scenario proposed in this disclosure, the codec is responsible for decoding video data. Specifically, the MediaCodec API uses MediaExtractor to extract encoded data from media files (such as video or audio files). In response to application layer instructions, the MediaCodec API uses the MediaFormat method to set decoding parameters (such as resolution, frame rate, etc.) and starts the codec to decode and generate media data based on these parameters.

[0080] See Figure 2 , Figure 2 This is a schematic diagram illustrating the workflow of a media codec according to an exemplary embodiment; Figure 2 The decoding process shown is as follows: an empty input buffer 21 is requested, the producer (media file) fills it with encoded data to obtain a filled input buffer 22, and then the codec is used for decoding; the codec outputs the decoded media data to a filled output buffer 23, and then the consumer (such as the surface proposed in this embodiment) obtains it from the decoder; in addition, due to buffer limitations, some data that cannot be read in time will be discarded from the discarded output buffers 24.

[0081] In this embodiment of the disclosure, the system libgui library defines four connection methods for the MediaCodec API and Camera API, as follows: First, NATIVE_WINDOW_API_EGL, which indicates the connection type when the native control is hardware accelerated for rendering; Second, NATIVE_WINDOW_API_CPU, which indicates the connection type when the native control is CPU rendered; Third, NATIVE_WINDOW_API_MEDIA, which indicates the connection type when the system is hardware decoded; Fourth, NATIVE_WINDOW_API_CAMERA, which indicates the type when directly connecting to the camera.

[0082] The connection methods defined by "NATIVE_WINDOW_API_MEDIA" and "NATIVE_WINDOW_API_CAMERA" are direct connections between the application interface and the surface. In this method, media data decoded by MediaCodec and preview image data frames obtained from the Camera API are directly displayed on the screen via the Surface, unlike the other two connection methods which involve secondary rendering by the application. Since the media data in the other two methods still requires application rendering, the relationship between the media data and multimedia controls / surfaces can be captured during the rendering process. Therefore, this disclosure only discusses how, in the scenario of direct configuration of the surface using MediaCodecAPI and Camera API, the multimedia controls can identify whether the data held by the surface comes from MediaCodecAPI or CameraAPI.

[0083] In this embodiment of the disclosure, configuration is performed when using the MediaCodecAPI or Camera API. This mainly includes configuring the target cache component through the connection method defined by the system libgui library, and configuring the description parameters of the media data, such as the frame rate, resolution, rendering method, or media data format mentioned above. Some of these configuration parameters can reflect the type of media data in the surface, or the media data producer from which it originates.

[0084] Based on this, in step 103 above, this embodiment of the present disclosure uses a pre-set configuration response mechanism to obtain the configuration parameters of the MediaCodecAPI or Camera API through the surface when the MediaCodecAPI or Camera API configuration surface occurs, and in response to the occurrence of the configuration event, sends the configuration parameters back to the multimedia control (SurfaceView or TextureView) holding the surface.

[0085] In some examples disclosed herein, the above configuration response mechanism can be implemented by setting a response interface on the surface. The methods encapsulated in the response interface can call the receive method in the multimedia control that created the surface, thereby sending the configuration parameters corresponding to the application interface to the multimedia control that created the surface.

[0086] Here, the configuration parameters of the application interface can be bound to the underlying layer of the surface when the application interface configures the surface. In this case, the method encapsulated by the response interface can periodically scan the underlying layer of the surface for configuration parameters after the surface is created. If configuration parameters are scanned, it proves that the surface has been configured by an application interface, and the response indicates successful configuration, sending the scanned configuration parameters to the multimedia control. If no configuration parameters are scanned, the surface is considered to be in an idle state, and no response is required. In addition, after scanning configuration parameters, the interval for the next scan can be increased to accommodate the reconfiguration of the surface by other media data producers.

[0087] In other examples of this disclosure, the above-described configuration response mechanism can also be implemented through a listener object created by the multimedia control in the surface; here, the listener object is generally used to listen to the creation and destruction of the surface, so that the multimedia control that creates the surface can manage the lifecycle of the surface. Thus, embodiments of this disclosure can modify the methods in the listener object to enable the listener object to listen to the target application interface's configuration of the surface, thereby calling back the interface function of the multimedia control that created the surface and transmitting the configuration parameters to the multimedia control.

[0088] In this embodiment of the disclosure, the listening service of the system framework can be located in the framework layer or the system service layer. It can listen to the target multimedia control. When configuration parameters are passed in to the target multimedia control, the configuration parameters can be read and the media data type displayed in the corresponding display window of the target multimedia control on the display interface can be determined according to the configuration parameters.

[0089] In some optional embodiments, the configuration parameters include the application interface identifier;

[0090] In response to the listening service in the system framework detecting configuration parameters in the target multimedia control, the system determines the media data type displayed on the display interface by the target multimedia control, including:

[0091] In response to the listening service detecting the application interface identifier, the media data type of the target multimedia control displayed on the display interface is determined based on the application interface identifier; wherein, the media data type includes video type or camera preview image type.

[0092] Here, the Camera API directly configures the surface using the connection method "NATIVE_WINDOW_API_CAMERA" defined in the system libgui library; the MediaCodec API directly configures the surface using the connection method "NATIVE_WINDOW_API_MEDIA". Thus, the configuration parameters include the connection method defined in the system libgui library, which carries the application interface identifier.

[0093] Thus, when the connection method field is passed to the target multimedia control through the configuration response mechanism, and the listening service obtains the connection method field from the target multimedia control, it can directly parse the connection method field to obtain the application interface identifier, and determine whether the application interface bound to the surface is the Camera API or the MediaCodec API based on the application interface identifier. In this way, it can know whether the media data in the surface held by the target multimedia control is a camera preview image or a video frame after media decoding.

[0094] It should be noted that when the target surface in this disclosure is created by the multimedia control SurfaceView, SurfaceView holds an independent surface, corresponds to an independent WindowState in the WMS service, and generates an independent layer in SurfaceFlinger. The media data in the surface held by SurfaceView is directly composited onto the screen by SurfaceFlinger, without relying on the main window's drawing process; that is, it is fixed to the bottom layer of the window through "holes". When the target surface in this disclosure is created by the multimedia control TextureView, TextureView itself does not hold an independent surface. Instead, it binds the content to the main window's surface through SurfaceTexture (OpenGL ES texture), and is ultimately composited uniformly by SurfaceFlinger. This method follows the conventional view hierarchy rules and supports dynamic adjustment. Therefore, after the listening service determines whether the media data in the surface created by the target multimedia control is a camera preview image or a video frame, it can know whether the display interface formed by the data composited and sent to the display screen by SurfaceFlinger is a camera page or a video playback page. In this way, the monitoring service can make targeted optimizations to the camera preview and / or video playback, such as dynamically adjusting the video frame rate and dynamically adjusting the brightness of the camera preview, etc., which are not limited in this disclosure.

[0095] In this embodiment of the disclosure, since the configuration parameters include the application interface identifier, when the system obtains the application interface identifier from the target multimedia control, it can directly determine the application interface bound to the surface created by the target multimedia control, thereby further determining whether the display interface includes a video playback page or a camera preview page, improving the accuracy and efficiency of monitoring and identifying the display interface, and also expanding the system's optimization capabilities for these two types of display interfaces.

[0096] Of course, in some other optional embodiments of this disclosure, the configuration parameters also include media data format. It is understood that the video format output by the MediaCodecAPI is typically H.264, H.265, MPEG-4, etc.; the image format output by the Camera API is typically JPEG, RAW, YUV, etc. Thus, after the media data format is transmitted to the target multimedia control through the above configuration response mechanism, the listening service can also obtain the media data format and determine the media data type in the surface by analyzing the data format.

[0097] In the monitoring method proposed in this disclosure, a configuration response mechanism between the target multimedia control and the corresponding target cache component enables the transmission of configuration parameters from the target application interface to the target multimedia control when the target application interface configures the target cache component for writing media data. This improves the information decoupling problem between the multimedia control and the media data producer in the native logic of the system framework, allowing the system to clearly identify the type of media played by the multimedia control on the display interface and thus obtain the image monitoring results for the display interface. Based on these results, the system can then optimize the display, expanding its ability to improve display effects. Furthermore, compared to using a whitelist to monitor the display interface, this disclosure employs a configuration response mechanism to directly transmit configuration parameters to the multimedia control when configuration occurs. This results in more accurate and reliable data, mitigating the problem of monitoring failures or errors caused by whitelist invalidation due to page changes or other factors. Additionally, the system does not need to maintain the whitelist, thus reducing system costs.

[0098] In some optional embodiments, in response to configuring a target cache component for a target interface, configuration parameters of the target application interface are transferred to the target multimedia control based on the configuration response mechanism between the target cache component and the target multimedia control, including:

[0099] In response to the target application's API call to the target cache component's first interface function to transmit configuration parameters, the target multimedia component's second interface function is called back based on a preset callback mechanism.

[0100] Based on the second interface function, the configuration parameters are returned to the target multimedia control.

[0101] Here, the target caching components include a buffer queue (BufferQueue), a producer-side interface (BufferProducer), and a consumer-side interface (BufferConsumer). BufferProducer is the producer-side interface of BufferQueue, used to establish connections with the MediaCodecAPI or Camera API. Specifically, it is implemented by the underlying surface and responsible for submitting media data to the BufferQueue. The core methods of the BufferProducer interface mainly include: connect(), used to initialize the connection with the producer, called when configuring the surface on the media data producer side, as the first interface function mentioned above; dequeueBuffer(), used to request a free buffer based on the producer's configuration parameters; queueBuffer(), used to fill the buffer with data from the producer side and submit it to the queue; and disconnect(), used to disconnect from the media data producer side. BufferConsumer is the consumer-side interface of BufferQueue, used to connect to the media data consumer side through interface methods, enabling the media data consumer side to read media data from the BufferQueue. In the direct-connection on-screen display scenario proposed in this disclosure, the media data consumer side is SurfaceFlinger. In some scenarios where the media data is not directly connected to the screen for display, the media data consumer can be other rendering controls.

[0102] See Figure 3 , Figure 3 This is a schematic diagram illustrating a process for displaying media data on a screen, based on relevant technologies. Figure 3 This demonstrates a process for natively decoding video in the Android system and directly displaying the camera preview on the screen. In response to a camera preview selection or a video playback selection, the application layer can trigger either a first multimedia control 31 or a second multimedia control 33 to create a cache component 32. The first multimedia control 31 is a SurfaceView, which creates a surface through a first multimedia interface 311 (SurfaceHolder); the second multimedia control 32 is a TextureView, which creates a surface through a second multimedia interface 331 (SurfaceTexture).

[0103] In response to a camera preview trigger event, the Camera API (34) configures the surface; or, in response to a video playback trigger event, the Media Codec API (35) configures the surface. Here, the Camera API uses the `createCaptureSession` interface to set configuration parameters and bind the surface configuration; the Media Codec API uses the `configure` interface to set configuration parameters and bind the surface configuration. Specifically, when configuring the target surface, either Media Codec or Camera calls the `connect()` method in `BufferProducer` to connect to the target surface and pass configuration parameters (connection method, frame rate, resolution, and / or media data format, etc.) to the surface. Furthermore, when calling the `connect()` method in `BufferProducer`, it negotiates with the surface to cache a buffer that meets the aforementioned frame rate, resolution, and media format, and returns it to Media Codec or Camera, completing the configuration binding between Media Codec and the surface or Camera and the surface, thus establishing a producer-consumer relationship.

[0104] Here, after the MediaCodec API or Camera API is configured with the target surface, the video frames decoded by the decoder and the camera preview image frames are sent directly to the compositing and display service 36 (SurfaceFlinger) for display via a direct data link (camera application interface 34 or media codec application interface 35 - producer interface 321 - buffer queue 322 - consumer interface 323) without undergoing secondary rendering by the application. The producer interface 321 of the buffer component 32 manages the production and consumption of graphics data buffers in the graphics system; both video and camera preview displays must pass through this stage.

[0105] In this embodiment, the connection process described above is a direct connection process. The multimedia control holds the surface, so the system only needs to set a callback mechanism as the configuration response mechanism when the MediaCodec API or Camera API calls the BufferProducer.connect() function to configure the surface. This callback mechanism calls back the second interface function SurfaceView.connect() or TextureView.connect() in the target multimedia control (such as SurfaceView or TextureView). During the callback, the configuration parameters passed to the underlying surface via BufferProducer.connect() are returned to the SurfaceView or TextureView. In this way, based on the return mechanism, the Surface can transmit information representing video playback and camera preview to the multimedia control, allowing upper-layer services on the system side (such as monitoring services) to obtain the MediaCodec or Camera configuration parameters, thereby enabling timely and effective monitoring of video playback and camera preview.

[0106] In this embodiment of the disclosure, a callback to the second interface function can be declared in the first interface function. Specifically, when the first interface function receives configuration parameters passed from the MediaCodec API or Camera API, such as the connection method, frame rate, or resolution mentioned above, the second interface function is called to return these parameters to the second interface function.

[0107] The embodiments of this disclosure utilize a callback mechanism between the first interface function and the second interface function to promptly and effectively send configuration data back to the multimedia control when the application interface configures the cache component, thereby improving the system's monitoring efficiency of the display interface.

[0108] In some optional embodiments, in response to the target application interface calling the first interface function of the target cache component to transmit configuration parameters, a second interface function of the target multimedia control is called back based on a preset callback mechanism, including:

[0109] In response to the target application's API call to the first API function to transmit configuration parameters, the second API function is called back based on the registered object in the target cache component.

[0110] The registration object is registered in the target cache component when the multimedia control creates the target cache component, and is used to implement the reference to the second interface function.

[0111] See here. Figure 4 , Figure 4 This is a schematic diagram illustrating a process for displaying media data on a screen according to an exemplary embodiment. For example... Figure 4 As shown, in order to effectively implement the callback of the second interface function to the surface, this embodiment of the disclosure can register the multimedia control to the cache component 32 through a registration object 4 when the first multimedia control 31 or the second multimedia control 33 creates the cache component 32 (i.e., SurfaceView and TextureView initialize the Surface). (SurfaceView registers an object SurfaceViewListener in the surface, and TextureView registers an object TextureViewListener in the surface). This registration object 4 is used to implement the reference to the second interface function in the multimedia control that registers the object.

[0112] The second interface function `SurfaceView.connect()` in `SurfaceView` or `TextureView.connect()` in `TextureView` defines the data type that needs to be called back from the surface; see the example:

[0113] connect(

[0114] int width;

[0115] int height;

[0116] int connectType;

[0117] int frameRate;

[0118] (void);

[0119] The above connect() example demonstrates that the second interface function can obtain the media data configuration resolution (corresponding to the width and height of the image / video frame), connection type (connectType, carrying the application interface identifier), and frame rate (frameRate) from the configuration parameters of the target application interface when it is called.

[0120] It should be noted that, since the embodiments of this disclosure implement function references through object registration, the framework layer can also update the code of the second interface function in different scenarios, thereby flexibly obtaining the required configuration parameters.

[0121] In this way, by registering an object in the cache component, the multimedia control can implement a callback reference to the second interface function when the first interface function is called. Compared with directly calling back the second interface function in the first interface function, the first interface function does not need to know the implementation details of the second interface function, and the functions have higher decoupling. In addition, by implementing the callback reference by registering an object, different second interface functions can be adapted, thereby enabling the transmission of different configuration parameters to the multimedia control in different scenarios, which is more flexible.

[0122] In some optional embodiments, the above-described monitoring method further includes:

[0123] In response to the listening service, determine the media data type played by at least one target multimedia control on the display interface, and determine the display scenario of the display interface;

[0124] Based on the display scenario, determine the adjustment strategy for the display interface;

[0125] The adjustment strategies include at least one of the following: display frame rate adjustment, screen refresh rate adjustment, and display window adjustment.

[0126] Here, the listening service in the system framework can monitor one or more multimedia controls in the framework layer and capture the configuration parameters passed in from the multimedia controls in a timely manner. Thus, the listening service in this embodiment can determine the media data type displayed by each target multimedia control in the corresponding display window, thereby determining which application interface the current display is in.

[0127] In some examples, if the monitoring service detects that one target multimedia control is playing media of the type camera image, it can be concluded that the current display interface is running a camera application and previewing a camera image. If the monitoring service detects that two target multimedia controls are playing media of the type camera image, it can be concluded that the current display interface is running a camera application, with one window used for previewing the camera image and the other window used to display the previously captured image. In these cases, the system can determine that the current display scenario is a camera application usage scenario, and then the system can make corresponding display adjustment strategies based on the actual usage state of the electronic device.

[0128] For example, if the system determines that the electronic device is in a hovering state (e.g., the device has been rotated to a certain angle), the system can proactively trigger the redraw logic of the SurfaceView or TextureView to keep the camera and screen orientation consistent. At the same time, the layout parameters of each surface are dynamically adjusted through the WindowManager Service, thereby adjusting the display window to match the physical shape of the screen when hovering.

[0129] For example, when a camera usage scenario is identified and the system determines that the folding device is unfolding from the inner screen to the outer screen, compatibility issues may arise due to the difference between the inner and outer screens. Accordingly, embodiments of this disclosure can use a system reverse adaptation scheme in the system framework to adapt and adjust to this abnormal phenomenon. For example, in a camera scenario, the camera page will be stretched. At this time, the system can actively adjust the size of the surface transmitted after camera decoding, that is, realize the size adjustment of the display window.

[0130] In other examples, if the monitoring service detects that a target multimedia control is playing video, it concludes that the current display is playing media data, indicating a user is watching video. In this case, the system can then adjust its display strategy based on the actual situation of the electronic device.

[0131] For example, in video scenarios, embodiments of this disclosure can instruct the composite display service to increase the screen refresh rate or video display frame rate, thereby improving the video display effect. Additionally, the display window can be adjusted in conjunction with other state conditions of the electronic device.

[0132] For example, when the system determines that the electronic device is consuming high power, it can notify SurfaceFlinger's synthesis strategy to reduce the frame rate of the data sent for display or reduce the screen refresh rate, thereby reducing system power consumption and saving power.

[0133] It should be noted that, in this embodiment of the disclosure, when the system detects the type of media data being played on the display interface, optimizations for other functions can be adaptively extended. For example, if the current display interface is detected to be in a video scene, the audio and video management service can be controlled to optimize audio and video synchronization effects, stereo effects, etc.

[0134] Thus, by listening to the configuration parameters in at least one target multimedia control through the system framework, the display scenario of the display interface can be determined. By opening up this public capability, the system can flexibly make targeted display adjustment strategies, further enhancing the comprehensiveness and functional diversity of the system framework.

[0135] In some optional embodiments, an adjustment strategy for the display interface is determined based on the display scenario, including:

[0136] In response to the listening for configuration parameters in the target multimedia control, where the configuration parameters include description parameters of the media data, an adjustment strategy for the display interface is determined based on the display scenario and the description parameters;

[0137] The description parameters include frame rate and / or resolution.

[0138] Specifically, when the surface responds to the callback mechanism and sends configuration parameters back to the SurfaceView or TextureView, including descriptive parameters describing the specific display effect of media data, such as frame rate and resolution configured for the camera or media codec, the listening service can determine the specific frame rate and / or resolution of the media data displayed in the foreground. In this case, the listening service can make more detailed adjustments to the display based on the current state of the electronic device.

[0139] In some examples, if the system recognizes that the current display interface is in a video playback scene and obtains the actual frame rate of the video, it can notify the SurfaceFlinger compositing and display service to adjust the refresh rate according to the matching rules of frame rate and refresh rate. Combined with the VSYNC signal synchronization mechanism of the Choreographer service, the video frame rate and screen refresh rate can be dynamically matched. The output frame rate of the codec can also be dynamically adjusted through CameraService or MediaCodec to reduce redundant rendering.

[0140] In other examples, once the actual video frame rate and resolution are determined on the system side, the display frame rate and resolution can be dynamically optimized based on the current power consumption of the electronic device. Optionally, this disclosure can determine the target video frame rate and / or target video resolution matching different power consumption stages based on the stage of power consumption of the electronic device (low power stage, normal power stage, and high power stage). If the frame rate in the currently passed configuration parameters does not match the target frame rate corresponding to the current power consumption stage, the current video frame rate can be adjusted to increase or decrease towards the target frame rate; similarly, if the resolution in the currently passed configuration parameters does not match the target resolution, it can also be adjusted to increase or decrease towards the target resolution.

[0141] In this way, by listening to descriptive parameters such as frame rate and resolution that describe the specific display effect of media data, the system can make more detailed display adjustment strategies, thereby further improving the system's optimization capabilities and display effects.

[0142] In some optional embodiments, the above-described monitoring method further includes:

[0143] In response to the unbinding of the target application interface from the target cache component, the third interface function of the target multimedia control is called back;

[0144] Based on the third interface function, clear the configuration parameters in the target multimedia control.

[0145] In this embodiment of the disclosure, it is considered that a cache component created and held by a multimedia control may switch the bound media data producer during use. In some examples, if the display interface is a camera preview interface and a camera switching event is triggered on the display side, the application layer responds to the event, instructing the first camera application interface to unbind the corresponding target cache component and instructing the second camera application interface to bind the target cache component. The first camera application interface is used to respond to the application layer's instruction to read data collected by the front-facing camera, encode and render it, and output it to the cache queue of the target cache component; the second camera application interface is used to respond to the application layer's instruction to read data collected by the rear-facing camera, encode it, and output it. In other examples, if the camera preview is turned off on the display interface and the video recording function is switched to in the same window, the application layer instructs the camera application interface to unbind the target cache component and controls the multimedia codec interface to bind to the target cache component.

[0146] In the above example, the unbinding and rebinding (i.e., configuration) between the application interface and the cache component occurred. If the multimedia control is unaware of the unbinding event, the system-side listening service will receive incorrect listening results. Therefore, this embodiment of the present disclosure can clear the configuration parameters in the multimedia control when the unbinding event between the target application interface and the cache component occurs.

[0147] In this embodiment of the disclosure, when the target application interface is unbound from the cache component, it is necessary to call the fourth interface function of the cache component, such as release() or disconnect(), to release the surface resources. Thus, this disclosure can call back the third interface function of the target multimedia control holding the cache component when the fourth interface function is called, and clear the cached configuration parameters in the target multimedia control based on the third interface function.

[0148] The implementation of calling back the third interface function of the target multimedia control holding the cache component when the target application interface calls the fourth interface function can be similar to calling the second interface function. For example, the third interface function can be called back directly in the fourth interface function; or, for example, when the fourth interface function release() or disconnect() in the BufferQueueProducer interface of the target cache component is called, the third interface function is called back through the reference corresponding to the aforementioned registered object (SurfaceViewListener or TerrainViewListener), and the configuration parameters in the target multimedia control are cleared according to the clearing method corresponding to the third interface function.

[0149] It should be noted that if the target cache component is re-bound by the same or other multimedia data producer after binding, the second interface function can be called back during reconfiguration to send the updated configuration parameters back to the target multimedia control, thus enabling the configuration parameters to be written again.

[0150] Thus, by calling back the third interface function when unbinding the target application interface and the target cache component, the cache state of the configuration data in the multimedia control can be synchronized with the binding state between the target application interface and the target cache component, thereby improving the real-time performance and accuracy of the monitoring results obtained by the system-side monitoring service.

[0151] The following example illustrates the above listening method using a mobile phone with the Android system framework.

[0152] Currently, some special interfaces on mobile phones are implemented using specialized controls. For example, video playback, camera preview, and game interfaces all use multimedia controls such as SurfaceView and TextureView. In the system's native logic, multimedia controls are not bound to video or camera information, so it's impossible to know whether the multimedia control is currently playing video or using the camera. We need the ability to identify whether a multimedia control is playing video or using the camera. By opening up this public capability, the system can perform many targeted optimizations: such as optimizing power consumption for dynamic frame rates in video and camera interfaces; implementing hover functionality for video scenes; and performing reverse adaptation for compatibility in continuous scenes, etc.

[0153] Currently, Android systems lack a precise solution for recognizing video and camera interfaces. All video- or camera-based functions typically rely on a list system. However, this list can become ineffective due to changes in application interfaces or version upgrades. This leads to several problems: these functions may malfunction without the user's knowledge, resulting in instability and a poor user experience. Furthermore, as the list changes, real-time testing and maintenance are required, incurring high manpower costs and significant investment. Because the native system does not support querying the content type loaded by current multimedia controls, precise recognition of video or camera interfaces remains limited, restricting the system's ability to optimize and adjust display effects.

[0154] In traditional application frameworks, when an application uses multimedia controls to load video or camera content, the multimedia control itself allocates a separate surface. When content needs to be played, it connects to the video module (MediaCodec) or the camera module (Camera). The system's libgui library defines the four connection methods described above in this disclosure. Based on these system-level defined connection methods, the MediaCodec API or Camera API can be configured and bound to the surface. This disclosure primarily focuses on direct connections where the MediaCodec API or Camera API directly binds to the surface.

[0155] Direct connection refers to the direct display of decoded video data from the MediaCodec API and camera preview data from the Camera API directly onto the screen via the surface, without secondary rendering by the application. Optionally, the direct display process is... Figure 2 As shown in the diagram. When the application connects to the MediaCodec API or Camera API, the application's settings and camera parameters (frame rate, resolution, rendering method, and media data format, etc.) are bound to the surface via the MediaCodec API or Camera API. This surface is constructed by the SurfaceView or TextureView using SurfaceHolder or SurfaceTexture respectively when the application wants to display video or camera content.

[0156] The above connection process is a direct connection process. After the Surface is configured, the data frames decoded by the decoder and previewed by the camera are sent directly to SurfaceFlinger for display through the above data link without secondary rendering by the application.

[0157] In the above-mentioned direct connection scenario, this disclosure provides a monitoring method applied to the system side. During the configuration of the surface using the Camera API or MediaCodec API, the configuration parameters can be sent back to the multimedia control (such as SurfaceView or TextureView) holding the surface via the direct connection link. Thus, the system side can obtain the configuration parameters for video playback and camera preview. The configuration parameters can be obtained through the monitoring service, and the video playback and camera preview on the display interface can be monitored accordingly.

[0158] Optionally, embodiments of this disclosure can be combined with Figure 4The illustrated process involves passing configuration parameters back to the multimedia controls. The specific steps are as follows: First, when the SurfaceView or TextureView initializes the Surface, the multimedia control registers itself with the Surface as an object using SurfaceViewListener or TextureViewListener. Simultaneously, the `connect()` interface function is implemented in both the TextureView and SurfaceView to set callback parameters, including the aforementioned frame rate, resolution, and connection method. Second, when the MediaCodec API configures the Surface via the `configure` interface and the Camera API configures the Surface via the `createCaptureSession` interface, the `connect` method of `BufferQueueProducer` is called. At this point, the Surface checks if a registered SurfaceViewListener or TextureViewListener exists. If so, the `connect` method of the multimedia control's SurfaceView and / or TextureView is called back based on the registered SurfaceViewListener or TextureViewListener, while simultaneously passing the aforementioned configuration parameters to the SurfaceView or TextureView. In this way, the system can obtain the relevant data from the multimedia controls for targeted optimization operations, such as optimizing the dynamic frame rate of the video and camera interface, and improving power consumption.

[0159] This disclosure achieves the following effects by proposing the above-described monitoring method: First, Android natively does not support binding camera and video to multimedia controls, thus failing to accurately identify whether the current interface contains camera and video playback. This disclosure enables this function through the above method, expanding system functionality. Second, after accurately identifying the camera and video interfaces on the system side, the system can be enhanced and optimized accordingly: such as optimizing the dynamic frame rate of the video and camera interfaces to improve power consumption; implementing hover functionality for video and camera scenes; and achieving reverse adaptation for continuous scenes, etc. Third, the method proposed in this disclosure can replace the traditional method of adding a whitelist to identify whether the current interface contains camera or video playback. This solution has better versatility and compatibility, can monitor the number of video interfaces and camera preview interfaces, and achieves significant improvements in monitoring accuracy and detection efficiency.

[0160] This disclosure also provides a listening device. See [link to relevant documentation]. Figure 5 , Figure 5This is a block diagram illustrating a listening device according to an exemplary embodiment. The listening device 500 includes:

[0161] The determination module 501 is configured to determine the target cache component to be configured for the target application interface in response to a trigger event for media playback on the display interface; the target application interface is used to output the media data obtained in response to the trigger event for media playback.

[0162] The transmission module 502 is configured to, in response to configuring the target cache component for the target interface, transmit the configuration parameters of the target application interface to the target multimedia control based on the configuration response mechanism between the target cache component and the target multimedia control; wherein, the target cache component is created by the target multimedia control in the system framework and is used to provide a cache channel between the target application interface and the composite display service;

[0163] The listening module 503 is configured to respond to the listening service in the system framework listening to the configuration parameters in the target multimedia control and determine the media data type displayed by the target multimedia control on the display interface.

[0164] In some optional embodiments, the transmission module 502 is further configured to transmit configuration parameters in response to the target application interface calling the first interface function of the target cache component, and to call back the second interface function of the target multimedia control based on a preset callback mechanism; and to return the configuration parameters to the target multimedia control based on the second interface function.

[0165] In some optional embodiments, the transmission module 502 is further configured to transmit configuration parameters in response to the target application interface calling the first interface function, and to call back the second interface function based on the registered object in the target cache component; wherein the registered object is registered in the target cache component when the multimedia control creates the target cache component, and is used to implement the reference to the second interface function.

[0166] In some optional embodiments, the configuration parameters include an application interface identifier; the listening module 503 is further configured to, in response to the listening service listening to the application interface identifier, determine the media data type displayed on the display interface based on the application interface identifier; wherein, the media data type includes video type or camera preview image type.

[0167] In some optional embodiments, the listening module 503 is further configured to, in response to the listening service determining the media data type played by at least one target multimedia control on the display interface, determine the display scene of the display interface; and, based on the display scene, determine an adjustment strategy for the display interface; wherein the adjustment strategy includes at least one of the following: display frame rate adjustment, screen refresh rate adjustment, and display window adjustment.

[0168] In some optional embodiments, the listening module 503 is further configured to, in response to listening to the configuration parameters in the target multimedia control, wherein the configuration parameters include description parameters of media data, determine an adjustment strategy for the display interface based on the display scenario and the description parameters; wherein the description parameters include frame rate and / or resolution.

[0169] In some optional embodiments, the transmission module 502 is further configured to call back a third interface function of the target multimedia control in response to the unbinding of the target application interface from the target cache component; and to clear the configuration parameters in the target multimedia control based on the third interface function.

[0170] Regarding the listening device in the above embodiments, the specific way in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0171] Figure 6 This is a structural block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0172] Reference Figure 6 The electronic device 600 may include one or more of the following components: processing component 602, memory 604, power supply component 606, multimedia component 608, audio component 610, input / output interface 612, sensor component 614, and communication component 616.

[0173] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0174] Memory 604 is configured to store various types of data to support operation on electronic device 600. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, and videos. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0175] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0176] Multimedia component 608 includes a screen that provides an output interface between electronic device 600 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When electronic device 600 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0177] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0178] Input / output interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0179] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 may detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or one of its components, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0180] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0181] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0182] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including executable instructions or a computer program, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0183] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the eavesdropping methods described in the embodiments of this disclosure. For example, the method includes:

[0184] In response to a media playback selection on the display interface, a target cache component is configured for the target application interface. Based on the configuration response mechanism between the target cache component and the target multimedia control, the configuration parameters of the target application interface are transmitted to the target multimedia control. The target cache component is created by the target multimedia control in the system framework and is used to provide a cache channel between the target application interface and the compositing and display service. The target application interface is used to output the media data obtained in response to the media playback selection.

[0185] In response to the listening service in the system framework listening to the configuration parameters in the target multimedia control, the media data type displayed in the corresponding display window of the target multimedia control on the display interface is determined.

[0186] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the eavesdropping methods described in this disclosure; wherein the computer device may be the aforementioned electronic device. Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0187] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A monitoring method, characterized in that, include: In response to media playback trigger events on the display interface, determine the target cache component to be configured for the target application interface; The target application interface is used to output media data obtained in response to the media playback trigger event; In response to configuring the target cache component for the target interface, the configuration parameters of the target application interface are transmitted to the target multimedia control based on the configuration response mechanism between the target cache component and the target multimedia control; wherein, the target cache component is created by the target multimedia control in the system framework and is used to provide a cache channel between the target application interface and the composite display service; In response to the listening service in the system framework detecting the configuration parameters in the target multimedia control, the media data type displayed by the target multimedia control on the display interface is determined.

2. The method according to claim 1, characterized in that, The step of configuring the target cache component for the target interface, based on the configuration response mechanism between the target cache component and the target multimedia control, involves transmitting the configuration parameters of the target application interface to the target multimedia control, including: In response to the target application interface calling the first interface function of the target cache component to transmit the configuration parameters, the second interface function of the target multimedia component is called back based on a preset callback mechanism; Based on the second interface function, the configuration parameters are returned to the target multimedia control.

3. The method according to claim 2, characterized in that, The step of responding to the first interface function of the target cache component by calling the target application interface to transmit the configuration parameters, and based on a preset callback mechanism, calling back the second interface function of the target multimedia component, includes: In response to the target application interface calling the first interface function to transmit the configuration parameters, the second interface function is called back based on the registered object in the target cache component; The registered object is registered in the target cache component when the multimedia control creates the target cache component, and is used to implement a reference to the second interface function.

4. The method according to any one of claims 1 to 3, characterized in that, The configuration parameters include the application interface identifier; The system framework's listening service detects the configuration parameters in the target multimedia control and determines the media data type displayed by the target multimedia control on the display interface, including: In response to the listening service detecting the application interface identifier, the media data type of the target multimedia control displayed on the display interface is determined based on the application interface identifier; The media data types include video types or camera preview image types.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: In response to the monitoring service determining the media data type played by at least one of the target multimedia controls on the display interface, the display scene of the display interface is determined; Based on the display scenario, determine the adjustment strategy for the display interface; The adjustment strategy includes at least one of the following: display frame rate adjustment, screen refresh rate adjustment, and display window adjustment.

6. The method according to claim 5, characterized in that, The step of determining the adjustment strategy for the display interface based on the display scenario includes: In response to the detection of the configuration parameters in the target multimedia control, wherein the configuration parameters include a description parameter of the media data, an adjustment strategy for the display interface is determined based on the display scenario and the description parameter; The description parameters include frame rate and / or resolution.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: In response to the unbinding of the target application interface from the target cache component, the third interface function of the target multimedia control is called back; Based on the third interface function, the configuration parameters in the target multimedia control are cleared.

8. A listening device, characterized in that, include: Determine the module, configured to respond to media playback trigger events on the display interface, and determine the target cache component to be configured for the target application interface; The target application interface is used to output media data obtained in response to the media playback trigger event; The transmission module is configured to, in response to configuring the target cache component for the target interface, transmit the configuration parameters of the target application interface to the target multimedia control based on the configuration response mechanism between the target cache component and the target multimedia control; wherein, the target cache component is created by the target multimedia control in the system framework and is used to provide a cache channel between the target application interface and the composite display service; The listening module is configured to respond to the listening service in the system framework listening for the configuration parameters in the target multimedia control, and to determine the media data type displayed by the target multimedia control on the display interface.

9. The apparatus according to claim 8, characterized in that, The transmission module is further configured to transmit the configuration parameters in response to the target application interface calling the first interface function of the target cache component, and to call back the second interface function of the target multimedia component based on a preset callback mechanism; and to return the configuration parameters to the target multimedia control based on the second interface function.

10. The apparatus according to claim 9, characterized in that, The transmission module is further configured to, in response to the target application interface calling the first interface function to transmit the configuration parameters, call back the second interface function based on the registered object in the target cache component; wherein, the registered object is registered in the target cache component when the multimedia control creates the target cache component, and is used to implement a reference to the second interface function.

11. The apparatus according to any one of claims 8 to 10, characterized in that, The monitoring module is further configured to, in response to the monitoring service, determine the media data type played by at least one of the target multimedia controls on the display interface, determine the display scene of the display interface, and, based on the display scene, determine an adjustment strategy for the display interface; wherein the adjustment strategy includes at least one of the following: display frame rate adjustment, screen refresh rate adjustment, and display window adjustment.

12. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 7.

13. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

14. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images