Mobile terminal and frame rate control method

Abstract

The invention provides a mobile terminal and a frame rate control method, and the method comprises the steps: obtaining a first screen refresh rate and a second screen refresh rate, and obtaining a decoding frame rate threshold value of a display device; determining a frame rate identifier according to the first screen refresh rate, the second screen refresh rate and a decoding frame rate threshold value of the display equipment; when the frame rate identifier is the first frame rate identifier, creating a stable frame rate thread, and outputting a screen projection picture with a stable frame rate through the stable frame rate thread; and encoding the projection screen picture with the stable frame rate, and transmitting the encoded projection screen picture to a display device, so that the display device decodes the projection screen picture and controls a display to output the decoded projection screen picture according to the stable frame rate after receiving the projection screen picture. According to the technical scheme, the mobile terminal can optimize the native screen recording mode based on the native screen recording mode, the screen projection picture with the stable frame rate is output through the added stable frame rate thread, and the problem that the output frame rate is unstable when the native screen recording mode is used for carrying out the mirroring function is solved.

Description

Technical Field

[0001] This application relates to the field of screen mirroring technology, and in particular to a mobile terminal and a method for controlling the frame rate. Background Technology

[0002] Mobile terminals and display devices can perform screen mirroring. Screen mirroring refers to a mobile terminal connecting to a display device via wired or wireless means, projecting the screen displayed on the mobile terminal onto the display device. The display device then displays the content corresponding to the projected screen data sent by the mobile terminal in real time. When performing screen mirroring, users can pre-connect the mobile terminal and display device to the same local area network (LAN) consisting of the same router. During screen mirroring, the mobile terminal records the content displayed on its interface in real time and forwards the projected screen to the display device through the router. The display device then decodes and displays the received projected screen in real time.

[0003] Taking a mobile phone as the mobile terminal and a TV as the display device as an example, in the implementation of the function of mirroring the mobile phone to the TV, the mobile phone can use MediaProjection to record the screen. MediaProjection is an application programming interface used to capture and record the screen content of a device. It allows applications to obtain access permissions to the screen content, thereby realizing functions such as screenshots, screen sharing, and screen recording.

[0004] Because mobile phones do not provide APIs for stable frame rates, the frame rate of the screen content captured by MediaProjection may fluctuate. This leads to an unstable frame rate in the encoded video stream, meaning the output frame rate using this method is unstable. Consequently, when encoding the output video stream, the frame rate of the encoded video stream will also be unstable, affecting the smoothness of decoding and playback on the TV. Therefore, during screen mirroring between mobile terminals and display devices, there is an issue of unstable output frame rates, impacting the user experience. Summary of the Invention

[0005] Some embodiments of this application provide a mobile terminal and a frame rate control method to solve the problem of unstable output frame rate during the mirroring process between the mobile terminal and the display device.

[0006] In a first aspect, some embodiments of this application provide a mobile terminal, including:

[0007] The display unit is configured to display the terminal interface;

[0008] The communication unit is configured to establish a projection connection with the display device;

[0009] The processing unit is configured as follows:

[0010] Obtain the first screen refresh rate and the second screen refresh rate, wherein the first screen refresh rate is the maximum screen refresh rate currently reached by the mobile terminal, and the second screen refresh rate is the maximum screen refresh rate supported by the mobile terminal.

[0011] Obtain the decoding frame rate threshold of the display device, wherein the decoding frame rate threshold is the maximum decoding rate value supported by the display device;

[0012] A frame rate identifier is determined based on the first screen refresh rate, the second screen refresh rate, and the decoding frame rate threshold of the display device; the frame rate identifier includes a first frame rate identifier and a second frame rate identifier, wherein the first frame rate identifier is used to characterize the frame rate output mode of the mobile terminal as a stable frame rate output mode, and the second frame rate identifier is used to characterize the frame rate output mode of the mobile terminal as an unstable frame rate output mode.

[0013] When the frame rate identifier is the first frame rate identifier, a stable frame rate thread is created, and a stable frame rate screen projection screen is output through the stable frame rate thread;

[0014] The screen projection image at the stable frame rate is encoded, and the encoded screen projection image is transmitted to the display device, so that after receiving the screen projection image, the display device decodes the screen projection image and controls the display to output the decoded screen projection image according to the stable frame rate.

[0015] Secondly, some embodiments of this application provide a frame rate control method, which can be applied to the mobile terminal of the first aspect, the mobile terminal including a display unit, a communication unit, and a processing unit, the method including:

[0016] Obtain the first screen refresh rate and the second screen refresh rate, wherein the first screen refresh rate is the maximum screen refresh rate currently reached by the mobile terminal, and the second screen refresh rate is the maximum screen refresh rate supported by the mobile terminal.

[0017] Obtain the decoding frame rate threshold of the display device, wherein the decoding frame rate threshold is the maximum decoding rate value supported by the display device;

[0018] A frame rate identifier is determined based on the first screen refresh rate, the second screen refresh rate, and the decoding frame rate threshold of the display device; the frame rate identifier includes a first frame rate identifier and a second frame rate identifier, wherein the first frame rate identifier is used to characterize the frame rate output mode of the mobile terminal as a stable frame rate output mode, and the second frame rate identifier is used to characterize the frame rate output mode of the mobile terminal as an unstable frame rate output mode.

[0019] When the frame rate identifier is the first frame rate identifier, a stable frame rate thread is created, and a stable frame rate screen projection screen is output through the stable frame rate thread;

[0020] The screen projection image at the stable frame rate is encoded, and the encoded screen projection image is transmitted to the display device, so that after receiving the screen projection image, the display device decodes the screen projection image and controls the display to output the decoded screen projection image according to the stable frame rate.

[0021] As can be seen from the above technical solutions, some embodiments of this application provide a mobile terminal and a frame rate control method. The method includes: obtaining a first screen refresh rate and a second screen refresh rate, wherein the first screen refresh rate is the maximum screen refresh rate currently reached by the mobile terminal, and the second screen refresh rate is the maximum screen refresh rate supported by the mobile terminal; obtaining a decoding frame rate threshold of the display device, wherein the decoding frame rate threshold is the maximum decoding rate value supported by the display device; determining a frame rate identifier based on the first screen refresh rate, the second screen refresh rate, and the decoding frame rate threshold of the display device; the frame rate identifier includes a first frame rate identifier and a second frame rate identifier, wherein the first frame rate identifier is used to characterize that the frame rate output mode of the mobile terminal is a stable frame rate output mode, and the second frame rate identifier is used to characterize that the frame rate output mode of the mobile terminal is an unstable frame rate output mode; when the frame rate identifier is the first frame rate identifier, creating a stable frame rate thread, and outputting a stable frame rate projection screen through the stable frame rate thread; then encoding the stable frame rate projection screen, and transmitting the encoded projection screen to the display device, so that after receiving the projection screen, the display device decodes the projection screen and controls the display to output the decoded projection screen according to the stable frame rate. The method can optimize the native screen recording method by adding a stable frame rate thread to output a stable frame rate screen projection, thus solving the problem of unstable output frame rate when using the native screen recording method for mirroring. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in some embodiments of this application or in the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0024] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0025] Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application;

[0026] Figure 4This is a schematic flowchart of a mobile terminal frame rate control method provided in some embodiments of this application;

[0027] Figure 5 This is a comparative diagram of the native screen recording method and the optimized screen recording method of mobile terminals provided in some embodiments of this application;

[0028] Figure 6 This is a schematic diagram illustrating the relationship between surface data, surface texture data, and texture data provided in some embodiments of this application;

[0029] Figure 7 A timing diagram illustrating the stable frame rate output of a mobile terminal provided in some embodiments of this application;

[0030] Figure 8 This application provides schematic diagrams illustrating mobile terminal refresh rate interface settings in some embodiments.

[0031] Figure 9 A flowchart illustrating the process of creating and initializing a media asset management class for screen recording for a mobile terminal provided in some embodiments of this application;

[0032] Figure 10 A flowchart illustrating the process of creating a stable frame rate thread and outputting a screen projection image with a stable frame rate through the stable frame rate thread, as provided in some embodiments of this application;

[0033] Figure 11 This is a schematic diagram illustrating the process of a mobile terminal generating target surface data according to some embodiments of this application. Detailed Implementation

[0034] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0035] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0036] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0037] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0038] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0039] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0040] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, a user can operate the display device 200 via touch operation, a mobile terminal 300, and a control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc.

[0041] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0042] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application that controls the display device 200.

[0043] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0044] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0045] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0046] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0047] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0048] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0049] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0050] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0051] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0052] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0053] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0054] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0055] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0056] To enable user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running an application. The operating system also allows users to interact with the display device 200.

[0057] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0058] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0059] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0060] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0061] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0062] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0063] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0064] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The drivers included in the kernel layer can be at least one of the following: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0065] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0066] The mobile terminal 300 and the display device 200 can establish a screen mirroring connection, allowing the display device 200 to synchronously play the screen content displayed by the mobile terminal 300. In this embodiment, when the mobile terminal 300 and the display device 200 establish a screen mirroring connection, a specific screen mirroring connection protocol can be used. For example, it can be through connection protocols such as Miracast, AirPlay, Digital Living Network Alliance (DLNA), Universal Serial Bus (USB), or other self-developed network protocols; this application does not specifically limit the specific network protocol used.

[0067] When the mobile terminal 300 and the display device 200 perform a screen mirroring connection, the mobile terminal 300 acts as the data source providing the screen mirroring data, and the display device 200 acts as the receiving end for receiving and displaying the screen mirroring data. Taking the mobile terminal 300 as a mobile phone device A and the display device 200 as a television device B as an example, in some embodiments, the screen mirroring connection between the mobile phone device A and the television device B can be achieved in the following manner.

[0068] For example, mobile device A acts as the data source, and TV device B acts as the screen projection display. After mobile device A and TV device B successfully communicate and connect via USB connection protocol or a self-developed protocol, mobile device A begins recording its screen and generating raw video frames. For example, the raw video frames can be h264 or h265 data frames. After generating the video frames, mobile device A can send them to TV device B via network or USB transmission protocol. After receiving the video frames, TV device B can decode the raw video frame data using a video decoder to form projection data, and control the display 260 to display the projection data. For example, in some embodiments, the projection data can be displayed using screen display controls such as SurfaceView or TextureView.

[0069] For example, mobile terminal 300 and display device 200 can perform screen mirroring. Screen mirroring refers to mobile terminal 300 connecting to display device 200 via wired or wireless means, so that the screen displayed on mobile terminal 300 is projected onto display device 200, and display device 200 displays the content corresponding to the screen mirroring data sent by mobile terminal 300 in real time. When performing screen mirroring between mobile terminal 300 and display device 200, users can pre-connect mobile terminal 300 and display device 200 within the same local area network consisting of the same router. During screen mirroring, mobile terminal 300 records the content displayed on its interface in real time and forwards the projected screen to display device 200 through the router. Display device 200 decodes and displays the projected screen in real time after receiving it.

[0070] In some embodiments, taking a mobile terminal 300 as a mobile phone and a display device 200 as a television as an example, in implementing the function of mirroring the mobile phone to the television, the mobile phone can use MediaProjection for screen recording. MediaProjection is an application programming interface (API) used to capture and record the screen content of a device. It allows applications to obtain access permissions to the screen content, thereby enabling functions such as screenshots, screen sharing, and screen recording. MediaProjection can capture the screen content of the current device. By creating a virtual display, developers can specify which surface the screen content is rendered on, thereby controlling the content and attributes of the captured screen. The captured screen content needs to be encoded into a video stream for transmission over a network.

[0071] Because mobile phones do not provide APIs for stable frame rates, the frame rate of the screen content captured by MediaProjection may fluctuate. This leads to an unstable frame rate in the encoded video stream, meaning the output frame rate using this method is unstable. Therefore, when encoding the output video stream, such as using MediaCodec, if the input frame rate is unstable, the encoded video stream's frame rate will also be unstable, affecting the smoothness of decoding and playback on the TV. Furthermore, if the mobile phone's screen refresh rate is higher than the TV's decoding capability, it may cause the TV to fail to decode correctly or experience stuttering, screen tearing, or other problems. Therefore, during screen mirroring between mobile terminal 300 and display device 200, there is an issue of unstable output frame rate, impacting the user experience.

[0072] To address the issue of unstable frame rates during screen mirroring between a mobile terminal 300 and a display device 200, certain embodiments of this application provide a mobile terminal 300 comprising a display unit 310, a communication unit 320, and a processing unit 330. The display unit 310 is configured to display the terminal interface, the communication unit 320 is configured to establish a communication connection with the display device 200, and the processing unit 330 executes a frame rate control method by running an application. The mobile terminal 300 optimizes the native screen recording method by adding a stable frame rate thread to output a stable frame rate screen mirroring image, thus resolving the issue of unstable output frame rates when using the native screen recording method for screen mirroring.

[0073] To facilitate understanding of the technical solutions in some embodiments of this application, the steps are described in detail below with reference to some specific embodiments and accompanying drawings. Figure 4 This is a schematic flowchart of a mobile terminal frame rate control method provided in some embodiments of this application, such as... Figure 4As shown, in some embodiments, when the mobile terminal 300 executes the frame rate control method, it may include the following steps S1-S5, the specific contents of which are as follows:

[0074] Step S1: The mobile terminal 300 obtains the first screen refresh rate and the second screen refresh rate.

[0075] In some embodiments, the native screen recording method of the mobile terminal 300 is first described. Taking the mobile terminal 300 as a mobile phone and the display device 200 as a television as an example, the mirroring process between the two is described, and how the mobile terminal 300 optimizes the native screen recording method to obtain the optimized screen recording method in this embodiment.

[0076] In some embodiments, when mirroring a mobile phone to a TV, the following native screen recording method can be used. The mobile phone can record screen data using MediaProjection and MediaCodec, encoding it into a fixed-format data stream frame by frame. Then, the data stream frame by frame can be transmitted to the TV via a network transmission protocol. The TV uses a player to decode the data frame by frame, and plays and renders it on the TV. The screen on the TV always matches the current screen on the mobile phone, thus achieving the function of mirroring the mobile phone to the TV.

[0077] However, because the frame rate of the screen encoding output data stream recorded by MediaProjection and MediaCodec is not fixed, the frame rate of a mobile phone dynamically changes under different screen conditions in real-world scenarios. For example, when the phone is on the home screen, the screen remains relatively still, and the interval between each frame output is long, resulting in a relatively low frame rate. When playing videos or games, the screen fluctuates more, the interval between each frame output becomes shorter, and the frame rate relatively increases. Thus, when this dynamically changing frame rate data stream is transmitted to the TV via network transmission protocols, the TV's decoding and playback of the mirrored screen data stream from the mobile phone also results in an unstable frame rate output; that is, the frame rate output by the TV is also unstable.

[0078] Based on this, this application optimizes the native screen recording method to solve the problem of unstable frame rate when mirroring from a mobile phone to a TV. It should be noted that the optimization method in this application is not only applicable to the scenario of mirroring from a mobile phone to a TV. This is only used as an example for illustration. It can also be applied to any scenario in other native screen recording methods that requires a stable frame rate. This application does not make any specific limitations on this.

[0079] Figure 5 This is a comparative illustration of the native screen recording method and the optimized screen recording method for mobile terminals provided in some embodiments of this application, such as... Figure 5As shown, in some embodiments, the mobile terminal 300 can add a stable frame rate to the native screen recording mode to achieve an optimization process from the native screen recording mode to a screen recording mode with a stable frame rate. The following describes this in conjunction with... Figure 5 This document provides a detailed explanation of the native screen recording method and its optimization process.

[0080] In some embodiments, in the native screen recording method of the mobile terminal 300, an instance of the class used for screen capture can first be obtained. For example, an instance of MediaProjectionManager can be obtained through getSystemService. Then, the user's screen recording permission authorization can be obtained. After the user agrees to screen recording, the mobile terminal 300, having obtained the user's screen recording permission, can create an instance of MediaProjection and initialize MediaCodec. Then, combined with... Figure 5 A VirtualDisplay object instance can be created from a pre-existing MediaProjection object. The Surface data object in the virtual screen can be created by a MediaCodec object and then registered with the VirtualDisplay object. Frame data during screen recording is captured by MediaProjection and sent to the Surface data. Following this method, the mobile terminal 300 then activates the encoder to perform loop encoding, outputting frame-by-frame data streams.

[0081] The above describes the native screen recording method. The image displayed on the VirtualDisplay is generated as a series of data streams output by the MediaProjection object. Each frame of this data stream corresponds to the input of the MediaCodec Surface object. Combined with... Figure 5 In other words, in native screen recording, the output of the surface data Surface in the virtual screen is the input surface, or InputSurface, in the screen recording instance. That is, the Surface in the virtual screen and the Surface in the screen recording instance are bound together. The encoder can then encode frame by frame of the data stream. Therefore, it is evident that the output speed of each frame of the data stream cannot be controlled during the output process, resulting in an unstable output frame rate in native screen recording.

[0082] Therefore, to stabilize the output frame rate, in optimizing the screen recording method, a new thread for stabilizing the frame rate can be added based on the surface data (Surface), such as adding an ImageRender thread. This eliminates the binding between the Surface in the virtual screen and the Surface in the screen recording instance. Figure 5Instead, an intermediate interface layer, such as an EGL instance, is created in the stable frame rate thread. The Surface in the EGL instance is bound to the Surface in the screen recording instance. The stable frame rate thread adjusts the rate of the stable frame rate, thereby achieving the goal of outputting a stable frame rate.

[0083] Specifically, Figure 6 This is a schematic diagram illustrating the relationship between surface data, surface texture data, and texture data provided in some embodiments of this application, such as... Figure 6 As shown, the surface data Surface is the user-visible interface and is the largest unit; the surface texture data SurfaceTexture is the next level unit that constitutes Surface; and the texture data Texture is the smallest unit that constitutes SurfaceTexture. In some embodiments, combined with Figure 5 The surface data Surface in the virtual screen object can be created as follows: first, create a SurfaceTexture using a Texture, and then create the Surface in the virtual screen using the SurfaceTexture.

[0084] Continue to combine Figure 5 Once the surface data (Surface) in the virtual screen object is created, based on the native screen recording method—that is, based on the existing surface data (Surface) and the existing screen recording instance—an optimized screen recording method can be executed by creating a stable frame rate thread. This thread outputs a stable frame rate screen projection. The following section will provide a detailed explanation of the application of the entire optimized screen recording method, along with specific implementation steps.

[0085] Figure 7 This is a timing diagram illustrating the stable frame rate output of a mobile terminal according to some embodiments of this application, such as... Figure 7 As shown, in some embodiments, to facilitate subsequent determination of the frame rate adaptation strategy with the display device 200, the mobile terminal 300 can obtain a first screen refresh rate and a second screen refresh rate. The first screen refresh rate is the maximum screen refresh rate currently achieved by the mobile terminal, i.e., the currently actually reached maximum refresh rate. The second screen refresh rate is the maximum screen refresh rate supported by the mobile terminal, i.e., the theoretically supported maximum refresh rate. It is understandable that when the phone is in home mode, the screen changes are minimal and the second screen refresh rate is not reached, while when the phone is in game mode, it may reach the second screen refresh rate.

[0086] In real-world usage scenarios, different mobile phones have different refresh rates. Some phones allow you to set the refresh rate, while others do not. For example, Figure 8 This application provides schematic diagrams illustrating mobile terminal refresh rate interface settings in some embodiments, such as... Figure 8As shown, in some mobile phones that allow setting the refresh rate, users can set the phone's refresh rate through the refresh rate settings page, and users can set it according to their actual needs. The phone's refresh rate can be obtained in various ways. For example, it can be viewed through the settings page or through third-party applications, etc., and this application does not specifically limit this. After step S1 is completed, step S2 can be included.

[0087] Step S2: The mobile terminal 300 obtains the decoding frame rate threshold of the display device.

[0088] To ensure that the screen does not stutter or drop frames due to insufficient decoding capability of the TV during mirror transmission, in some embodiments, the mobile terminal 300 can obtain the decoding frame rate threshold of the display device 200, wherein the decoding frame rate threshold is the maximum decoding rate value supported by the display device 200.

[0089] In some embodiments, when the mobile terminal 300 obtains the decoding frame rate threshold of the display device, it can first send a decoding capability detection request to the display device 200. The decoding capability detection request is used to request the decoding frame rate threshold of the display device 200, and then receive the decoding frame rate threshold returned by the display device 200 according to the decoding capability detection request.

[0090] For example, after the mobile phone sends a decoding capability detection request to the TV, the TV can calculate the decoding frame rate threshold based on its hardware and software configuration. This threshold determines the upper limit of the frame rate at which the TV can successfully decode and display video content, and the TV returns this threshold to the mobile phone. Once the mobile phone obtains the decoding frame rate threshold, it can determine the TV's maximum decoding capability, providing data for subsequent stable frame rate output. After step S2 is completed, step S3 can be included.

[0091] Step S3: The mobile terminal 300 determines the frame rate identifier based on the first screen refresh rate, the second screen refresh rate, and the decoding frame rate threshold of the display device 200.

[0092] In multi-screen interaction or screen mirroring scenarios, mobile terminal 300 sends screen content to display device 200 for display. To ensure smoothness and compatibility during screen mirroring, the performance capabilities of both mobile device 300 and display device 200 need to be considered. Screen refresh rate refers to the number of times the screen can refresh per second, measured in Hertz (Hz). The decoding frame rate threshold refers to the maximum frame rate that display device 200 can stably decode and display. During screen mirroring, if the screen refresh rate of mobile device 300 exceeds the decoding frame rate threshold of display device 200, the display device 200 may be unable to decode frame rates exceeding its threshold in a timely manner, resulting in screen stuttering or latency.

[0093] To avoid the above situation, in some embodiments, if the first screen refresh rate is greater than the decoding frame rate threshold, or the second screen refresh rate is greater than the decoding frame rate threshold, the mobile terminal 300 can generate a negotiation instruction and send the negotiation instruction to the display device 200. The negotiation instruction is used to request the display device 200 the frame rate value that should be set when the mobile device 300 and the display device 200 project their screens, and then receive the negotiated frame rate value fed back by the display device 200 according to the negotiation instruction, wherein the negotiated frame rate value is less than or equal to the decoding frame rate threshold.

[0094] For example, if the refresh rate of the first screen is greater than the decoding frame rate threshold, or the refresh rate of the second screen is greater than the decoding frame rate threshold, a negotiation instruction is generated. The purpose of this negotiation instruction is to request the display device 200 to provide a suitable frame rate value to ensure smoothness during screen mirroring. The mobile terminal 300 can send the negotiation instruction to the display device 200, requesting the display device 200 to provide a negotiated frame rate value based on its decoding capability. This value should be less than or equal to the decoding frame rate threshold. The display device 200 calculates and returns a negotiated frame rate value based on the negotiation instruction. The negotiated frame rate value can be a frame rate that the display device 200 recommends using during screen mirroring to ensure that it does not exceed its decoding capability. In this way, the mobile terminal 300 can ensure that the decoding capability of the display device 200 is not exceeded during screen mirroring, solving the problem of the output frame rate exceeding the decoding capability of the receiving television, thereby avoiding screen stuttering or delay and improving the user's viewing experience.

[0095] To select the most suitable frame rate output mode based on the mobile phone's refresh rate and the TV's decoding capability, the mobile terminal 300 can determine a frame rate identifier based on the first screen refresh rate, the second screen refresh rate, and the decoding frame rate threshold of the display device 200. In some embodiments, the frame rate identifier may include a first frame rate identifier and a second frame rate identifier. The first frame rate identifier indicates that the mobile terminal 300's frame rate output mode is a stable frame rate output mode, meaning that the mobile terminal 300 will output at a stable frame rate. The second frame rate identifier indicates that the mobile terminal 300's frame rate output mode is an unstable frame rate output mode, meaning that the second frame rate identifier may indicate that the output frame rate will fluctuate according to actual conditions, such as inheriting the frame rate output mode from the native screen recording method.

[0096] For example, in actual mirroring scenarios, the stable frame rate function is not enabled by default for every mirroring, as this would be complex and time-consuming. Therefore, the decision to enable the stable frame rate output mode can be based on the decoding capabilities of the receiving TV and the refresh rate of the mobile phone screen. In other words, in this embodiment, the decision to enable the stable frame rate output mode can be made based on whether the current refresh rate of the mobile phone is higher than the maximum frame rate threshold supported by the TV's decoding capability.

[0097] To determine the frame rate identifier, in some embodiments, the mobile terminal 300 may perform the following process. Continuing in conjunction with... Figure 7 If the refresh rate of the first screen is greater than the decoding frame rate threshold, or the refresh rate of the second screen is greater than the decoding frame rate threshold, the mobile terminal 300 generates a first frame rate identifier; if the refresh rate of the first screen is less than or equal to the decoding frame rate threshold, and the refresh rate of the second screen is less than or equal to the decoding frame rate threshold, the mobile terminal 300 generates a second frame rate identifier.

[0098] For example, taking the mobile phone's current maximum refresh rate as ScreenRefreshRate, the mobile phone's theoretically supported maximum refresh rate as MaxinumScreenRefreshRate, and the TV's supported maximum decoding rate as THRESHOLD, by comparison, if the relationship between the mobile phone's first and second screen refresh rates and the maximum decoding rate is ScreenRefreshRate > THRESHOLD, or MaxinumScreenRefreshRate > THRESHOLD, then the mobile phone can generate a first frame rate flag that needs to negotiate a stable frame rate value with the TV, i.e., negotiate the frame rate value, and activate the stable frame rate function. Otherwise, the mobile phone sets a second frame rate flag that does not activate the stable frame rate function. In this way, the mobile terminal 300 can determine the frame rate output mode based on the frame rate flag, avoiding screen stuttering or delays on the TV. After step S3 is completed, step S4 can be executed.

[0099] Step S4: When the frame rate identifier is the first frame rate identifier, the mobile terminal 300 creates a stable frame rate thread and outputs a stable frame rate projection screen through the stable frame rate thread.

[0100] When the frame rate identifier is the first frame rate identifier, that is, when the mobile terminal 300 determines to use the stable frame rate output mode, the mobile terminal 300 can create a new thread to be specifically responsible for generating and outputting a stable frame rate, so as to ensure that the data stream can be stably transmitted to the display device 200, solve the problem of screen discontinuity caused by unstable frame rate, and improve the user's visual experience.

[0101] After determining the frame rate identifier, the mobile terminal 300 can create and initialize a media asset management class for screen recording, in preparation for subsequent stable frame rate output. Figure 9 A flowchart illustrating the process of creating and initializing a media asset management class for screen recording on a mobile terminal, as provided in some embodiments of this application, is shown below. Figure 9As shown, in some embodiments, the mobile terminal 300 can first create a media asset management class for screen recording, obtain user screen recording permissions based on the media asset management class, and the user screen recording permissions include agreeing to screen recording permissions and refusing screen recording permissions; when the user screen recording permission is refusing screen recording permissions, the screen projection screen is stopped from being sent to the display device 200; when the user screen recording permission is agreeing to screen recording permissions, the media asset management class is initialized, and a screen recording instance is initialized based on the media asset management class, and the initialized screen recording instance is passed to the media asset management class.

[0102] For example, continue to combine Figure 7 You can create a MediaProjectionManager for screen recording, then obtain screen recording permissions from the user. If the user does not agree to screen recording, the current mirroring ends. If the user agrees to screen recording, the MediaProjectionManager is initialized, and a screen recording instance, such as Screen Record, is initialized and passed to the MediaProjectionManager.

[0103] After the media asset management class is created and initialized, the mobile terminal 300 can create an input surface and an encoder in the screen recording instance, and initialize the encoder to encode the screen projection screen when there is a screen projection screen in the input surface. The input surface is used to input the screen projection screen; the encoder is used to encode the screen projection screen.

[0104] For example, when screen recording is performed on a mobile terminal 300, a series of components are required to work together to capture, encode, process, and ultimately save or stream the screen content. For example, combined with Figure 5 In the Screen Record instance, you can create an input surface and an encoder. The encoder can be selected from options such as Advanced Video Coding (AVC) or High Efficiency Video Coding (HEVC). The input surface is responsible for receiving data from the screen. It can be created by creating a SurfaceView or a similar view, and it can directly interact with the screen buffer and capture activity on the screen. As the data source, the input surface passes the captured pixel data to the next stage, the encoder. The encoder is responsible for converting the screen feed from the input surface into a compressed format to reduce the video file size and optimize storage or transmission efficiency.

[0105] In some embodiments, before the screen recording instance begins, the encoder can be initialized, including configuring encoder parameters such as resolution, frame rate, bit rate, etc., and preparing the resources required by the encoder. The initialization process ensures that the encoder can begin encoding immediately when a projected image is present on the input surface, thus seamlessly capturing screen content. Once the encoder initialization is complete and the input surface begins providing a projected image, the encoder begins operation. The projected image from the input surface can be received and then encoded into video frames. These frames can then be saved to a file or sent over a network, such as to display device 200.

[0106] Continue to combine Figure 7 After the screen recording instance is initialized, the mobile terminal 300 can determine whether a stable frame rate is needed by using the frame rate identifier. If a stable frame rate is not needed, i.e., when the frame rate identifier is the second frame rate identifier, the mobile terminal 300 can fill the input surface with surface data, start the encoder thread, and obtain the image encoding from the input surface based on the encoder thread, and transmit the encoded image to the display device 200.

[0107] For example, in combination Figure 5 When a stable frame rate is not required, the mobile terminal 300 can continue with the native screen recording method. First, a VirtualDisplay object instance can be created, and then the surface data Surface is passed to the InputSurface in the screen recording instance. Next, the encoder thread can be started, which can perform encoding on the InputSurface. For example, it can read image encoding, such as image frames, from the input surface, and the encoder can encode the obtained image encoding. Then, the encoded image data is sent to the TV. These image frames are the media content that the user finally sees.

[0108] When a stable frame rate is needed, the mobile terminal can create a stable frame rate thread and output a screen projection with a stable frame rate through the stable frame rate thread. Figure 10 This application provides flowcharts illustrating the process of creating a stable frame rate thread and outputting a stable frame rate projection screen through the stable frame rate thread in some embodiments, such as... Figure 10 As shown, in some embodiments, the mobile terminal 300 can first create a stable frame rate thread; create an intermediate interface layer EGL instance based on the stable frame rate thread; create target texture data based on the intermediate interface layer EGL instance; wherein, the target texture data is initially an empty texture; acquire the surface data of the virtual screen, and extract the texture data from the surface data; fill the target texture data with the texture data; set the waiting time of the filled target texture data according to a preset value and a negotiated frame rate value, and then output the screen projection image with a stable frame rate based on the stable frame rate thread according to the waiting time.

[0109] For example, when creating a stable frame rate thread, the mobile terminal 300 can create it by inheriting the Thread class. For instance, a Thread subclass can be defined, and its run() method can be overridden, containing the tasks the thread needs to execute. Creating an instance of this subclass creates a thread object, and calling the start() method of this thread object starts the thread. Other methods can also be used, and this application does not specifically limit them. After the stable frame rate thread is created, the mobile terminal 300 can create an intermediate interface layer, such as an Embedded-System Graphics Library instance, i.e., an EGL instance, within the stable frame rate thread. EGL is a cross-platform graphics interface for embedded systems, providing a way for applications to interact with the underlying windowing system and graphics processor.

[0110] Continue to combine Figure 5 In some embodiments, using an EGL instance, the mobile terminal 300 can create a texture data object, which is essentially an image data storage object that can be rendered onto the screen. Initially, this texture data is an empty texture, meaning it does not contain any actual image data. The texture data can be extracted from the virtual screen's surface data (Surface). This texture contains the visual elements currently displayed on the screen, including the actual media content displayed. Then, the texture extracted from the virtual screen object can be used to populate the previously created empty target texture data. Thus, the texture data in the stable frame rate thread contains the actual media content that needs to be displayed on the screen.

[0111] To ensure that each frame of media data has the same time interval before display, the mobile terminal 300 can set a waiting time for the filled texture data after it is filled. Specifically, the waiting time can be calculated based on a preset value and a negotiated frame rate value. For example, if the preset value is set to 1000 milliseconds and the negotiated frame rate value is 40, then the waiting time is 1000 / 40 = 25. That is, if 40 frames are output within 1 second, then one frame is refreshed every 25ms. After the waiting time is determined, the stable frame rate thread can control the output of texture data according to the set waiting time, thereby ensuring that the media data is played at a stable frame rate. In this way, by controlling the display time of each frame, the smoothness and stability of the media content can be effectively guaranteed, avoiding screen jitter or slippage, and solving problems such as discontinuous images caused by unstable frame rates.

[0112] It should be noted that the texture data created through the EGL instance is small-unit texture data, while the final display is large-unit surface data. Therefore, the mobile terminal 300 also needs to draw the large-unit surface data based on the small-unit texture data. Figure 11A schematic diagram illustrating the process of a mobile terminal generating target surface data provided in some embodiments of this application, such as... Figure 11 As shown, in some embodiments, the mobile terminal 300 can initialize an intermediate interface layer EGL instance based on a stable frame rate thread; pass context information into the intermediate interface layer EGL instance, wherein the context information is used to record information during the intermediate interface layer EGL rendering process; based on the intermediate interface layer EGL instance, draw the target texture data onto the surface according to the waiting time to generate target surface data; and then input the target surface data into the input surface in the screen recording instance.

[0113] For example, when initializing the EGL instance, the mobile terminal 300 can set various attributes to ensure the rendering environment meets the application's requirements, such as setting the color buffer format and depth buffer size. Simultaneously, it can select the EGL rendering interface. EGL supports multiple rendering interfaces, such as OpenGL ES and OpenVG, allowing selection of a suitable interface based on actual needs. Then, context information can be passed into the intermediate interface layer EGL instance. This information records the state during the EGL rendering process, such as the currently bound texture and rendering target. Context information helps synchronize resource states between different rendering processes or threads, avoiding resource access conflicts. By maintaining context information, EGL can reduce unnecessary resource creation and destruction, thereby optimizing rendering performance. Afterward, based on the intermediate interface layer EGL, the target texture data can be drawn onto the surface according to a preset waiting time. This waiting time ensures consistent rendering intervals for each frame, achieving a stable frame rate. After rendering is complete, the result is stored in the target surface data, which is then input into the input surface of the screen recording instance as the data source for screen recording.

[0114] In other words, in this embodiment, based on the native screen recording method of the mobile terminal 300, the original surface data Surface is input into the input surface of the screen recording instance. This is optimized by adding a stable frame rate thread. In the stable frame rate thread, target texture data is created using an EGL instance. After filling the target texture data in the stable frame rate thread with the texture data from the original surface data, the filled target texture data is output at fixed time intervals, i.e., at a fixed rate, thereby achieving a stable frame rate. Small units of target texture data are then drawn onto the surface, and the redrawn target texture data is input into the input surface of the screen recording instance. This modifies the binding relationship of the input surface data in the native screen recording method; therefore, it is an optimization based on the native screen recording method. In this way, a stable frame rate data stream can be output through the stable frame rate thread, solving the problem of unstable output frame rate during screen mirroring between the mobile terminal 300 and the display device 200. After step S4 is completed, step S5 can be included.

[0115] Step S5: The mobile terminal 300 encodes the projection screen with a stable frame rate and transmits the encoded projection screen to the display device, so that after receiving the projection screen, the display device decodes the projection screen and controls the display to output the decoded projection screen according to the stable frame rate.

[0116] In order to enable the display device 200 to output a stable frame rate, in some embodiments, the mobile terminal 300 can encode the projected screen with a stable frame rate and transmit the encoded projected screen to the display device 200, so that after receiving the projected screen, the display device 200 can decode the projected screen and control the display 260 to output the decoded projected screen according to the stable frame rate.

[0117] For example, after a stable frame rate is generated by the stable frame rate thread, the mobile device can encode the stable frame rate so that it can be sent to the TV via a transmission interface such as HDMI or USB-C. Upon receiving these encoded frames, the TV will decode them and display the decoded frames on the screen. Because the frame rate is stable, the user will see smooth media content without frame drops, thus resolving the issue of unstable output frame rates during screen mirroring between the mobile terminal 300 and the display device 200.

[0118] To save memory resources, in some embodiments, the mobile terminal 300 may turn off the encoder and input surface and release the resources they occupy when the screen recording ends, in order to avoid memory leaks and other potential performance problems and save memory resources.

[0119] As can be seen from the above technical solutions, the above embodiments provide a mobile terminal 300, which can obtain a first screen refresh rate and a second screen refresh rate. The first screen refresh rate is the maximum screen refresh rate currently reached by the mobile terminal, and the second screen refresh rate is the maximum screen refresh rate supported by the mobile terminal. It can also obtain a decoding frame rate threshold for the display device, which is the maximum decoding rate value supported by the display device. Furthermore, it can determine a frame rate identifier based on the first screen refresh rate, the second screen refresh rate, and the decoding frame rate threshold of the display device. The frame rate identifier includes a first frame rate identifier and a second frame rate identifier. The first frame rate identifier is used to characterize the frame rate output mode of the mobile terminal as a stable frame rate output mode, and the second frame rate identifier is used to characterize the frame rate output mode of the mobile terminal as an unstable frame rate output mode. When the frame rate identifier is the first frame rate identifier, a stable frame rate thread is created, and a stable frame rate projection screen is output through the stable frame rate thread. Then, the stable frame rate projection screen is encoded, and the encoded projection screen is transmitted to the display device, so that after receiving the projection screen, the display device 200 decodes the projection screen and controls the display 260 to output the decoded projection screen according to the stable frame rate. The mobile terminal 300 can optimize the native screen recording method by adding a stable frame rate thread to output a stable frame rate screen projection, thus solving the problem of unstable output frame rate when using the native screen recording method for mirroring.

[0120] Based on the mobile terminal 300 described above, some embodiments of this application also provide a frame rate control method, which can be applied to the mobile terminal 300 in the above embodiments. In some embodiments, the method may include the following:

[0121] Obtain the first screen refresh rate and the second screen refresh rate, wherein the first screen refresh rate is the maximum screen refresh rate currently reached by the mobile terminal, and the second screen refresh rate is the maximum screen refresh rate supported by the mobile terminal.

[0122] Obtain the decoding frame rate threshold of the display device, wherein the decoding frame rate threshold is the maximum decoding rate value supported by the display device;

[0123] A frame rate identifier is determined based on the first screen refresh rate, the second screen refresh rate, and the decoding frame rate threshold of the display device; the frame rate identifier includes a first frame rate identifier and a second frame rate identifier, wherein the first frame rate identifier is used to characterize the frame rate output mode of the mobile terminal as a stable frame rate output mode, and the second frame rate identifier is used to characterize the frame rate output mode of the mobile terminal as an unstable frame rate output mode.

[0124] When the frame rate identifier is the first frame rate identifier, a stable frame rate thread is created, and a stable frame rate screen projection screen is output through the stable frame rate thread;

[0125] The screen projection image at the stable frame rate is encoded, and the encoded screen projection image is transmitted to the display device, so that after receiving the screen projection image, the display device decodes the screen projection image and controls the display to output the decoded screen projection image at the stable frame rate.

[0126] As can be seen from the above technical solutions, the above embodiments provide a frame rate control method. The method can optimize the native screen recording method based on the native screen recording method, and output a stable frame rate screen projection screen by adding a stable frame rate thread, thereby solving the problem of unstable output frame rate when using the native screen recording method for mirroring function.

[0127] The same or similar parts between the various embodiments in this specification can be referred to each other, and will not be repeated here.

[0128] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0130] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A mobile terminal, characterized by comprising: The method comprises the following steps: displaying a terminal interface by a display unit; establishing a screen projection connection with a display device by a communication unit; obtaining a first screen refresh rate and a second screen refresh rate, wherein the first screen refresh rate is a maximum screen refresh rate currently reached by the mobile terminal, and the second screen refresh rate is a maximum screen refresh rate supported by the mobile terminal; obtaining a decoding frame rate threshold of the display device, wherein the decoding frame rate threshold is a maximum decoding rate value supported by the display device; determining a frame rate identifier according to the first screen refresh rate, the second screen refresh rate and the decoding frame rate threshold of the display device, wherein the frame rate identifier comprises a first frame rate identifier and a second frame rate identifier, the first frame rate identifier is used to represent that a frame rate output mode of the mobile terminal is a stable frame rate output mode, and the second frame rate identifier is used to represent that the frame rate output mode of the mobile terminal is an unstable frame rate output mode; when the frame rate identifier is the first frame rate identifier, creating a stable frame rate thread, and outputting a screen projection picture with a stable frame rate through the stable frame rate thread; performing encoding on the screen projection picture with the stable frame rate, and transmitting the encoded screen projection picture to the display device, so that the display device decodes the screen projection picture and controls a display to output the decoded screen projection picture according to the stable frame rate after receiving the screen projection picture. The processing unit performing the step of obtaining the decoding frame rate threshold of the display device is further configured to:

2. The mobile terminal of claim 1, wherein, sending a decoding capability detection request to the display device, wherein the decoding capability detection request is used to request the decoding frame rate threshold of the display device; receiving the decoding frame rate threshold returned by the display device according to the decoding capability detection request. The processing unit is further configured to:

3. The mobile terminal of claim 1, wherein, if the first screen refresh rate is greater than the decoding frame rate threshold, or the second screen refresh rate is greater than the decoding frame rate threshold, generating a negotiation instruction, and sending the negotiation instruction to the display device, wherein the negotiation instruction is used to request a frame rate value that should be set when the mobile device projects the screen to the display device; receiving a negotiation frame rate value fed back by the display device according to the negotiation instruction, wherein the negotiation frame rate value is less than or equal to the decoding frame rate threshold. The processing unit performing the step of determining the frame rate identifier according to the first screen refresh rate, the second screen refresh rate and the decoding frame rate threshold of the display device is further configured to:

4. The mobile terminal of claim 3, wherein, if the first screen refresh rate is greater than the decoding frame rate threshold, or the second screen refresh rate is greater than the decoding frame rate threshold, generating the first frame rate identifier; if the first screen refresh rate is less than or equal to the decoding frame rate threshold, and the second screen refresh rate is less than or equal to the decoding frame rate threshold, generating the second frame rate identifier. The processing unit is further configured to:

5. The mobile terminal of claim 3, wherein, creating a media management class for screen recording; obtaining a user screen recording permission based on the media management class, wherein the user screen recording permission comprises an agree screen recording permission and a refuse screen recording permission; when the user screen recording permission is the refuse screen recording permission, stopping sending the screen projection picture to the display device. ​ When the user screen recording permission is the consent screen recording permission, the media management class is initialized, a screen recording instance is initialized based on the media management class, and the initialized screen recording instance is transmitted to the media management class.

6. The mobile terminal of claim 5, wherein, The processing unit is further configured to: create an input surface and an encoder in the screen recording instance; the input surface is used to input a projection picture; the encoder is used to encode the projection picture; initialize the encoder to perform encoding on the projection picture when the projection picture exists in the input surface.

7. The mobile terminal of claim 6, wherein, The processing unit executes a stable frame rate thread and outputs a projection picture with a stable frame rate through the stable frame rate thread, and is further configured to: create a stable frame rate thread; create an intermediate interface layer EGL instance based on the stable frame rate thread; create target texture data based on the intermediate interface layer EGL instance; the initial state of the target texture data is empty texture; obtain surface data of a virtual screen, and extract texture data in the surface data; fill the texture data into the target texture data; set a waiting time of the filled target texture data according to a preset value and the negotiated frame rate value; output a projection picture with a stable frame rate based on the stable frame rate thread according to the waiting time.

8. The mobile terminal of claim 7, wherein, The processing unit is further configured to: initialize the intermediate interface layer EGL instance based on the stable frame rate thread; transmit context information in the intermediate interface layer EGL instance, the context information is used to record information in the intermediate interface layer EGL rendering process; based on the intermediate interface layer EGL instance, draw the target texture data to a surface according to the waiting time to generate target surface data; input the target surface data into the input surface in the screen recording instance.

9. The mobile terminal of claim 7, wherein, The processing unit is further configured to: when the frame rate identifier is the second frame rate identifier, fill the surface data into the input surface; start an encoder thread, and obtain image encoding in the input surface based on the encoder thread; send the encoded image encoding to the display device. 10.A frame rate control method applied to the mobile terminal of any one of claims 1 to 9, the mobile terminal comprising a display unit, a communication unit, and a processing unit, the method comprising: The method comprises: obtain a first screen refresh rate and a second screen refresh rate, the first screen refresh rate is a maximum screen refresh rate currently reached by the mobile terminal, and the second screen refresh rate is a maximum screen refresh rate supported by the mobile terminal; obtain a decoding frame rate threshold of the display device, the decoding frame rate threshold is a maximum decoding rate value supported by the display device; determine a frame rate identifier according to the first screen refresh rate, the second screen refresh rate and the decoding frame rate threshold of the display device; the frame rate identifier comprises a first frame rate identifier and a second frame rate identifier, the first frame rate identifier is used to represent that the frame rate output mode of the mobile terminal is a stable frame rate output mode, and the second frame rate identifier is used to represent that the frame rate output mode of the mobile terminal is a non-stable frame rate output mode; when the frame rate identifier is the first frame rate identifier, create a stable frame rate thread, and output a projection picture with a stable frame rate through the stable frame rate thread. Encode the projection picture with the stable frame rate, and transmit the encoded projection picture to a display device, so that the display device decodes the projection picture and controls the display to output the decoded projection picture at the stable frame rate after receiving the projection picture.

Images