Frame-reducing screen recording method, system and storage medium based on Android

Abstract

The present application provides an Android-based frame-reducing screen recording method, system and storage medium, including: when obtaining the user's authorization permission for screen recording, creating and starting a callback thread by calling a system service, and if the callback state is triggered during screen recording, real-time screen image data is fed back through the callback thread; during the callback process of the callback thread, the screen image data is frame-reduced according to the current callback time and the preset inter-frame interval length to obtain target screen image data, and the target screen image data is data-encoded to obtain a recorded screen video; the use of the present application can avoid dependence on the OpenGL ES graphics library, reduce the difficulty of operation, operation steps and learning cost, thereby improving the efficiency of Android's frame-reducing screen recording.

Description

Technical Field

[0001] The present application relates to the field of screen recording technology, and more particularly to an Android-based frame-reducing screen recording method and system. Background Art

[0002] More and more applications (APPs) have screen recording functions, such as screen recording and live streaming, and screen recording and saving to files. The implementation of screen recording functions on Android phones depends on the application programming interface (API) related to the MediaProjection tool class provided by the system, but the MediaProjection tool class does not provide a method to control the frame rate. The frame rate of the recorded screen is often related to the configuration of the mobile phone screen. Now the frame rate of the mobile phone screen usually reaches 60fps (Frames per second), and even higher 90fps, 120fps or 144fps. High frame rate has a smoother and better experience, but it will occupy too much network bandwidth and disk memory in functions such as screen recording and live streaming, and saving to files. At the same time, the codec needs to have higher performance to meet the high frame rate encoding and decoding. In order to avoid these problems, it is necessary to control the frame rate. The existing screen recording frame reduction solution on Android is to use OpenGL ES (OpenGL for Embedded Systems) to reduce the frame.

[0003] OpenGL ES is a subset of the 3D graphics application programming interface in OpenGL (Open Graphics Library). Although it removes some of the functions of OpenGL, it is still a very large system with powerful functions, which leads to a high learning threshold, high difficulty, complex use and difficulty in understanding. In addition, in order to achieve the frame reduction requirement, it is necessary to create an EGL (Electronic Graphic Library) environment. EGL is the interface between rendering OpenGL ES and the native window system, which can shield the differences between different windows on different platforms. In addition, many complex steps are required, such as writing shader language code and off-screen rendering. Summary of the invention

[0004] The purpose of this application is to address the deficiencies of the above-mentioned existing related technologies and propose an Android-based frame reduction screen recording method and system, which can reduce dependence on external graphics libraries and improve screen recording efficiency.

[0005] In a first aspect, the present application provides a frame-reducing screen recording method based on Android, comprising:

[0006] When obtaining the user's authorization permission to record the screen, a callback thread is created and started by calling the system service. If the callback state is triggered during screen recording, the real-time screen data is fed back through the callback thread;

[0007] During the callback process of the callback thread, the screen image data is downgraded according to the current callback time and the preset inter-frame interval to obtain the target screen image data, and the target screen image data is encoded to obtain the screen recording video.

[0008] The present application performs frame reduction processing on the screen image data through the current callback time and the preset inter-frame interval length, so that the better the frame reduction effect is, the higher the screen recording efficiency is, thereby greatly improving the efficiency of frame reduction screen recording based on Android; and, when obtaining the screen recording authorization permission, a special callback thread is created to feedback the screen image data. When the callback state is triggered, the screen image data can be quickly obtained through the preset callback thread, and there is no need to wait for the callback state to be triggered and then create and start the callback thread, thereby improving the efficiency of Android frame reduction screen recording.

[0009] Further, the screen image data is subjected to frame reduction processing by using the current callback time and the preset inter-frame interval duration to obtain the target screen image data, including:

[0010] The initial frame interval duration is obtained through the limited frame rate required by the user, and the initial frame interval duration is corrected according to the standard callback interval of the screen image data to obtain the standard inter-frame interval duration. According to the inter-frame interval duration and the current callback time, the screen image data is downgraded to obtain the target screen image data.

[0011] The present application corrects the initial frame interval duration through the standard callback interval of the screen image data to obtain the standard inter-frame interval duration, so that the standard inter-frame interval duration is generally smaller than the initial frame interval duration. When the initial frame interval duration is reduced and corrected, more non-target screen image data that meets the requirements of frame loss to achieve frame reduction is screened out, and less target screen image data is screened out, thereby making the frame reduction effect better and the screen recording efficiency higher. That is, a frame reduction process is performed through the standard inter-frame interval duration, thereby improving the frame reduction screen recording efficiency based on Android.

[0012] Further, the screen picture data is subjected to frame reduction processing according to the inter-frame interval duration and the current callback time to obtain target screen picture data, including:

[0013] The difference between the current callback time and the last data encoding end time is used as the actual frame time interval duration. If the actual frame time interval duration is not less than the inter-frame interval duration, the screen image data is used as the target screen image data.

[0014] Otherwise, the screen image data is discarded to reduce the frame rate, and the next callback state is triggered to wait.

[0015] This application is based on a frame reduction process based on the standard inter-frame interval duration, and then determines whether the actual frame time interval duration is not less than the inter-frame interval duration. If the actual frame time interval duration is less than the inter-frame interval duration, the screen image data is discarded for a second frame reduction, and waits for the next callback state to be triggered, thereby eliminating the need to rely on an external graphics library, and the frame reduction process is simple to operate, thereby greatly improving the efficiency of frame reduction screen recording based on Android.

[0016] Further, the initial frame interval duration is corrected according to the standard callback interval of the screen picture data to obtain the standard inter-frame interval duration, including:

[0017] The standard inter-frame interval duration is subtracted from the standard callback interval of the screen image data to eliminate callback interval fluctuations caused by the callback of the screen image data, thereby obtaining a standard inter-frame interval duration.

[0018] Further, the target screen image data is encoded to obtain a screen recording video, including:

[0019] The target screen image data is encoded multiple times and compressed into target format data, and an audio and video mixer is created. The target format data is written into the audio and video mixer and then started to output the audio and video data. The audio and video data is then passed into the rendering tool class of the system service to obtain the screen recording video.

[0020] This embodiment uses multiple encoding to compress the target screen image data into a target format, which can reduce the memory or bandwidth occupied by the data, thereby improving the efficiency of frame reduction screen recording based on Android.

[0021] Further, the target screen image data is compressed into target format data after being encoded multiple times, including:

[0022] The target screen image data in RGBA data format is converted into YUV format data, and the YUV format data is compressed and encoded into H264 format data through a preset data encoder, and the H264 format data is used as the target format data.

[0023] Furthermore, the basis for determining the triggering callback status includes:

[0024] After the screen capture starts, if the size of the screen capture area changes, the callback is triggered; or,

[0025] When the screen recording session expires, the callback is triggered; or,

[0026] After the screen capture starts, if a visibility event occurs in the screen capture area, a callback is triggered; wherein the visibility event includes: page switching, user leaving or returning to a specified location.

[0027] Further, after obtaining the screen recording video, the method further includes:

[0028] If you need to end screen recording, call the screen recording stop method, thread end method, data callback stop method, data encoding stop method and data mixing stop method of the system service respectively to stop recording the screen, stop using the callback thread, stop data callback, stop data encoding and stop generating audio and video data.

[0029] In a second aspect, the present application provides an Android-based frame reduction screen recording system, including: a data acquisition module and a frame reduction and encoding module; wherein,

[0030] The data acquisition module is used to create and start a callback thread by calling the system service when obtaining the user's authorization permission to allow screen recording. If the callback state is triggered during screen recording, the real-time screen image data is fed back by the callback thread;

[0031] The frame reduction and encoding module is used to reduce the frame size of the screen image data according to the current callback time and the preset inter-frame interval duration during the callback process of the callback thread to obtain the target screen image data, and to encode the target screen image data to obtain the screen recording video.

[0032] The present application uses the current callback time and the preset inter-frame interval length to perform frame reduction processing on the screen image data through the frame reduction and encoding module, so that the better the frame reduction effect is, the higher the screen recording efficiency is, thereby greatly improving the efficiency of frame reduction screen recording based on Android; and, through the data acquisition module, when obtaining the screen recording authorization permission, a special callback thread is created to feedback the screen image data. When the callback state is triggered, the screen image data can be quickly obtained through the preset callback thread, and there is no need to wait for the callback state to be triggered and then create and start the callback thread, thereby improving the efficiency of Android frame reduction screen recording.

[0033] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the Android-based frame reduction screen recording method as described in the first aspect.

[0034] This application implements the Android-based frame reduction screen recording method in the form of a computer program and stores it in a readable computer storage medium, so that it can be run on different operating systems, servers or different smart devices. Smart devices include: personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices, so that it can be applied to different devices, thereby realizing Android frame reduction screen recording on different devices, with higher applicability and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of the existing related technology provided in the embodiment of the present application using OpenGL ES for frame reduction screen recording;

[0036] Figure 2 It is a flowchart of a frame reduction screen recording method based on Android provided in an embodiment of the present application;

[0037] Figure 3 This is a schematic diagram of the data format conversion process provided by an embodiment of the present application;

[0038] Figure 4 This is a flowchart of another frame reduction screen recording method based on Android provided in an embodiment of the present application;

[0039] Figure 5 It is a structural diagram of an Android-based frame reduction screen recording system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] It is worth noting that high frame rate provides a smoother and better experience, but it will take up too much network bandwidth and disk memory for functions such as screen recording, live streaming, and saving screen recordings to files. At the same time, the codec needs to have higher performance to meet the encoding and decoding of high frame rates. In order to avoid these problems, it is necessary to control the frame rate. The existing related technology uses OpenGL ES to reduce the frame rate. See Figure 1 , is a flow chart of the existing related technology provided by the embodiment of the present application using OpenGL ES for frame reduction screen recording, including: steps S11 to S16, specifically:

[0042] Step S11, introduce OpenGL ES components into the APP project and create an EGL environment;

[0043] Step S12: Create a vertex shader and a fragment shader in the EGL environment, and write shader codes using a shader language;

[0044] Step S13: Use the EGL environment to create a Surface for inputting screen recording data, and create an EGL off-screen rendering texture;

[0045] Step S14, when calling the createVirtualDisplay method of MediaProjection, pass in the created Surface for inputting the recorded screen data; then continuously render the recorded screen data onto the off-screen rendered texture;

[0046] Step S15, periodically fetch data from the off-screen rendered texture; for example, 30fps, then 33.33 milliseconds;

[0047] Step S16: Encode the retrieved data to obtain encoded data, and encapsulate the encoded data for streaming or save it to a file.

[0048] Figure 1 The existing related technologies shown in the document require the use of a large OpenGL ES, which has a high learning threshold and is difficult to learn. It is complex to use and difficult to understand. To achieve the frame reduction requirement, it is necessary to create an EGL environment, write shader language code, and perform off-screen rendering and many other complex steps. Based on this, the embodiments of the present application provide an Android-based frame reduction screen recording method, system, and storage medium. Through the built-in frame reduction operation, it avoids dependence on the OpenGL ES graphics library, reduces the difficulty of operation, operation steps, and learning cost, thereby improving the efficiency of Android's frame reduction screen recording.

[0049] In order to more clearly illustrate the scheme of the present application, it will be described in detail from the following examples.

[0050] Example 1

[0051] See also Figure 2 , is a flowchart of a frame-reducing screen recording method based on Android provided in an embodiment of the present application, including: steps S21 to S22, specifically:

[0052] Step S21, when obtaining the user's authorization permission to allow screen recording, create and start a callback thread by calling the system service. If the callback state is triggered during screen recording, real-time screen image data is fed back through the callback thread.

[0053] In some embodiments, obtaining the authorization permission of the user to allow screen recording includes: obtaining the screen recorder management class MediaProjectionManager, and applying for authorization of the screen recording function from the user through the screen recorder management class MediaProjectionManager, and after the user agrees to the authorization of the screen function, obtaining a screen recorder class MediaProjection through the screen recorder management class MediaProjectionManager. Among them, the screen recorder management class MediaProjectionManager is a class used to manage the screen recorder class MediaProjection.

[0054] In some embodiments, calling a system service to create and start a callback thread includes: using a thread class HandlerThread to create a callback thread, and starting the callback thread.

[0055] In some embodiments, after the callback thread is started, the status callback of the screen device class MediaProjection is set, and the callback status is monitored. If the callback status is triggered, a callback is performed in the created callback thread.

[0056] In some embodiments of the present application, the basis for determining whether to trigger the callback state includes: after the screen recording capture starts, if the size of the screen recording capture area changes, triggering the callback; or, when the screen recording session expires, triggering the callback; or, after the screen recording capture starts, if a visibility event occurs in the screen recording capture area, triggering the callback; wherein, visibility events include: page switching, user leaving or returning to a specified location.

[0057] Step S22: During the callback process of the callback thread, the screen image data is down-framed according to the current callback time and the preset inter-frame interval to obtain the target screen image data, and the target screen image data is encoded to obtain the screen recording video.

[0058] In some embodiments of the present application, the screen image data is downgraded according to the current callback time and the preset inter-frame interval duration to obtain target screen image data, including: obtaining the initial frame interval duration through the limited frame rate required by the user, correcting the initial frame interval duration according to the standard callback interval of the screen image data to obtain the standard inter-frame interval duration, and downgrading the screen image data according to the inter-frame interval duration and the current callback time to obtain the target screen image data.

[0059] In some embodiments of the present application, the screen image data is downgraded according to the inter-frame interval duration and the current callback time to obtain target screen image data, including: taking the difference between the current callback time and the last data encoding end time as the actual frame time interval duration; if the actual frame time interval duration is not less than the inter-frame interval duration, the screen image data is taken as the target screen image data; otherwise, the screen image data is discarded for frame downgrading, and waits for the next callback state to be triggered.

[0060] In some embodiments of the present application, the initial frame interval duration is corrected according to the standard callback interval of the screen image data to obtain the standard inter-frame interval duration, including: the initial frame interval duration minus the standard callback interval of the screen image data to eliminate the callback interval fluctuation caused by the callback of the screen image data to obtain the standard inter-frame interval duration.

[0061] In some embodiments of the present application, the acquireLatestImage method of the screen data receiver ImageReader is called during the callback to obtain the latest screen data (the latest screen data can be an Image class) from the queue of the screen data receiver ImageReader, and then the current callback time (nanoseconds) is subtracted from the time after the last callback data is encoded. If it is greater than or equal to the inter-frame interval, the screen data of this callback is used; if it is less than the inter-frame interval, the screen data of this callback is discarded.

[0062] In some embodiments of the present application, the inter-frame interval duration = (1000000000 / limited frame rate required by the user) - 2000000, in nanoseconds. For example, if the limited frame rate required by the user is 30 frames, 1000000000 / 30fps-2000000=31333333 nanoseconds=31.333333 milliseconds; if the limited frame rate required by the user is 25 frames, 1000000000 / 25-2000000=38000000 nanoseconds=38 milliseconds, that is, the overall inter-frame interval duration is longer.

[0063] The embodiment of the present application corrects the initial frame interval duration through the standard callback interval of the screen image data to obtain the standard inter-frame interval duration, so that the standard inter-frame interval duration is generally smaller than the initial frame interval duration. The reduction and correction of the initial frame interval duration will further improve the efficiency of frame reduction, and the less the target screen image data is filtered out, so that the effect of frame reduction is better and the screen recording efficiency is higher. That is, a frame reduction process is performed through the standard inter-frame interval duration, and then it is determined whether the actual frame time interval duration is not less than the inter-frame interval duration. If the actual frame time interval duration is less than the inter-frame interval duration, the screen image data is discarded for a second frame reduction, and wait for the next callback state to be triggered, thereby further improving the efficiency of frame reduction, and thus greatly improving the efficiency of frame reduction screen recording based on Android.

[0064] In some embodiments of the present application, target screen image data is encoded to obtain a screen recording video, including: compressing the target screen image data into target format data after multiple encodings, creating an audio and video mixer, writing the target format data to the audio and video mixer and then starting it, outputting the audio and video data, and passing the audio and video data into a rendering tool class of a system service to obtain a screen recording video.

[0065] In some embodiments of the present application, the target screen image data is encoded multiple times and then compressed into target format data, including: converting the target screen image data in RGBA data format into YUV format data, and compressing and encoding the YUV format data into H264 format data through a preset data encoder, and using the H264 format data as the target format data.

[0066] The embodiment of the present application compresses and encodes YUV format data into H264 format data, which can avoid the situation where YUV data occupies more memory or bandwidth and causes a decrease in screen recording efficiency, thereby improving screen recording efficiency.

[0067] It is worth noting that the data flow direction of the embodiment of the present application is: RGBA data->YUV data->H264 data->MP4 data.

[0068] In some embodiments of the present application, the target screen image data in RGBA data format is converted into YUV format data, and the YUV format data is compressed and encoded into H264 format data through a preset data encoder, and the H264 format data is used as the target format data, including: sub-steps S201 to S204, see Figure 3 , is a schematic diagram of the data format conversion process provided by an embodiment of the present application, specifically:

[0069] Sub-step S201, accessing RGBA format data by obtaining the HardwareBuffer handle of the Image class, and obtaining the RGBA format data in the NDK (Native Development Kit) toolkit; wherein, NDK is a toolkit belonging to the Android service system itself.

[0070] Sub-step S202: using the libyuv library to convert the RGBA format data into YUV format data.

[0071] Sub-step S203: determine whether the data encoder MediaCodec is created. If the data encoder MediaCodec is not created, create the data encoder MediaCodec, configure the encoder parameters, and start the encoder to prepare for encoding.

[0072] Sub-step S204: sending the converted YUV format data to the data encoder MediaCodec for encoding, and outputting the H264 format data as the target format data.

[0073] In some embodiments of the present application, an audio and video mixer is created, and the target format data is written to the audio and video mixer and then started to output the audio and video data, including: judging the data type of the target format data, if it is encoder format data, creating an MP4 audio and video mixer MediaMuxer, adding a video stream to the MP4 audio and video mixer MediaMuxer, and starting the MP4 audio and video mixer MediaMuxer.

[0074] In some embodiments of the present application, audio and video data are passed into the rendering tool class of the system service to obtain a screen recording video, including: obtaining the surface component of the screen data receiver ImageReader, and calling the createVirtualDisplay method of the screen recorder MediaProjection to pass it into the surface component for rendering to obtain the screen recording video, that is, start recording the screen.

[0075] In some embodiments of the present application, after obtaining the screen recording video, it also includes: if it is necessary to end the screen recording, the screen recording stop method, thread end method, data callback stop method, data encoding stop method and data mixing stop method of the system service are called respectively to stop recording the screen, stop using the callback thread, stop data callback, stop data encoding and stop generating audio and video data.

[0076] In some embodiments of the present application, when it is necessary to end screen recording, the screen recording stop method of the screen recorder MediaProjection is called to stop recording the screen image, the end thread quitSafely of the thread class HandlerThread ends the created thread, the screen image data receiver ImageReader stops the screen data callback by the stop data callback close method, the data encoder MediaCodec stops the data encoding by the AMediaCodec_stop method to stop the encoder, and the audio and video mixer MediaMuxer stops the data mixing by the AMediaMuxer_stop method to stop the audio and video mixer MediaMuxer. That is, the screen recording video file (for example: MP4 file) can be seen in the specified path.

[0077] The embodiment of the present application corrects the initial frame interval duration through the standard callback interval of the screen image data to obtain the standard inter-frame interval duration, so that the standard inter-frame interval duration is generally smaller than the initial frame interval duration. When the initial frame interval duration is reduced and corrected, more non-target screen image data that meets the requirements of frame loss to achieve frame reduction is screened out, and less target screen image data is screened out, thereby making the frame reduction effect better and the screen recording efficiency higher. That is, a frame reduction process is performed through the standard inter-frame interval duration, thereby improving the frame reduction screen recording efficiency based on Android.

[0078] Example 2

[0079] See also Figure 4 , is a flowchart of another frame-reducing screen recording method based on Android provided in an embodiment of the present application, including: steps S41 to S47, specifically:

[0080] Step S41, obtain the screen recorder management class MediaProjectionManager, and apply to the user for authorization of the screen recording function.

[0081] For example, the statement can be expressed as:

[0082]

[0083] Step S42: After the user agrees to the authorization of the screen function, the screen recorder class MediaProjection is obtained through the screen recorder management class MediaProjectionManager.

[0084] For example, the statement can be expressed as:

[0085]

[0086] Step S43: Use the thread class HandlerThread to create a callback thread and start the callback thread.

[0087] For example, the statement can be expressed as:

[0088]

[0089] Step S44, set the callback state of the screen recorder class MediaProjection, and set the callback in the callback thread created in step S43.

[0090] For example, the statement can be expressed as:

[0091]

[0092] Step S45, create a screen data receiver ImageReader, which is used to receive the callback of the recorded screen data (RGBA format data), and set the callback in the callback thread created in step S43; wherein, after the screen recording starts, the screen data (RGBA format data) will be continuously transmitted to the screen data receiver ImageReader, and frame reduction processing and data encoding will be performed in the callback.

[0093] The frame reduction processing and data encoding of step S45 include: sub-steps S451 to S456, specifically:

[0094] Sub-step S451, during the callback process of the callback thread, the screen image data is downgraded by the current callback time and the preset inter-frame interval duration to obtain the target screen image data. Specifically, during the callback, the acquireLatestImage method of the screen data receiver ImageReader is called to obtain the screen data (Image class), and then the current callback time (nanoseconds) minus the time after the last callback data is encoded is determined. If it is greater than or equal to the inter-frame interval duration, the screen image data of this callback is used, and if it is less than the inter-frame interval duration, the screen image data of this callback is discarded.

[0095] For example, the statement can be expressed as:

[0096]

[0097]

[0098] Sub-step S452, screen image data encoding: access RGBA format data by obtaining the HardwareBuffer handle of the Image class, and obtain the RGBA format data in the NDK.

[0099] For example, the statement can be expressed as:

[0100]

[0101] Sub-step S453: Use libyuv library to convert RGBA format data into YUV format data. This is because the data encoder MediaCodec only supports YUV format data.

[0102] For example, the statement can be expressed as:

[0103]

[0104] Sub-step S454, determine whether the data encoder MediaCodec is created, if not, create the data encoder MediaCodec, configure the encoder parameters and start the encoder to prepare for encoding.

[0105] For example, the statement can be expressed as:

[0106]

[0107]

[0108] Sub-step S455, sending the converted YUV format data to the data encoder MediaCodec for encoding, and outputting the H264 format data as the target format data.

[0109] For example, the statement can be expressed as:

[0110]

[0111] Sub-step S456, add the video stream to the MP4 audio and video mixer MediaMuxer, start the MP4 audio and video mixer MediaMuxer, and output the audio and video data. Specifically, if the target format data is the encoder format data (H264 format), create the MP4 audio and video mixer MediaMuxer, add the video stream to the MP4 audio and video mixer MediaMuxer (the encoder format data will only appear for the first time), and start the MP4 audio and video mixer MediaMuxer to output the audio and video data (MP4 data).

[0112] For example, the statement can be expressed as:

[0113]

[0114] Step S46, obtain the surface component of the screen data receiver ImageReader, and call the createVirtualDisplay method of the screen recorder MediaProjection to input the audio and video data (MP4 data) into the surface component for rendering. The surface component outputs the final screen recording video, and starts to obtain the recorded screen image.

[0115] It is worth noting that in step S46, a connection is established between the screen data receiver ImageReader and the screen recorder MediaProjection. If the connection is not established, the screen data callback described in step S45 will not be called back, that is, the screen data cannot be obtained.

[0116] Step S47, when it is necessary to end screen recording, call the screen recording stop method of the screen recorder MediaProjection to stop recording the screen image, the end thread quitSafely of the thread class HandlerThread to end the created thread, the screen image data receiver ImageReader's stop data callback close method to stop the screen data callback, the data encoder MediaCodec stops data encoding using the AMediaCodec_stop method to stop the encoder, and the audio and video mixer MediaMuxer stops data mixing using the AMediaMuxer_stop method to stop the audio and video mixer MediaMuxer. That is, you can see the screen recording video file (MP4 file) in the specified path.

[0117] The embodiment of the present application uses the screen recorder, thread class, screen image data receiver, data encoder and audio and video mixer of the Android system service itself to realize the correction of the initial frame interval duration. First, by using the components of the Android system service itself, it can effectively avoid the external graphics library OpenGL ES dependency is reduced, thereby reducing the difficulty of operation, operation steps and learning costs, thereby improving the efficiency of frame reduction screen recording based on Android; secondly, by correcting the initial frame interval duration, the standard frame interval duration is generally less than the initial frame interval duration, and the initial frame interval duration is reduced and corrected, then the more non-target screen image data that meets the frame loss to achieve frame reduction is screened out, and the less target screen image data is screened out, so that the frame reduction effect is better and the screen recording efficiency is higher, that is, a frame reduction process is performed through the standard frame interval duration, thereby improving the efficiency of frame reduction screen recording based on Android; then, based on a frame reduction process performed through the standard frame interval duration, it is determined whether the actual frame time interval duration is not less than the frame interval duration. If the actual frame time interval duration is less than the frame interval duration, the screen image data is discarded for a second frame reduction, and wait for the next callback state to be triggered, so there is no need to rely on an external graphics library, and the frame reduction process is simple to operate, thereby greatly improving the efficiency of frame reduction screen recording based on Android.

[0118] Example 3

[0119] See also Figure 5 , is a structural diagram of an Android-based frame reduction screen recording system provided in an embodiment of the present application, including: a data acquisition module 51 and a frame reduction and encoding module 52.

[0120] In some embodiments, after the data acquisition module 51 obtains the user's authorization permission to allow screen recording, it outputs real-time screen image data through a callback thread and transmits the screen image data to the frame reduction and encoding module 52; after receiving the screen image data, the frame reduction and encoding module 52 outputs the screen recording video.

[0121] The data acquisition module 51 is used to create and start a callback thread by calling the system service when obtaining the user's authorization permission to allow screen recording. If the callback state is triggered during screen recording, real-time screen image data is fed back through the callback thread.

[0122] In some embodiments of the present application, the basis for determining whether to trigger the callback state includes: after the screen recording capture starts, if the size of the screen recording capture area changes, triggering the callback; or, when the screen recording session expires, triggering the callback; or, after the screen recording capture starts, if a visibility event occurs in the screen recording capture area, triggering the callback; wherein, visibility events include: page switching, user leaving or returning to a specified location.

[0123] The frame reduction and encoding module 52 is used to reduce the frame of the screen image data according to the current callback time and the preset inter-frame interval duration during the callback process of the callback thread to obtain the target screen image data, and to encode the target screen image data to obtain the screen recording video.

[0124] In some embodiments of the present application, the screen image data is downgraded according to the current callback time and the preset inter-frame interval duration to obtain target screen image data, including: obtaining the initial frame interval duration through the limited frame rate required by the user, correcting the initial frame interval duration according to the standard callback interval of the screen image data to obtain the standard inter-frame interval duration, and downgrading the screen image data according to the inter-frame interval duration and the current callback time to obtain the target screen image data.

[0125] In some embodiments of the present application, the screen image data is downgraded according to the inter-frame interval duration and the current callback time to obtain target screen image data, including: taking the difference between the current callback time and the last data encoding end time as the actual frame time interval duration; if the actual frame time interval duration is not less than the inter-frame interval duration, the screen image data is taken as the target screen image data; otherwise, the screen image data is discarded for frame downgrading, and waits for the next callback state to be triggered.

[0126] In some embodiments of the present application, the initial frame interval duration is corrected according to the standard callback interval of the screen image data to obtain the standard inter-frame interval duration, including: the initial frame interval duration minus the standard callback interval of the screen image data to eliminate the callback interval fluctuation caused by the callback of the screen image data to obtain the standard inter-frame interval duration.

[0127] In some embodiments of the present application, target screen image data is encoded to obtain a screen recording video, including: compressing the target screen image data into target format data after multiple encodings, creating an audio and video mixer, writing the target format data to the audio and video mixer and then starting it, outputting the audio and video data, and passing the audio and video data into a rendering tool class of a system service to obtain a screen recording video.

[0128] In some embodiments of the present application, the target screen image data is encoded multiple times and then compressed into target format data, including: converting the target screen image data in RGBA data format into YUV format data, and compressing and encoding the YUV format data into H264 format data through a preset data encoder, and using the H264 format data as the target format data.

[0129] In some embodiments of the present application, after obtaining the screen recording video, it also includes: if it is necessary to end the screen recording, the screen recording stop method, thread end method, data callback stop method, data encoding stop method and data mixing stop method of the system service are called respectively to stop recording the screen, stop using the callback thread, stop data callback, stop data encoding and stop generating audio and video data.

[0130] In the embodiment of the present application, the frame reduction and encoding module 52 uses the current callback time and the preset inter-frame interval to perform frame reduction processing on the screen image data, so that the better the frame reduction effect is, the higher the screen recording efficiency is, thereby greatly improving the efficiency of frame reduction screen recording based on Android; and, when the screen recording authorization permission is obtained, the data acquisition module 51 creates a special callback thread to feedback the screen image data. When the callback state is triggered, the screen image data can be quickly obtained through the preset callback thread, and there is no need to wait for the callback state to be triggered and then create and start the callback thread, thereby improving the efficiency of Android frame reduction screen recording.

[0131] Example 4

[0132] It is a readable computer storage medium provided in an embodiment of the present application, on which a computer program is stored. When the computer program is executed by a processor, the steps of the Android-based frame reduction screen recording method as described in Example 1 or Example 2 are implemented.

[0133] This application implements the Android-based frame reduction screen recording method in the form of a computer program and stores it in a readable computer storage medium, so that it can be run on different operating systems, servers or different smart devices. Smart devices include: personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices, so that it can be applied to different devices, thereby realizing Android frame reduction screen recording on different devices, with higher applicability and practicality.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may also provide computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A frame-reducing screen recording method based on Android, characterized in that: include: When obtaining the user's authorization permission to record the screen, a callback thread is created and started by calling the system service. If the callback state is triggered during screen recording, real-time screen data is fed back through the callback thread; During the callback process of the callback thread, the screen image data is down-framed according to the current callback time and the preset inter-frame interval duration to obtain target screen image data, and the target screen image data is data-encoded to obtain a screen recording video; The step of performing frame reduction processing on the screen image data according to the current callback time and the preset inter-frame interval duration to obtain the target screen image data includes: The initial frame interval duration is obtained through the limited frame rate required by the user. The initial frame interval duration is corrected according to the standard callback interval of the screen image data to obtain the standard inter-frame interval duration. The difference between the current callback time and the last data encoding end time is used as the actual frame time interval duration. If the actual frame time interval duration is not less than the inter-frame interval duration, the screen image data is used as the target screen image data; otherwise, the screen image data is discarded for frame reduction, and wait for the next callback state to be triggered.

2. The frame-reducing screen recording method according to claim 1, characterized in that: The correcting the initial frame interval duration according to the standard callback interval of the screen picture data to obtain the standard inter-frame interval duration includes: The standard inter-frame interval duration is subtracted from the standard callback interval of the screen image data to eliminate callback interval fluctuations caused by the callback of the screen image data, thereby obtaining a standard inter-frame interval duration.

3. The frame-reducing screen recording method according to claim 1, characterized in that: The step of encoding the target screen image data to obtain a screen recording video includes: The target screen image data is encoded multiple times and compressed into target format data, and an audio and video mixer is created. The target format data is written into the audio and video mixer and then started to output the audio and video data. The audio and video data is then passed into the rendering tool class of the system service to obtain the screen recording video.

4. The frame-reducing screen recording method according to claim 3, characterized in that: The step of encoding the target screen image data multiple times and compressing the target screen image data into target format data comprises: The target screen image data in RGBA data format is converted into YUV format data, and the YUV format data is compressed and encoded into H264 format data through a preset data encoder, and the H264 format data is used as the target format data.

5. The frame-reducing screen recording method according to claim 1, characterized in that: The basis for determining the triggering callback status includes: After the screen capture starts, if the size of the screen capture area changes, the callback is triggered; or, When the screen recording session expires, the callback is triggered; or, After the screen capture starts, if a visibility event occurs in the screen capture area, a callback is triggered; wherein the visibility event includes: page switching, user leaving or returning to a specified location.

6. The frame-reducing screen recording method according to claim 1, characterized in that: After obtaining the screen recording video, the method further includes: If you need to end screen recording, call the screen recording stop method, thread end method, data callback stop method, data encoding stop method and data mixing stop method of the system service respectively to stop recording the screen, stop using the callback thread, stop data callback, stop data encoding and stop generating audio and video data.

7. A frame-reducing screen recording system based on Android, characterized in that: include: Data acquisition module and frame reduction and encoding module; wherein, The data acquisition module is used to create and start a callback thread by calling the system service when obtaining the user's authorization permission to allow screen recording. If the callback state is triggered during screen recording, the real-time screen image data is fed back through the callback thread; The frame reduction and encoding module is used to reduce the frame of the screen image data according to the current callback time and the preset inter-frame interval duration during the callback process of the callback thread to obtain the target screen image data, and to perform data encoding on the target screen image data to obtain the screen recording video; The step of performing frame reduction processing on the screen image data according to the current callback time and the preset inter-frame interval duration to obtain the target screen image data includes: The initial frame interval duration is obtained through the limited frame rate required by the user. The initial frame interval duration is corrected according to the standard callback interval of the screen image data to obtain the standard inter-frame interval duration. The difference between the current callback time and the last data encoding end time is used as the actual frame time interval duration. If the actual frame time interval duration is not less than the inter-frame interval duration, the screen image data is used as the target screen image data; otherwise, the screen image data is discarded for frame reduction, and wait for the next callback state to be triggered.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the Android-based frame reduction screen recording method as described in any one of claims 1-6 are implemented.

Images