Screen video recording method and device and electronic equipment

Abstract

The invention discloses a screen video recording method and device and electronic equipment, and the method is applied to remote control software, and comprises the steps: responding to the creation of a remote window of the remote control software, capturing a screen where the remote window is currently located, and obtaining a screenshot image at the current moment; converting the screenshot image at the current moment into a first target image with a target image type; and obtaining a recorded video of the remote window based on synthesis of the first target image at the current moment with the second target image and / or the third target image. Through the technical scheme provided by the embodiment of the invention, the remote control software also has a video recording function in a remote control process.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a screen recording method, device and electronic device. Background Art

[0002] Remote control software is a network-based software that allows users to remotely control one or more computers (controlled terminals) through one computer (the master terminal).

[0003] The above remote control software includes TightVNC. TightVNC is a software that uses the Remote Frame Buffer (RFB) protocol for screen sharing and remote operation. TightVNC can transmit keyboard and mouse actions and real-time screen images over the network, and provides basic remote control and file transfer capabilities.

[0004] However, TightVNC does not have the screen recording function during remote control. Summary of the invention

[0005] This application provides a screen recording method, device and electronic device to solve the problem that remote control software (such as TightVNC) does not have the screen recording function during remote control. The specific implementation scheme is as follows:

[0006] In a first aspect, the present application provides a screen recording method, which is applied to remote control software, and the method comprises:

[0007] In response to the creation of the remote window of the remote control software, capturing the screen where the remote window is currently located to obtain a screenshot image at the current moment;

[0008] Converting the screenshot image at the current moment into a first target image having a target image type; wherein the file size of the first target image is smaller than the file size of the screenshot image;

[0009] Based on the synthesis of the first target image at the current moment with the second target image and / or the third target image, the recorded video of the remote window is obtained; wherein the second target image is the target image before the current moment, and the third target image is the target image after the current moment.

[0010] Through the above application embodiment, after the remote window of the remote control software is successfully created, the screenshot image at the current moment obtained by capturing the current screen of the remote window is converted into a first target image with a target image type. Since the file size of the first target image is smaller than the file size of the screenshot image, the file size of the image is effectively reduced, saving storage space. Then, based on the synthesis of the first target image at the current moment with the second target image and / or the third target image, the recorded video of the remote window is obtained, thereby completing the recording of the remote window of the remote control software, so that the remote control software (such as TightVNC) has the screen recording function during the remote control process.

[0011] In a possible implementation, capturing the current screen of the remote window to obtain a screenshot image at the current moment includes:

[0012] Determine whether the remote window meets a first screenshot condition; wherein the first screenshot condition is that the remote window is not closed;

[0013] If so, determine whether the remote window meets the second screenshot condition. If it is determined that the remote window meets the second screenshot condition, capture the screen where the remote window is currently located to obtain the screenshot image at the current moment; wherein, the second screenshot condition is that the remote window is not a minimized window.

[0014] Through the above-mentioned application embodiment, when the remote window meets the first screenshot condition, that is, when the remote window is not closed, there is no need to end the recording of the remote window. Then, when the remote window also meets the second screenshot condition (that is, the remote window is not a minimized window), the current screen of the remote window is captured, thereby reducing the capture of invalid images.

[0015] In a possible implementation manner, after determining whether the remote window satisfies the second screenshot condition, the method further includes:

[0016] If it is determined that the remote window does not meet the second screenshot condition, the screenshot of the remote window is paused, and after pausing for a preset time, it is continued to determine whether the remote window meets the first screenshot condition.

[0017] Through the above-mentioned application embodiment, under the premise that the remote window does not meet the second screenshot condition, that is, when the remote window is a minimized window, the screenshot of the remote window is paused, thereby further reducing the capture of invalid images, and after pausing for a preset period of time, it is continued to be determined whether the remote window meets the first screenshot condition, so as to continuously capture images of the screen where the remote window is currently located at different times.

[0018] In a possible implementation, capturing the current screen of the remote window to obtain a screenshot image at the current moment includes:

[0019] According to target requirements, the screen where the remote window is currently located is captured to obtain the screenshot image at the current moment; wherein the target requirements include the current position and window size of the remote window.

[0020] Through the above-mentioned application embodiments, the current screen of the remote window is captured according to the target requirements, so that the screenshot image captured at the current moment meets the target requirements, thereby facilitating the accurate screen recording of the remote window. Compared with full-screen recording that records all the content the user sees on the screen, it can avoid the problem of privacy information leakage caused by the user's privacy information being recorded, thereby improving the security of user information.

[0021] In a possible implementation, capturing the current screen of the remote window to obtain a screenshot image at the current moment includes:

[0022] Determine whether the image driver is installed in the terminal;

[0023] If yes, then the screen where the remote window is currently located is captured by using the mirror drive technology to obtain the screenshot image at the current moment;

[0024] If not, the screen where the remote window is currently located is captured through the application program hook API Hook technology to obtain the screenshot image at the current moment.

[0025] Through the above application embodiment, regardless of whether the mirror driver is installed in the terminal, the current screen of the remote window can be captured. Then, when the mirror driver is installed in the terminal, the mirror driver technology is preferentially used to capture the screen, which significantly reduces the time consumption of a screenshot, thereby helping to improve the synthesis efficiency of the recorded video of the remote window.

[0026] In a possible implementation manner, the synthesizing the first target image at the current moment with the second target image and / or the third target image to obtain the recorded video of the remote window includes:

[0027] Using video synthesis software, synthesize the first target image at the current moment, the second target image and / or the third target image into a segmented video with segmented duration;

[0028] In response to the remote window not satisfying a first screenshot condition, a plurality of segmented videos containing target images at different moments are merged into the recorded video of the remote window; wherein the first screenshot condition is that the remote window is not closed.

[0029] Through the above application embodiment, it is avoided that all screenshots of the remote window captured are invalid due to a fault during the recording process. Even if a fault suddenly occurs during the recording of the remote window, some information before the fault is still saved, so that the recording of the remote window has fault tolerance.

[0030] In a possible implementation, the video synthesis software is: software that is tailored for special content in the original video synthesis software; wherein the special content includes a non-target encoder, a non-target decoder and a network communication module in the original video synthesis software.

[0031] Through the above-mentioned application embodiments, based on the tailoring of special content, non-essential content in the original video synthesis software is removed, thereby greatly improving the efficiency of the video synthesis process.

[0032] In a second aspect, the present application further provides a screen recording device, the device being applied to remote control software, the device comprising:

[0033] A capture module, for responding to the creation of a remote window of the remote control software, capturing the screen where the remote window is currently located, and obtaining a screenshot image at the current moment;

[0034] A conversion module, configured to convert the screenshot image at the current moment into a first target image having a target image type; wherein the file size of the first target image is smaller than the file size of the screenshot image;

[0035] A processing module is used to obtain the recorded video of the remote window based on the synthesis of the first target image at the current moment with the second target image and / or the third target image; wherein the second target image is the target image before the current moment, and the third target image is the target image after the current moment.

[0036] In a possible implementation, the capture module is specifically used to determine whether the remote window satisfies a first screenshot condition; wherein the first screenshot condition is that the remote window is not closed; if so, determine whether the remote window satisfies a second screenshot condition, and if it is determined that the remote window satisfies the second screenshot condition, capture the screen where the remote window is currently located to obtain the screenshot image at the current moment; wherein the second screenshot condition is that the remote window is not a minimized window.

[0037] In a possible implementation, the capture module is specifically configured to pause the screenshot of the remote window if it is determined that the remote window does not meet the second screenshot condition, and continue to determine whether the remote window meets the first screenshot condition after pausing for a preset period of time.

[0038] In a possible implementation, the capture module is specifically configured to capture the screen where the remote window is currently located according to target requirements to obtain the screenshot image at the current moment; wherein the target requirements include the current position and window size of the remote window.

[0039] In a possible implementation, the capture module is specifically used to determine whether a mirror driver is installed in the terminal; if so, the screen where the remote window is currently located is captured through the mirror driver technology to obtain the screenshot image at the current moment; if not, the screen where the remote window is currently located is captured through the application hook API Hook technology to obtain the screenshot image at the current moment.

[0040] In a possible implementation, the processing module is specifically used to use video synthesis software to synthesize the first target image at the current moment, the second target image and / or the third target image into a segmented video with segmented duration; in response to the remote window not satisfying the first screenshot condition, merge multiple segmented videos containing target images at different moments into the recorded video of the remote window; wherein the first screenshot condition is that the remote window is not closed.

[0041] In a possible implementation, the video synthesis software is: software that is tailored for special content in the original video synthesis software; wherein the special content includes a non-target encoder, a non-target decoder and a network communication module in the original video synthesis software.

[0042] In a third aspect, the present application provides an electronic device, including:

[0043] Memory, used to store computer programs;

[0044] The processor is used to implement the above-mentioned screen recording method steps when executing the computer program stored in the memory.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned screen recording method steps are implemented.

[0046] For each aspect from the second to the fourth aspect and the technical effects that may be achieved by each aspect, please refer to the above description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a screen recording method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the processing process of the screen recording method provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a screen recording device provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The specific operating methods in the method embodiments can also be applied to device embodiments or system embodiments. It should be noted that in the description of the present application, "multiple" is understood as "at least two". "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A is connected to B, which can represent: A is directly connected to B and A is connected to B through C. In addition, in the description of the present application, words such as "first" and "second" are only used to distinguish the purpose of description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0052] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0053] Screen recording is a technology that can capture all activities on the screen. Through screen recording, all user operations on the screen can be recorded, and then the video obtained through screen recording can be converted into a video file, which can help users convey information more effectively and save a lot of time. Screen recording also includes screen recording during remote control. Through screen recording during remote control, enterprises can monitor and record all activities of operators on the target device through the recorded video obtained by screen recording, providing strong evidence support for subsequent quality control supervision. Therefore, screen recording during remote control is of great significance. However, some remote control software (such as TightVNC) does not support screen recording during remote control, that is, it does not have a recording function during remote control.

[0054] Therefore, the present application proposes a screen recording method, which is applied to remote control software. In response to the creation of a remote window of the remote control software, the screen where the remote window is currently located is captured to obtain a screenshot image at the current moment, and then the screenshot image at the current moment is converted into a first target image with a target image type, and then based on the first target image at the current moment and the second target image (the second target image is the target image before the current moment) and / or the third target image (the third target image is the target image after the current moment) merged, the recorded video of the remote window is obtained, so that the remote control software has a recording function during the remote control process. In addition, through the conversion of the screenshot image at the current moment and the first target image at the current moment, the file size of the image is reduced, saving storage space.

[0055] In the embodiment of the present application, the remote control software may be TightVNC, but is not limited thereto.

[0056] Taking TightVNC in remote control software as an example, in order to enable TightVNC to also have a recording function during the remote control process, a screen recording component can be implanted in the TightVNC, so as to realize the recording function based on the screen recording component. The screen recording component can be used to respond to the creation of a remote window of the remote control software, capture the current screen of the remote window, obtain the screenshot image at the current moment, and then convert the screenshot image at the current moment into a first target image with a target image type, and then obtain the recorded video of the remote window based on the synthesis of the first target image at the current moment with the second target image and / or the third target image, so that TightVNC also has a recording function during the remote control process.

[0057] Reference Figure 1 The figure is a flow chart of a screen recording method provided by an embodiment of the present application, the method comprising:

[0058] S101, in response to the creation of a remote window of remote control software, capturing the screen where the remote window is currently located to obtain a screenshot image at the current moment.

[0059] The screenshot image may be an image in a bitmap (English: Bitmap, abbreviated as BMP) format. The BMP format may be a device-independent bitmap (English: Device-independent bitmap, abbreviated as DIB) format, that is, the screenshot image may be a DIB bitmap.

[0060] When the remote window of the remote control software is successfully created, the recording function of the remote window can be enabled to obtain the recorded video of the remote window later.

[0061] In the embodiment of the present application, the creation of the remote window can be created by a first function for creating a window. For example, the first function is a vncsession_handle_new_window function, so that the remote window is created based on the vncsession_handle_new_window function.

[0062] In addition, a start function for starting the recording function is also embedded in the first function, so that the recording function can be started immediately after the remote window is successfully created. The start function may be a record_start_fun function.

[0063] Furthermore, when the remote window of the remote control software is successfully created, the screen where the remote window is currently located is captured to obtain a screenshot image at the current moment.

[0064] Optionally, in order to avoid capturing irrelevant images in the screen where the remote window is currently located, the above-mentioned capturing the screen where the remote window is currently located and obtaining the screenshot image at the current moment can be captured under the premise of satisfying the screenshot condition. The screenshot condition includes a first screenshot condition and a second screenshot condition. Among them, the first screenshot condition is that the remote window is not closed; the second screenshot condition is that the remote window is not minimized.

[0065] Specifically, firstly, it is determined whether the remote window satisfies the first screenshot condition.

[0066] If it is determined that the remote window meets the first screenshot condition, that is, the remote window is not closed at this time, so there is no need to end the recording of the remote window, then continue to determine whether the remote window meets the second screenshot condition.

[0067] If it is determined that the remote window satisfies the second capture condition, that is, the remote window is not a minimized window at this time, then the screen where the remote window is currently located is captured to obtain a screenshot image at the current moment.

[0068] In other words, when it is determined that the remote window is not closed (i.e., it is not necessary to end the recording of the remote window) and the remote window is not a minimized window, the current screen of the remote window is captured to obtain the screenshot image at the current moment, thereby avoiding continuing to capture the screen when the recording needs to be ended, thereby reducing invalid images. At the same time, under the premise that the remote window is not closed (i.e., it is not necessary to end the recording of the remote window), the current screen of the remote window is captured when the remote window is not a minimized window, so that the information of the current screen of the remote window can be obtained, thereby ensuring the integrity of the first target image obtained, and avoiding capturing the image of the current screen of the remote window when the remote window is minimized, thereby further reducing invalid images.

[0069] The above remote window can be closed by a second function for closing the remote window. For example, the second function is a vncsession_handle_new_window function, so that the remote window is closed based on the vncsession_handle_new_window function.

[0070] In the second function, an end function for closing the recording function can also be embedded, so as to close the recording function when necessary. The end function can be a record_stop_fun function.

[0071] The above determination of whether the remote window satisfies the second screenshot condition may be performed through the third function and the fourth function.

[0072] The third function may be an IsWindow function. The IsWindow function may be used to determine whether a given window handle identifies an existing window. The judgment of the IsWindow function is a pre-judgment to determine whether there is a remote window that has been created.

[0073] The fourth function may be an IsIconic function. The IsIconic function may be used to determine whether a given window is a minimized window, so that after determining that there is a currently created remote window through the IsWindow function (i.e., the third function), the IsIconic function (i.e., the fourth function) may be used to determine whether the remote window is a minimized window, and further determine whether the remote window satisfies the second screenshot condition, so as to perform subsequent corresponding steps.

[0074] Optionally, after determining that the remote window satisfies the first screenshot condition, when continuing to determine whether the remote window satisfies the second screenshot condition, if it is determined that the remote window does not meet the second screenshot condition, that is, the remote window is a minimized window at this time, then the screenshot of the remote window is paused, and after pausing for a preset period of time, continue to determine whether the remote window meets the first screenshot condition, thereby achieving cyclic screenshots of the screen where the remote window is currently located, so as to obtain more screenshot images at different times.

[0075] The above-mentioned pause of the screenshot of the remote window can be realized by calling the pause function. The pause function is used to pause the screenshot of the remote window. For example, the pause function is the record_pause_fun function.

[0076] The above-mentioned pause function may include a preset duration, so that after the pause function is started, after pausing for the preset duration in the pause function, that is, after the current screenshot has been paused for the preset duration in the pause function, it will automatically continue to determine whether the remote window meets the first screenshot condition, thereby realizing the automatic start of the cyclic screenshot process of the remote window.

[0077] The above preset time length may be 100 to 300 milliseconds, but is not limited thereto, and the specific preset time length may be adjusted according to the specific application scenario.

[0078] Optionally, in order to accurately capture the screen where the remote window is currently located to protect user privacy, the screen where the remote window is currently located can be captured according to target requirements to obtain a screenshot image at the current moment.

[0079] The target requirement may include the current location and size of the remote window, and may also include the display where the remote window is currently located, so that the information in the obtained screenshot image meets the target requirement, thereby satisfying the needs of the user.

[0080] In addition, screenshots are taken according to target needs (i.e., the screen where the remote window is currently located is captured), avoiding global recording of the screen (which will cause all content the user sees on the screen to be recorded), and preventing chat push messages, short messages, and various other push and pop-up messages received by the user from being recorded and saved in the recorded video during the recording process, thereby avoiding the leakage of user privacy information and threats to users, and improving the security of user information.

[0081] In addition, compared to full-screen recording, by taking screenshots according to target requirements, the resulting screenshot image can only include the information required by the user, making its file size significantly smaller than the file size of the image in full-screen recording, thereby saving storage space.

[0082] In an embodiment of the present application, the current position and window size of the remote window can be obtained by the fifth function. The fifth function can be a GetWindowRect function. The GetWindowRect function can be used to retrieve the frame and size of the specified window, and the size is provided in screen coordinates relative to the upper left corner of the screen, so that the current position and window size of the remote window can be accurately obtained by the GetWindowRect function (i.e., the fifth function). The position can be represented by coordinates, such as (X, Y), and the window size can be represented by (Width, Height). Wherein, Width represents the width of the remote window, and Height represents the height of the remote window.

[0083] Optionally, the aforementioned interception of the current screen of the remote window to obtain the current image can be captured by screen image capture technology. The screen image capture technology can include application programming interface hooking (English: API Hook) technology and mirror drive technology.

[0084] The above API Hook technology is: using the screenshot function to capture the device-dependent bitmap (DDB) of the screen where the remote window is currently located, and then converting the DDB bitmap into a DIB bitmap to obtain a screenshot image, thereby capturing screen changes. The screenshot function can be an API Bitblt function.

[0085] The above-mentioned mirror driver technology is: directly copy the image data of the intercepted area from the DIB bitmap managed by the mirror driver to the application program, and a screenshot image can be obtained, thereby significantly reducing the time consumption of a screenshot. In the process of intercepting the current screen of the remote window through the mirror driver technology, the image data of the intercepted area is directly copied from the DIB bitmap managed by the mirror driver to the application program, that is, the image data of the intercepted area is directly copied from the DIB bitmap managed by the mirror driver to the image generation process. The creation of the image generation process can be created after the remote window is successfully created and the recording function is turned on. Then in the image generation process, the current screen of the remote window is intercepted to obtain the screenshot image at the current moment.

[0086] In the embodiment of the present application, whether to use API Hook technology or mirror drive technology to capture the current screen of the remote window can be determined in the following manner.

[0087] Specifically, firstly, it is determined whether a mirror driver is installed in the terminal, which is the controlled terminal of the screen where the remote window is currently located.

[0088] If it is determined that the mirror driver has been installed in the terminal, the mirror driver technology is used to capture the current screen of the remote window.

[0089] If it is determined that the image driver is not installed in the terminal, the API Hook technology is used to capture the current screen of the remote window.

[0090] The above judgment on whether the mirror driver is installed in the terminal can be judged by the sixth function and the seventh function in the Windows API, so as to accurately judge whether the mirror driver is installed in the terminal based on the sixth function and the seventh function. The sixth function can be the CreateToolhelp32Snapshot function, and the seventh function can be the Module32First function, so as to accurately judge whether the mirror driver is installed in the terminal through the CreateToolhelp32Snapshot function and the Module32First function.

[0091] In the above manner, when the mirror driver is installed in the terminal, the mirror driver technology is preferentially used to capture the screen, which can significantly reduce the time consumption of a screenshot.

[0092] S102: Convert the screenshot image at the current moment into a first target image of a target image type.

[0093] The file size of the first target image is smaller than the file size of the screenshot image, so that the file size of the image can be effectively reduced by converting the screenshot image into the first target image, thereby saving storage space.

[0094] Since images in BMP format use lossless compression, each pixel in the image is accurately saved, which makes the file size of the screenshot image larger; while images in JPG format use a lossy compression algorithm, which can reduce the file size of the screenshot image according to the set compression quality.

[0095] Therefore, by converting the image type, the image in BMP format (ie, the screenshot image) is converted into an image in JPEG format (ie, the first target image), which effectively reduces the file size of the image and saves storage space.

[0096] In addition, the above-mentioned conversion of the screenshot image at the current moment into the first target image having the target image type is also image compression.

[0097] Optionally, after converting the screenshot image into the first target image, the preset time length can be paused and the aforementioned step S101 can be continued until the remote window fails to meet the first screenshot condition, so as to achieve cyclic screenshots of the screen where the remote window is currently located at different times, so as to obtain multiple screenshot images at different times, thereby obtaining multiple target images at different times.

[0098] S103, obtaining a recorded video of the remote window based on a synthesis of the first target image at the current moment and the second target image and / or the third target image.

[0099] After the first target image is obtained in step S102, the first target image at the current moment is merged with N (where N is a positive integer) second target images obtained before the current moment, or the first target image at the current moment is synthesized with M (where M is a positive integer) third target images obtained after the current moment, or the first target image at the current moment is merged with N second target images and M third target images to generate a recorded video of the remote window. Among them, N and M can be different or the same, which needs to be determined according to the specific segmentation time.

[0100] In the embodiment of the present application, the first target image, the second target image, and the third target image may all be images in the Joint Photographic Experts Group (English: Joint Photographic Experts Group, abbreviated as JPEG) format.

[0101] The method for acquiring the second target image and the third target image is the same as the method for acquiring the first target image, and will not be described in detail here.

[0102] However, it should be noted that the first target image, the second target image, and the third target image can all be acquired by precise capture (i.e., obtaining screenshot images according to target requirements), or by partially precise capture, or by non-static capture. They can be flexibly adjusted according to specific application scenarios so that the recorded video can better meet user needs.

[0103] The recorded video may be a video in the format of a multi-functional player (English: Moving Picture Experts Group 4, abbreviated as MP4). Since the MP4 format uses an efficient video compression algorithm, the obtained recorded video can maintain a relatively high video quality while reducing the size of the video file. In addition, the recorded video may be saved in the form of a video file.

[0104] In order to avoid the invalidation of all screen images of the captured remote window due to failures during the recording process, in the process of synthesizing the first target image at the current moment with the second target image obtained before the current moment and / or the third target image obtained after the current moment into a recorded video, a method of synthesizing the segmented video in time periods can be adopted to generate a recorded video of the remote window.

[0105] Specifically, the first target image at the current moment is synthesized with the second target image and / or the third target image into a segmented video with a segmented duration through video synthesis software, and then when the remote window does not meet the first screenshot condition, multiple segmented videos containing target images at different moments are merged into a recorded video of the remote window, thereby avoiding the failure during the recording process that causes all screenshot images of the remote window to become invalid. Even if a failure suddenly occurs during the recording of the remote window, some information before the failure is still saved, so that the recording of the remote window is fault-tolerant.

[0106] That is to say, when the duration of the currently synthesized segmented video is less than the segmentation duration, the first target image at the current moment is added to the currently synthesized segmented video through the video synthesis software to obtain a segmented video with the segmentation duration. Then the segmented video at this time at least includes the first target image at the current moment and the second target image before the current moment.

[0107] When the duration of the currently synthesized segmented video is greater than or equal to the segmentation duration, based on the first target image and the third target image at the current moment, the next segmented video with the segmentation duration is resynthesized through the video synthesis software.

[0108] Then, when the remote window does not meet the first screenshot condition, that is, when the remote window is closed, multiple segmented videos are merged to obtain a recorded video of the remote window, wherein different segmented videos contain target images at different times.

[0109] The above segmented video may also be a video in MP4 format, and may also be saved in the form of a video file. The segmented video may also be saved in the form of a temporary video file.

[0110] The video synthesis software can be a software that is cut out for the special content in the original video synthesis software. The special content can include: a non-target encoder, a non-target decoder and a network communication module in the original video synthesis software. Thus, by cutting out the special content, the efficiency of the video synthesis process is greatly improved.

[0111] The target encoder may be a video encoder, such as an h264 encoder.

[0112] Since the screenshot image is an image in BMP format and the first target image is an image in JPEG format, the target decoders are BMP decoder and JPEG decoder. Therefore, the non-target decoder may be a decoder other than BMP decoder and JPEG decoder.

[0113] Exemplarily, the original video synthesis software is ffmpeg. Then, the special content is tailored for ffmpeg. Specifically, the h265 encoding library, audio encoding library, and image encoding library in ffmpeg are removed, and only the required h264 encoding library is retained, and then the h264 encoding library is recompiled to provide a faster and more compressed encoding solution. At the same time, the decoders except BMP and JPEG decoders in ffmpeg are disabled, and all network communication modules in ffmpeg are disabled to provide a more efficient video synthesis solution.

[0114] In addition, the above-mentioned merging process of recorded videos and the synthesis process of segmented videos can be performed in the video file process.

[0115] In an embodiment of the present application, the target image (such as the first target image) can be transmitted from the image generation process to the video file process by means of a named pipe, so that the transmission process of the target image is more accurate and efficient, so that in the video file process, through the video synthesis software, multiple target images (such as the first target image, the second target image, and the third target image) can be accurately and efficiently synthesized into a segmented video.

[0116] Furthermore, after obtaining the recorded video of the remote window, the recording of the remote window can be ended. Furthermore, the obtained recorded video can be sent to the user.

[0117] In the embodiment of the present application, the recording function can be turned off by the aforementioned end function, thereby ending the recording of the remote window.

[0118] In addition, after synthesizing multiple segmented videos to obtain the recorded video of the remote window, the temporary video files of the original multiple segmented videos can be deleted to reduce storage pressure, thereby further saving storage space and avoiding problems caused by leakage of temporary video files.

[0119] In summary, the screen recording method proposed in the present application implants the screen recording function in the remote control software (such as TightVNC), solving the problem that the remote control software (such as TightVNC) does not support synchronous screen recording during remote control. Specifically, the screen recording method proposed in the present application is applied to the remote control software, responds to the creation of the remote window of the remote control software, captures the current screen of the remote window, obtains the screenshot image at the current moment, and then converts the screenshot image into a first target image with the target image type, and then based on the merger of the first target image at the current moment with the second target image and / or the third target image, obtains the recorded video of the remote window, so that the remote control software has the recording function during the remote control process. In addition, through the conversion of the screenshot image and the first target image, the file size of the image is reduced, saving storage space.

[0120] At the same time, the current screen of the remote window is captured according to the target requirements, thereby realizing accurate screen recording of the remote window and solving the user privacy problem of full-screen recording. In addition, by synthesizing and segmenting the video, image compression (i.e., the conversion of the screenshot image and the first target image), and judging the first screenshot condition and the second screenshot condition by time period, the occupation of the video file storage capacity is optimized, and the problems of the fault tolerance and excessive space occupation of the screen recording of remote control software are solved. As a result, the screen recording method provided in the embodiment of the present application solves the problems of the fault tolerance and excessive space occupation of the screen recording of remote control software, and solves the user privacy problem of full-screen recording, compared with the screen recording for personal computers (English: Personal Computer, abbreviated as PC) desktops on the market.

[0121] The technical solution of this application is further explained below in conjunction with a specific application process.

[0122] like Figure 2The figure shows a processing diagram of the screen recording method. First, download the open source version code 2.8.27-gpl of TightVNC from the official website of TightVNC, and compile it with VS2019. Then, add the record_start_fun function (i.e., start function), record_stop_fun function (i.e., end function), and record_pause_fun function (i.e., pause function) to the viewer-core class or RemoteViewerCore class of the TightVNC kernel code module, and encapsulate the image capture, image compression, and video synthesis logic in the screen recording component. Then, implant the screen recording component into TightVNC, so that TightVNC also has the recording function during the remote control process.

[0123] In addition, when creating a remote window of TightVNC through the vncsession_handle_new_window function, the record_start_fun function is implanted in the vncsession_handle_new_window function; when closing the TightVNC remote window through the handle_vncviver_window_close function, the record_stop_fun function is implanted; and a window detection program is embedded in the TightVNC kernel, so that when the window detection program detects that the remote window is minimized, the record_pause_fun function is called to pause the screen recording. The record_start_fun function, record_stop_fun function, record_pause_fun function and window detection program can also be configured in the screen recording component, so that TightVNC can achieve better recording capabilities during remote control through the screen recording component.

[0124] Furthermore, the specific process of enabling TightVNC to have a screen recording function (i.e., the specific process of obtaining a recorded video during remote control by using a screen recording component embedded in TightVNC) is as follows:

[0125] In the start module, the screen recording component creates a remote window instance of TightVNC and receives a successful creation event. It then calls the ffmpeg process (i.e., the video file process) through command line parameters to create an image generation process. The ffmpeg used in the ffmpeg process is the ffmpeg that has been trimmed for special content. Then, it is determined whether the image generation process is successfully created. If successful, the segmented video is generated in the loop check module. If unsuccessful, the remote window recording is exited.

[0126] In the loop checking module, the screen recording component determines whether it is necessary to end the recording of the remote window according to whether the remote window is closed. In other words, it determines whether the remote window meets the first screenshot condition. If the remote window is not closed, it is not necessary to end the recording of the remote window, that is, the first screenshot condition is met. If the remote window is closed, it is necessary to end the recording of the remote window, that is, the first screenshot condition is not met.

[0127] After determining that the recording of the remote window does not need to be ended (ie, after determining that the remote window meets the first screenshot condition), it is determined whether the current remote window is minimized (ie, whether the remote window meets the second screenshot condition).

[0128] If the remote window is not a minimized window (i.e., the remote window meets the second screenshot condition), a precise screenshot is performed, i.e., according to the target requirements, the current screen of the remote window is captured to obtain the screenshot image at the current moment. Then, image compression is performed, i.e., the screenshot image in BMP format is converted into a first target image in JPEG format. The first target image is then transmitted to the ffmpeg process through a named pipe, so that the first target image is synthesized into a segmented video by ffmpeg in the ffmpeg process.

[0129] In the process of synthesizing the first target image into a segmented video by ffmpeg, if the duration of the segmented video currently being synthesized is less than the segmentation duration, then continue to synthesize the complete segmented video based on the first target image at the current moment in the segmented video being synthesized. If the duration of the segmented video currently being synthesized is greater than or equal to the segmentation duration, then resynthesize the next segmented video with the segmentation duration based on the first target image at the current moment.

[0130] In addition, after converting the screenshot image at the current moment into the first target image in JPEG format, it is also possible to pause and wait for about 100-300ms, and then determine whether to end the recording of the remote window according to whether the remote window is closed.

[0131] If the remote window is a minimized window (ie, the remote window does not meet the second screenshot condition), the waiting is paused for about 100-300ms, and then whether the remote window needs to be terminated is determined based on whether the remote window is closed.

[0132] After determining that the recording of the remote window needs to be ended (i.e., after determining that the remote window does not meet the first screenshot condition), in the video synthesis module, the synthesized multiple segmented videos are synthesized into one video, which is the recorded video of the remote window, and the recording of the remote window is ended.

[0133] Based on the same inventive concept, the present application also provides a screen recording device in an embodiment, which is applied to remote control software, such as Figure 3The figure is a schematic diagram of the structure of a screen recording device provided by the present application, the device comprising:

[0134] The capture module 301 is used to capture the current screen of the remote window in response to the creation of the remote window of the remote control software, and obtain the screenshot image at the current moment;

[0135] A conversion module 302, configured to convert the screenshot image at the current moment into a first target image having a target image type; wherein the file size of the first target image is smaller than the file size of the screenshot image;

[0136] The processing module 303 is used to obtain the recorded video of the remote window based on the synthesis of the first target image at the current moment with the second target image and / or the third target image; wherein the second target image is the target image before the current moment, and the third target image is the target image after the current moment.

[0137] In a possible implementation, the capture module 301 is specifically used to determine whether the remote window satisfies a first screenshot condition; wherein the first screenshot condition is that the remote window is not closed; if so, determine whether the remote window satisfies a second screenshot condition, and if it is determined that the remote window satisfies the second screenshot condition, capture the screen where the remote window is currently located to obtain the screenshot image at the current moment; wherein the second screenshot condition is that the remote window is not a minimized window.

[0138] In a possible implementation, the capture module 301 is specifically configured to pause the screenshot of the remote window if it is determined that the remote window does not meet the second screenshot condition, and continue to determine whether the remote window meets the first screenshot condition after pausing for a preset period of time.

[0139] In a possible implementation, the capture module 301 is specifically configured to capture the screen where the remote window is currently located according to target requirements to obtain the screenshot image at the current moment; wherein the target requirements include the current position and window size of the remote window.

[0140] In a possible implementation, the capture module 301 is specifically used to determine whether a mirror driver is installed in the terminal; if so, the screen where the remote window is currently located is captured through the mirror driver technology to obtain the screenshot image at the current moment; if not, the screen where the remote window is currently located is captured through the application hook API Hook technology to obtain the screenshot image at the current moment.

[0141] In one possible implementation, the processing module 303 is specifically used to use video synthesis software to synthesize the first target image at the current moment, the second target image and / or the third target image into a segmented video with a segmented duration; in response to the remote window not satisfying the first screenshot condition, merge multiple segmented videos containing target images at different moments into the recorded video of the remote window; wherein the first screenshot condition is that the remote window is not closed.

[0142] In a possible implementation, the video synthesis software is: software that is tailored for special content in the original video synthesis software; wherein the special content includes a non-target encoder, a non-target decoder and a network communication module in the original video synthesis software.

[0143] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. The electronic device can realize the function of the aforementioned screen recording device. Figure 4 , the electronic equipment includes:

[0144] At least one processor 401, and a memory 402 connected to the at least one processor 401. The specific connection medium between the processor 401 and the memory 402 is not limited in the embodiment of the present application. Figure 4 In the example, the processor 401 and the memory 402 are connected via the bus 400. The bus 400 is Figure 4 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 400 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 401 can also be called a controller, and there is no limitation on the name.

[0145] In the embodiment of the present application, the memory 402 stores instructions that can be executed by at least one processor 401. The at least one processor 401 can execute the screen recording method discussed above by executing the instructions stored in the memory 402. The processor 401 can implement Figure 3 The functions of each module in the device shown.

[0146] Among them, the processor 401 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 402 and calling the data stored in the memory 402, the various functions of the device and process data, the device can be monitored as a whole.

[0147] In one possible design, the processor 401 may include one or more processing units, and the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 401. In some embodiments, the processor 401 and the memory 402 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0148] Processor 401 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the screen recording method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0149] The memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 402 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (English: Random Access Memory, abbreviated as RAM), a static random access memory (English: Static Random Access Memory, abbreviated as SRAM), a programmable read-only memory (English: Programmable Read Only Memory, abbreviated as PROM), a read-only memory (English: Read Only Memory, abbreviated as ROM), an electrically erasable programmable read-only memory (English: Electrically Erasable Programmable Read-Only Memory, abbreviated as EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 402 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0150] By programming the processor 401, the code corresponding to the screen recording method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 1 The steps of the screen recording method of the embodiment shown are as follows: How to design and program the processor 401 is a technique known to those skilled in the art and will not be described in detail here.

[0151] Based on the same inventive concept, an embodiment of the present application further provides a storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the screen recording method discussed above.

[0152] In some possible implementations, various aspects of the screen recording method provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a device, the program code is used to enable the control device to execute the steps of the screen recording method according to various exemplary implementations of the present application described above in this specification.

[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0157] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A screen recording method, applied to remote control software, characterized in that: include: In response to the creation of the remote window of the remote control software, capturing the screen where the remote window is currently located to obtain a screenshot image at the current moment; Converting the screenshot image at the current moment into a first target image having a target image type; wherein the file size of the first target image is smaller than the file size of the screenshot image; Based on the synthesis of the first target image at the current moment with the second target image and / or the third target image, the recorded video of the remote window is obtained; wherein the second target image is the target image before the current moment, and the third target image is the target image after the current moment.

2. The method according to claim 1, characterized in that The capturing of the current screen of the remote window to obtain the current screenshot image includes: Determine whether the remote window meets a first screenshot condition; wherein the first screenshot condition is that the remote window is not closed; If so, determine whether the remote window meets the second screenshot condition. If it is determined that the remote window meets the second screenshot condition, capture the screen where the remote window is currently located to obtain the screenshot image at the current moment; wherein, the second screenshot condition is that the remote window is not a minimized window.

3. The method according to claim 2, characterized in that After determining whether the remote window satisfies the second screenshot condition, the method further includes: If it is determined that the remote window does not meet the second screenshot condition, the screenshot of the remote window is paused, and after pausing for a preset time, it is continued to determine whether the remote window meets the first screenshot condition.

4. The method according to claim 1, characterized in that The capturing of the current screen of the remote window to obtain the current screenshot image includes: According to the target requirement, the screen where the remote window is currently located is captured to obtain the screenshot image at the current moment; wherein the target requirement includes the current position and window size of the remote window.

5. The method according to any one of claims 1 to 4, characterized in that: The capturing of the current screen of the remote window to obtain the current screenshot image includes: Determine whether the image driver is installed in the terminal; If yes, then the screen where the remote window is currently located is captured by using the mirror drive technology to obtain the screenshot image at the current moment; If not, the screen where the remote window is currently located is captured through the application program hook API Hook technology to obtain the screenshot image at the current moment.

6. The method according to claim 1, characterized in that The step of obtaining the recorded video of the remote window by synthesizing the first target image at the current moment with the second target image and / or the third target image includes: Using video synthesis software, synthesize the first target image at the current moment, the second target image and / or the third target image into a segmented video with segmented duration; In response to the remote window not satisfying a first screenshot condition, a plurality of segmented videos containing target images at different moments are merged into the recorded video of the remote window; wherein the first screenshot condition is that the remote window is not closed.

7. The method according to claim 6, characterized in that The video synthesis software is software that is tailored for the special content in the original video synthesis software; wherein the special content includes the non-target encoder, non-target decoder and network communication module in the original video synthesis software.

8. A screen recording device, applied to remote control software, characterized in that: include: A capture module, for responding to the creation of a remote window of the remote control software, capturing the screen where the remote window is currently located, and obtaining a screenshot image at the current moment; A conversion module, configured to convert the screenshot image at the current moment into a first target image having a target image type; wherein the file size of the first target image is smaller than the file size of the screenshot image; A processing module is used to obtain the recorded video of the remote window based on the synthesis of the first target image at the current moment with the second target image and / or the third target image; wherein the second target image is the target image before the current moment, and the third target image is the target image after the current moment.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps of any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images