Screen sharing method and device

Abstract

The invention discloses a screen sharing method and device, and the method comprises the steps: collecting a video stream presented in a screen of a mobile terminal through an extension program, so as to obtain original video stream data; the extension program performs YUV decoding on original image information included in a video frame in the original video stream data to obtain intermediate data; the extension program splits the intermediate data to obtain frame header data, basic information, a Y data block, a U data block and a V data block, and converts the frame header data, the basic information, the Y data block, the U data block and the V data block into binary data; the expansion program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block and the binarized V data block to the main application; the main application determines that the transmission of the video frame is ended based on the received binarized frame header data; and the main application splices the received binarized Y data block, the binarized U data block and the binarized V data block to obtain a complete video frame. The memory space occupied by the mobile terminal by screen sharing can be reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a screen sharing method and device. Background Art

[0002] Screen sharing technology in real-time audio and video calls enables participants to share the content on their device screens. Screen sharing technology has become an indispensable part of modern communication and collaboration tools, greatly expanding the way people interact and share information. It is widely used in various scenarios, including remote work, online meetings, distance education, technical support, virtual meetings, online games, etc.

[0003] At present, when a mobile terminal is performing screen sharing, it consumes too much memory on the mobile terminal itself. In particular, the memory space occupied by Apple phones when performing screen sharing is forced to be less than 50M. Once the memory space occupied by screen sharing exceeds 50M, the extension program for collecting screen streams will crash. The current solution to similar problems in the industry is to limit the video quality to no more than 720P, or the video frame rate to within 30. These solutions sacrifice video quality and are only a temporary solution. The problem of excessive memory space occupied when processing screen video streams still exists. Summary of the invention

[0004] The present application provides a screen sharing method and device, which can reduce the memory space occupied by screen sharing on a mobile terminal. The technical solution is as follows.

[0005] In a first aspect, a screen sharing method is provided, which is applied to a mobile terminal, wherein the mobile terminal runs a main application and an extension program, and the method includes:

[0006] The extension program collects the video stream presented on the screen of the mobile terminal to obtain original video stream data;

[0007] The extension program performs YUV decoding on the original image information included in the video frame in the original video stream data to obtain intermediate data;

[0008] The expansion program disassembles the intermediate data to obtain frame header data, basic information, Y data block, U data block and V data block, and converts the frame header data, the basic information, the Y data block, the U data block and the V data block into binary data respectively;

[0009] The extension program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block to the main application respectively;

[0010] The main application determines, based on the received binarized frame header data, that the video frame transmission ends;

[0011] The main application splices the received binarized Y data block, the binarized U data block, and the binarized V data block to obtain the complete video frame.

[0012] In some implementations, the extension program disassembles the intermediate data to obtain frame header data, basic information, a Y data block, a U data block, and a V data block, and converts the frame header data, the basic information, the Y data block, the U data block, and the V data block into binary data, including:

[0013] The extension program obtains basic information, a Y data block, a U data block, and a V data block from the intermediate data;

[0014] The expansion program allocates memory space for the basic information, the Y data block, the U data block, and the V data block, and initializes the memory block;

[0015] The expansion program copies the basic information, the Y data block, the U data block, and the V data block to the initial position of the memory block, and updates the initial position of the memory block;

[0016] The extension program encapsulates the basic information, the Y data block, the U data block and the V data block into a binary data block and marks an index;

[0017] After the extension program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block to the main application respectively, the method further includes:

[0018] The expansion program releases the memory space occupied by the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block.

[0019] In some embodiments, the method further comprises:

[0020] The extension program and the main application establish a connection via a socket;

[0021] The extension program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block to the main application respectively, including:

[0022] The extension program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block and the binarized V data block to the main application through a socket.

[0023] In some implementations, before the main application splices the received binarized Y data block, the binarized U data block, and the binarized V data block, the method further includes:

[0024] The main application crops, compresses and rotates the received binarized Y data block, the binarized U data block and the binarized V data block respectively.

[0025] In some implementations, the intermediate data is data in I420 format, and the extension program performs YUV decoding on the original image information included in the video frame in the original video stream data, including:

[0026] The extension program obtains member variables contained in the original video stream data;

[0027] The extension program obtains a pointer to the YUV data from the member variable, and accesses the YUV data contained in the member variable based on the pointer;

[0028] If the pixel format type of the member variable is ARGB format, the extension program converts the pixel format type of the member variable from ARGB format to I420 format;

[0029] If the pixel format type of the member variable is NV12 format, the extension program converts the pixel format type of the member variable from NV12 format to I420 format.

[0030] In some embodiments, the frame header data includes the total data length of the video frame and the sequence number of the video frame, and the total data length represents the sum of the length of the binarized basic information, the length of the binarized Y data block, the length of the binarized U data block, and the length of the binarized V data block.

[0031] In some implementations, the main application determines the end of the video frame transmission based on the received binarized frame header data, including:

[0032] The main application determines whether the received data block is frame header data;

[0033] If the received data block is frame header data, the main application obtains the total data length of the video frame based on the frame header data, and initializes the current data length to 0, where the current data length is used to record the data length of the currently accumulated spliced ​​video frame;

[0034] If the received data block is not frame header data, the main application adds the length of the received data block to the current data length. If the accumulated current data length is greater than or equal to the total data length of the video frame, the main application determines that the video frame transmission is completed.

[0035] In some implementations, after the main application determines whether the received data block is frame header data, the method further includes:

[0036] If the received data block is frame header data, the main application clears a buffer, where the buffer is used to splice the video frames;

[0037] The main application adds the length of the received data block to the current data length, and the method further includes:

[0038] If the accumulated current data length is less than the total data length of the video frame, the main application determines that the video frame has not been transmitted, and the main application adds the received binarized Y data block, the binarized U data block, and the binarized V data block to the buffer in order;

[0039] After obtaining the complete video frame, the method further includes:

[0040] The main application outputs the video frame and clears the buffer.

[0041] In a second aspect, a screen sharing device is provided, characterized in that it includes a main application and an extension program;

[0042] The extension program is used to collect the video stream presented on the screen of the mobile terminal to obtain original video stream data; perform YUV decoding on the original image information included in the video frame in the original video stream data to obtain intermediate data; disassemble the intermediate data to obtain frame header data, basic information, Y data block, U data block and V data block, and convert the frame header data, the basic information, the Y data block, the U data block and the V data block into binary data respectively; send the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block and the binarized V data block to the main application respectively;

[0043] The main application is used to determine the end of the video frame transmission based on the received binarized frame header data; and to splice the received binarized Y data block, the binarized U data block and the binarized V data block to obtain the complete video frame.

[0044] In some embodiments, the extension program is used to obtain basic information, Y data block, U data block and V data block from the intermediate data; allocate memory space for the basic information, Y data block, U data block and V data block, and initialize the memory block; copy the basic information, Y data block, U data block and V data block to the initial position of the memory block, and update the initial position of the memory block; encapsulate the basic information, Y data block, U data block and V data block into binary data blocks, and mark indexes; release the memory space occupied by the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block and the binarized V data block.

[0045] In some embodiments, the extension program is also used to establish a connection with the main application through a socket; and send the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block to the main application through the socket.

[0046] In some implementations, the main application is further used to crop, compress and rotate the received binarized Y data block, the binarized U data block and the binarized V data block, respectively.

[0047] In some embodiments, the intermediate data is data in I420 format, and the extension program is further used to obtain member variables contained in the original video stream data; obtain a pointer to YUV data from the member variable, and access the YUV data contained in the member variable based on the pointer; if the pixel format type of the member variable is ARGB format, convert the pixel format type of the member variable from ARGB format to I420 format; if the pixel format type of the member variable is NV92 format, convert the pixel format type of the member variable from NV92 format to I420 format.

[0048] In some embodiments, the frame header data includes the total data length of the video frame and the sequence number of the video frame, and the total data length represents the sum of the length of the binarized basic information, the length of the binarized Y data block, the length of the binarized U data block, and the length of the binarized V data block.

[0049] In some embodiments, the main application is also used to determine whether the received data block is frame header data; if the received data block is frame header data, the total data length of the video frame is obtained based on the frame header data, and the current data length is initialized to 0, and the current data length is used to record the data length of the currently accumulated spliced ​​video frame; if the received data block is not frame header data, the length of the received data block is accumulated to the current data length, if the accumulated current data length is greater than or equal to the total data length of the video frame, the main application determines that the video frame transmission is completed.

[0050] In some embodiments, if the received data block is frame header data, the main application is also used to clear a buffer, which is used to splice the video frame; if the accumulated current data length is less than the total data length of the video frame, it is determined that the video frame has not yet been transmitted, and the received binarized Y data block, the binarized U data block, and the binarized V data block are added to the buffer in sequence; the video frame is output and the buffer is cleared.

[0051] It can be seen that the embodiments of the present application have the following beneficial effects:

[0052] The screen sharing method provided in the embodiment of the present application reduces the memory space occupied by the screen sharing process on the mobile terminal based on the frame disassembly method. Specifically, considering that if the extension program packages all the information contained in the data structure of the video frame into a binary stream, and then sends the binary stream to the main application, it is easy for the CPU to not have time to release these temporary variables, resulting in a sudden increase in the occupied memory space. The embodiment of the present application disassembles the intermediate data into basic information, Y data blocks, U data blocks, and V data blocks, and processes them separately, and sends them to the main application separately. On the one hand, it reduces the amount of data processing and avoids a short-term surge in memory space. On the other hand, because the smaller the data, the faster the processing, which also speeds up the information processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of screen sharing provided by an embodiment of the present application;

[0054] Figure 2 is a schematic diagram of memory optimization provided by an embodiment of the present application;

[0055] Figure 3 This is a flowchart of screen sharing frame processing and return provided by an embodiment of the present application;

[0056] Figure 4 is a flowchart of a YUV decoding process provided by an embodiment of the present application;

[0057] Figure 5 It is a flow chart of a process of disassembling a video frame and converting it into binary format provided by an embodiment of the present application;

[0058] Figure 6 is a flow chart of a frame splicing process provided by an embodiment of the present application;

[0059] Figure 7 is a schematic diagram of a real-time audio and video call between two devices provided in an embodiment of the present application;

[0060] Figure 8 is a schematic diagram of memory space consumption provided by an embodiment of the present application;

[0061] Fig. 9 It is a schematic diagram of memory occupancy when frame disassembly is not performed, provided in an embodiment of the present application;

[0062] Fig.10 This is a schematic diagram of memory occupancy when performing frame disassembly provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0064] The following is an explanation of some terminology concepts involved in the embodiments of the present application.

[0065] Real-time audio and video calls: The video stream is collected through the camera, and the audio stream is collected through the microphone; the client conducts media coordination and network negotiation with the signaling server to establish a connection; the collected audio and video streams are then encoded, transmitted, decoded, and rendered to achieve real-time audio and video calls.

[0066] Screen sharing: is a function that can display the screen image of a device on another device in real time, which can help users collaborate remotely, demonstrate content, etc. The screen sharing involved in this article is to replace the camera video stream with the screen video stream during a real-time audio and video call, and then continue to perform encoding, decoding, and transmission. That is, the video stream object collected and transmitted is changed, and the encoding, decoding, network transmission, rendering, etc. remain the same.

[0067] YUV: A video encoding method. "Y" represents brightness (Luminance or Luma), which is the grayscale value. "U" and "V" represent chrominance (Chrominance or Chroma). It is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0068] The following is an example of an application scenario of the embodiment of the present application.

[0069] Screen sharing technology in real-time audio and video has become an indispensable part of modern communication and collaboration tools, greatly expanding the ways of interaction and information sharing between people. It is widely used in various scenarios, including remote work, online meetings, distance education, technical support, virtual meetings, online games, etc.

[0070] With the development of technology and its widespread application, the technical requirements and standards for screen sharing are constantly improving. One of the most prominent problems is that when a mobile terminal is sharing its screen, the memory consumption of the mobile terminal itself is too large. In particular, when an Apple mobile phone is sharing its screen, the memory space required for screen sharing is limited to less than 50M. Once the memory space required for screen sharing exceeds 50M, the extension program for collecting screen streams will crash.

[0071] The industry's solution to similar problems is to limit the video quality to no more than 720P, or to limit the video frame rate to 30. These solutions sacrifice video quality, only treat the symptoms but not the root cause, and do not fundamentally solve the problem of large memory usage when processing screen video streams. At the same time, the memory space occupied by screen sharing applications is accumulated and consumed for a long time, which may lead to memory overflow risk.

[0072] In view of this, an embodiment of the present application provides a mobile terminal screen sharing memory optimization method based on frame disassembly, which performs lightweight decoding, frame disassembly, transmission of screen video streams between the main application (App) and the extension program, frame splicing and other processing on the screen video frames of the mobile terminal (such as an Apple mobile phone), thereby reducing memory usage from all aspects, alleviating the device operation burden, improving application operation efficiency, improving video fluency and quality, and avoiding application crashes and flashbacks.

[0073] The main process of the embodiment of the present application includes the following S1 to S7.

[0074] S1. The extension starts collecting video streams.

[0075] S2. The extended program collects and obtains the original video stream data (hereinafter referred to as CMSampleBufferRef).

[0076] S3. The extension program performs YUV light decoding on the original image information included in the video frame in the original video stream data to obtain intermediate data, which is, for example, data in I420 format.

[0077] S4. The extension program disassembles the intermediate data to obtain frame header data, basic information, Y data block, U data block and V data block, and converts the frame header data, the basic information, the Y data block, the U data block and the V data block into binary data (hereinafter referred to as NSData) to obtain the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block and the binarized V data block, thereby reducing data traffic.

[0078] S5, the extension program sends each data obtained in S4 to the main application through socket (real-time complex data transmission method), thereby speeding up processing efficiency;

[0079] S6. The main application determines the end of the video frame transmission based on the received binarized frame header data, and correctly splices the video frames.

[0080] S7. The main application sends the video frame to the remote terminal.

[0081] The embodiment of the present application is mainly used in the screen sharing process. The screen sharing process mainly includes: the extension program collects the screen video stream, returns the screen video stream to the main application, and then the main application transmits the screen video stream to another device to complete the screen sharing in the real-time audio and video call. For the specific process of screen sharing, please refer to the attached Figure 1 .

[0082] Attached Figure 1 The premise of the screen sharing process shown is that a real-time audio and video call channel is established between the main application of device A and the main application of device B, and an extension program is established for the main application of device B to capture the screen. The screen sharing process mainly includes the following steps 1 to 8.

[0083] Step 1: Device A transmits the video stream captured by the camera to device B through the audio and video call channel.

[0084] Step 2: Device B's main app binds the extension.

[0085] Step 3: Device B's main app triggers the extension's screen sharing.

[0086] Step 4: The extension program collector of device B collects the screen video stream.

[0087] Step 5: The extension program of device B processes the screen video stream, including decoding, conversion and other operations.

[0088] Step 6: The extension program of device B returns the processed screen video stream data to the main application of device B.

[0089] Step 7: The main application of device B transmits the processed screen stream data to device A through the audio and video call channel, and the screen sharing is completed.

[0090] Step 8: When it is time to end, screen sharing ends and the extension notifies the main app to clean up resources.

[0091] During the entire interaction process between the main application and the extension program, only steps 4, 5, and 6 involve the processing of the video stream, and video stream processing is a memory-consuming operation. Therefore, the memory optimization solution of the embodiment of the present application starts from steps 4, 5, and 6, and proposes lightweight decoding, frame disassembly, a specified transmission method (i.e., socket) between the main application and the extension program, frame splicing, etc. to deal with the memory optimization problem.

[0092] The memory optimization solution provided in the embodiment of the present application is further described below.

[0093] According to the attached Figure 1 The screen sharing process analysis shown requires optimization of both video processing and video return.

[0094] Compare the video processing memory usage to a channel, see the attached Figure 2 , attached Figure 2 The idea of ​​memory optimization is shown. The memory space of a mobile terminal (such as a mobile phone) is limited. The limited memory space is equivalent to the fixed width of the channel. Whether the channel is sufficient depends on two aspects, the processing speed and the amount of data. The embodiment of the present application reduces the amount of data by splitting YUV data, reducing temporary variables, and using an intermediate data format with a high compression ratio; the embodiment of the present application improves the processing speed by a high-speed transmission method, a small amount of data to be processed, and multi-tasking. The embodiment of the present application achieves the effect of memory optimization from both the reduction and acceleration aspects through a series of operations such as lightweight decoding of I420 format data, frame disassembly, specified transmission method (ie, socket) between the main application and the extension program, and frame splicing.

[0095] For screen sharing frame processing and return flow, please refer to the attached Figure 3 , and then a detailed analysis is carried out.

[0096] S1. The extension program starts to collect the video stream presented on the screen of the mobile terminal. For example, an Apple mobile phone collects the screen video stream through the system extension program.

[0097] S2. The extended program collector collects and obtains original video stream data, which will be referred to as CMSampleBufferRef data hereinafter.

[0098] The original video stream data takes up almost no memory space. However, the data format cannot be converted without decoding the original video stream data. As a result, the extension program cannot send the original video stream data directly to the main application. Instead, it needs decoding, conversion, communication and other operations. However, these operations will consume too much memory space, so we will optimize these operations next.

[0099] S3. The extended program performs YUV decoding on the original image information included in the video frame in the original video stream data to obtain intermediate data.

[0100] In some embodiments, the intermediate data is data in I420 format. Specifically, in order to achieve efficient and lightweight decoding of the original video stream data CMSampleBufferRef, the original video stream data cannot be directly converted into a bitmap. Because each video frame consumes a lot of memory in the case of a large screen resolution, an intermediate data is required, and the intermediate data meets the following conditions.

[0101] Condition 1: Image information can be obtained from the original video stream data CMSampleBufferRef, but it is lighter than imageData itself;

[0102] Condition 2: After being transferred from the extension to the main application, the intermediate data can be restored to image information, and the image information can be rotated, cropped, compressed, and other operations can be performed.

[0103] Therefore, data in I420 format is used as intermediate data.

[0104] In some embodiments, when the extension program performs YUV decoding, a lightweight decoding method is adopted. Specifically, in the prior art, after performing YUV decoding, the extension program will further crop, compress and rotate the decoded data block, resulting in excessive memory space required for the extension program. In view of this, when the extension program performs YUV decoding in the embodiment of the present application, only the necessary YUV decoding function is performed, and other non-essential operations outside the YUV decoding function, such as cropping, compression and rotation, are processed in the main application later. Since the extension program does not need to perform cropping, compression and rotation operations, lightweight decoding is achieved, thereby minimizing any processing in the extension program and reducing memory usage.

[0105] Please refer to the attached Figure 4 , attached Figure 4 A YUV decoding process provided in an embodiment of the present application is shown, and the YUV decoding process specifically includes the following steps.

[0106] S301: The extension program receives input original video stream data CMSampleBufferRef.

[0107] S302: The extension program obtains the member variable CVPixelBufferRef contained in the original video stream data.

[0108] S303: The extension program obtains a pointer to the YUV data from the member variable CVPixelBufferRef, and accesses the YUV data contained in the member variable based on the pointer.

[0109] S304. The extension program determines the pixel format type of the member variable CVPixelBufferRef.

[0110] If the pixel format type of the member variable CVPixelBufferRef is ARGB format, the extension program executes S305. If the pixel format type of the member variable CVPixelBufferRef is NV12 format, the extension program executes S306.

[0111] S305: The extension program converts the pixel format type of the member variable from ARGB format to I420 format. The process of converting the ARGB format to the I420 format is a color space conversion, which involves the rearrangement of color channels.

[0112] The data in ARGB format is arranged as follows: A1 R1 G1 B1 A2 R2 G2 B2...

[0113] The data in I420 format is arranged as follows:

[0114] Y plane: Y1 Y2 Y3 Y4...

[0115] U plane: U1 U2 U3 U4...

[0116] V plane: V1 V2 V3 V4...

[0117] The extension uses the following formula to calculate the components of the Y plane, the U plane, and the V plane in the ARGB format data, and copies the calculated components of the Y plane, the U plane, and the V plane to the Y plane, U plane, and V plane of the I420 format, respectively, to achieve color space conversion.

[0118] Y=0.299*R+0.587*G+0.114*B

[0119] U=-0.14713*R-0.288862*G+0.436*B

[0120] V=0.615*R-0.51498*G-0.10001*B

[0121] S306. The extension program converts the pixel format type of the member variable from NV12 format to I420 format.

[0122] The process of converting NV12 format to I420 format is color space conversion, which involves the rearrangement of YUV color planes.

[0123] The data in NV12 format is arranged as follows:

[0124] Y plane: Y1 Y2 Y3 Y4...

[0125] UV plane: U1 V1 U2 V2...

[0126] The data in I420 format is arranged as follows:

[0127] Y plane: Y1 Y2 Y3 Y4...

[0128] U plane: U1 U2 U3 U4...

[0129] V plane: V1 V2 V3 V4...

[0130] The Y plane data, U plane data and V plane data extracted from the NV12 format are copied to the Y plane, U plane and V plane of the I420 format respectively to achieve conversion. After the extension program executes S305 and S306, it finally outputs the data in the I420 format.

[0131] S4. The extension program disassembles the intermediate data to obtain frame header data, basic information, Y data block, U data block and V data block, and converts the frame header data, basic information, Y data block, U data block and V data block into binary data blocks NSData respectively, thereby reducing data traffic.

[0132] Usually, the way to transmit intermediate data between the extension program and the main application is that the extension program packages all the information contained in the data structure of the intermediate data into an NSData binary data stream, and then the extension program sends the binary data stream to the main application. However, on a mobile terminal with too high resolution and too large size, when the information itself is very rich in the picture, when the CPU does not have time to release these temporary variables, it is easy to cause the occupied memory space to surge, and the memory space eventually occupied exceeds 50M, causing the screen recording process to crash. Therefore, the embodiment of the present application disassembles the intermediate data into basic information, Y data blocks, U data blocks and V data blocks, and after processing the basic information, Y data blocks, U data blocks and V data blocks respectively, the processed basic information, Y data blocks, U data blocks and V data blocks are sent to the main application respectively, which reduces the amount of data processing on the one hand, avoids the short-term surge in the occupied memory space, and on the other hand, the smaller the data, the faster the processing, thereby also speeding up the information processing speed.

[0133] Refer to the attached video frame disassembly and binary conversion process Figure 5 The frame disassembly process specifically includes the following steps S401 to S408.

[0134] S401: The extension program receives input intermediate data, where the intermediate data is data in I420 format.

[0135] S402: The extension program obtains basic information in the intermediate data in the I420 format, such as width, height, and word length (i420 Data Length).

[0136] S403, the expansion program reallocates memory and initializes memory blocks for the data blocks of the Y plane, the data blocks of the U plane, the data blocks of the V plane, and the basic information each time.

[0137] S404, the expansion program copies the data block of the Y plane, the data block of the U plane, the data block of the V plane and the basic information to the starting position of the memory block, and then updates the starting position of the memory block.

[0138] S405, the extension program encapsulates the data block of the Y plane, the data block of the U plane, the data block of the V plane and the basic information into a binary data block (NSData data block), and marks the index. The index is initially 0, and the index increases in the repeated steps.

[0139] S406: The extension program outputs the binary data block of the basic information to the main application, and releases the memory space occupied by the binary data block of the basic information.

[0140] S407. The extension program disassembles the three plane data of Y plane, U plane and V plane in the intermediate data in I420 format, and repeats steps S403 to S405 respectively, that is, reallocating memory, copying data to memory, encapsulating binary data blocks, and releasing memory, and outputs the binarized Y data block (Y NSData data block), the binarized U data block (U NSData data block), and the binarized V data block (V NSData data block), respectively.

[0141] S408: The extension program outputs the frame header data to the main application. The frame header data is, for example, metadata. The frame header data includes the total length of the data (data_len) and the serial number (serial_id).

[0142] The total data length (data_len) represents the sum of the length of the binarized basic information, the length of the binarized Y data block, the length of the binarized U data block, and the length of the binarized V data block. The total data length is determined by referring to the following formula.

[0143] Total data length data_len = basic information.length + Y data block.length + U data block.length + V data block length

[0144] The frame header data is used to later splice complete video frames.

[0145] Each time the extension disassembles and converts a piece of data, it sends it to the main application, thereby reducing the amount of data transmitted each time. At the same time, temporary variables are released as soon as possible to keep the memory value at an average level.

[0146] S5. The extension program sends each data to the main application separately to speed up the processing efficiency.

[0147] Regarding the communication methods for sending data from the extension to the main application, the process communication methods include low-frequency data transmission (URLScheme), simple value transmission (NSUserDefault), real-time complex data transmission (Socket transmission), etc. Among them, the Socket method supports complex data, is highly efficient, and can effectively speed up the transmission speed between process communications, thereby reducing the continuous high consumption of memory, so the Socket transmission method is selected.

[0148] Based on this, in some implementations of the present application, the extension program and the main application establish a connection through a Socket. For each piece of data including frame header data, basic information, Y data block, U data block, and V data block, every time the extension program disassembles a piece of data from the intermediate data and converts the data into a binary data block, it sends a piece of data to the main application through the Socket.

[0149] S6. The main application determines the end of a video frame information based on the frame header data and correctly splices the video frames.

[0150] In the previous steps, the extension program has disassembled the screen video frame and transmitted it to the main application separately. When the main application receives the fragmented video frame, it needs to splice the fragments into a complete video frame. The frame splicing process is shown in the attached Figure 6 .

[0151] S601 : The main application receives data blocks of the disassembled video frame from the extension program.

[0152] S602: The main application determines whether the received data block is frame header data, and if the received data block is frame header data, the main application executes S603. If the received data block is not frame header data, the main application executes S605.

[0153] S603: The main application clears a buffer, where the buffer is used to start assembling a new frame.

[0154] S604, the main application parses the frame header data, obtains the total data length targetDataSize, and initializes the current data length currenDataSize = 0. The total data length targetDataSize here is data_len in S408, and currenDataSize is used to calculate the length of the current accumulated spliced ​​data.

[0155] S605. The main application adds the data length of the received data block to the current data length currenDataSize according to the following formula.

[0156] currentDataSize=currentDataSize+data.length

[0157] S606. The main application determines whether the current data length (currentDataSize) is greater than or equal to the total data length (targetDataSize). If the current data length is less than the total data length, the main application executes S607. If the current data length is greater than or equal to the total data length, the main application executes S609.

[0158] S607: The main application determines that the spliced ​​data blocks are not enough for one video frame. In other words, the video frame has not yet been completely transmitted.

[0159] S608, the main application adds the received binarized Y data block, the binarized U data block and the binarized V data block to the buffer in order, and executes S601, S602, S605, S607 and S608 operations in a loop for the received binarized Y data block, the binarized U data block and the binarized V data block to splice most of the video frame data.

[0160] S609: The main application determines that the spliced ​​data blocks and the newly received data blocks can satisfy a video frame after splicing. In other words, the transmission of the video frame is completed. The main application executes S608 to add the received data blocks to the buffer in order, that is, to add the binarized V data block after the binarized Y data block and the binarized U data block, so as to splice and obtain a complete video frame.

[0161] S610: The main application outputs a complete video frame and clears the buffer.

[0162] In some implementations, the main application also performs the process of cropping, compressing, and rotating. For example, after the main application obtains a complete video frame through splicing, the main application performs operations such as cropping, compressing, and rotating on the spliced ​​video frame as needed. After performing the operations such as cropping, compressing, and rotating, the main application of device A sends the operated video frame to device B.

[0163] S7. The main application receives the screen video frame.

[0164] The above embodiment provides a mobile terminal screen sharing memory optimization method based on screen video frame disassembly, which reduces memory usage from various aspects, lightens the operating burden of the mobile terminal, and reduces the risk of memory overflow and 50M memory limit by lightweight decoding and frame disassembly of screen video frames, using sockets for communication between the main application and the extension program, frame splicing and other processing.

[0165] Based on the premise of real-time audio and video calls, the captured screen video stream replaces the original camera video stream, as shown in the attached Figure 7 , a real-time audio and video call is made between the two devices. Device A initiates screen sharing, and the screen of device A is displayed in the audio and video call interface of device B.

[0166] After multiple experiments and comparisons, we compared the memory consumption of screen sharing before and after optimization operations such as lightweight decoding, frame disassembly, transmission, and frame splicing. We also compared the memory consumption in three different scenarios: not enabled, just enabled, and enabled for 2 minutes. We found that the memory space consumed was greatly reduced after the optimization operation, and the memory space consumed was stable at around a certain value and would not increase. The overall situation is as follows: Figure 8 As shown in the figure. The memory consumption diagram during the experiment is captured, please refer to the attached Fig. 9 and attached Fig.10 .

[0167] Attached Fig. 9 This is the memory usage of the main application when screen sharing is enabled without optimization operations such as frame disassembly. The memory space occupied when screen sharing is not enabled is 55MB, the memory space occupied when screen sharing is just enabled is 215MB, and the memory space occupied after screen sharing lasts for 2 minutes is 2.8GB*1024=2928.6MB. The memory space occupied is very large, and the memory space consumed increases over time.

[0168] Attached Fig.10 This is the memory usage of the main application after optimization operations such as frame disassembly are performed, and the memory space occupied by screen sharing is turned on. When screen sharing is not turned on, the memory space occupied is 55.5MB, which is almost the same as the memory space occupied when no optimization is performed in the same period. When screen sharing is just turned on, the memory space occupied is 63.2MB, which is a year-on-year decrease of 70.6% compared with the memory space occupied when no optimization is performed in the same period; after screen sharing lasts for 2 minutes, the memory space occupied is 62.8MB, which is a year-on-year decrease of 97.8% compared with the memory space occupied when no optimization is performed in the same period. The occupied memory space is not high and remains at a stable value. That is, screen sharing only increases memory consumption by 63-55MB=7~8MB, which is much lower than the limited 50MB requirement, which improves the stability of the screen sharing function and reduces the burden on device operation.

[0169] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0170] A refers to B, which means that A is the same as B or A is a simple variant of B.

[0171] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in accordance with the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0172] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A screen sharing method, characterized in that: Applied to a mobile terminal, the mobile terminal runs a main application and an extension program, the method comprises: The extension program collects the video stream presented on the screen of the mobile terminal to obtain original video stream data; The extension program performs YUV decoding on the original image information included in the video frame in the original video stream data to obtain intermediate data; The expansion program disassembles the intermediate data to obtain frame header data, basic information, Y data block, U data block and V data block, and converts the frame header data, the basic information, the Y data block, the U data block and the V data block into binary data respectively; The extension program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block to the main application respectively; The main application determines, based on the received binarized frame header data, that the video frame transmission ends; The main application splices the received binarized Y data block, the binarized U data block, and the binarized V data block to obtain the complete video frame.

2. The method according to claim 1, characterized in that The expansion program disassembles the intermediate data to obtain frame header data, basic information, Y data block, U data block and V data block, and converts the frame header data, the basic information, the Y data block, the U data block and the V data block into binary data respectively, including: The extension program obtains basic information, a Y data block, a U data block, and a V data block from the intermediate data; The expansion program allocates memory space for the basic information, the Y data block, the U data block, and the V data block, and initializes the memory block; The expansion program copies the basic information, the Y data block, the U data block, and the V data block to the initial position of the memory block, and updates the initial position of the memory block; The extension program encapsulates the basic information, the Y data block, the U data block and the V data block into a binary data block and marks an index; After the extension program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block to the main application respectively, the method further includes: The expansion program releases the memory space occupied by the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block.

3. The method according to claim 1, characterized in that The method further comprises: The extension program and the main application establish a connection via a socket; The extension program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block to the main application respectively, including: The extension program sends the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block and the binarized V data block to the main application through a socket.

4. The method according to claim 1, characterized in that Before the main application splices the received binarized Y data block, the binarized U data block, and the binarized V data block, the method further includes: The main application crops, compresses and rotates the received binarized Y data block, the binarized U data block and the binarized V data block respectively.

5. The method according to claim 1, characterized in that The intermediate data is data in I420 format, and the extension program performs YUV decoding on the original image information included in the video frame in the original video stream data, including: The extension program obtains member variables contained in the original video stream data; The extension program obtains a pointer to the YUV data from the member variable, and accesses the YUV data contained in the member variable based on the pointer; If the pixel format type of the member variable is ARGB format, the extension program converts the pixel format type of the member variable from ARGB format to I420 format; If the pixel format type of the member variable is NV12 format, the extension program converts the pixel format type of the member variable from NV12 format to I420 format.

6. The method according to claim 1, characterized in that The frame header data includes the total data length of the video frame and the serial number of the video frame, and the total data length represents the sum of the length of the binarized basic information, the length of the binarized Y data block, the length of the binarized U data block and the length of the binarized V data block.

7. The method according to claim 6, characterized in that The main application determines, based on the received binarized frame header data, that the video frame transmission ends, including: The main application determines whether the received data block is frame header data; If the received data block is frame header data, the main application obtains the total data length of the video frame based on the frame header data, and initializes the current data length to 0, where the current data length is used to record the data length of the currently accumulated spliced ​​video frame; If the received data block is not frame header data, the main application adds the length of the received data block to the current data length. If the accumulated current data length is greater than or equal to the total data length of the video frame, the main application determines that the video frame transmission is completed.

8. The method according to claim 7, characterized in that After the main application determines whether the received data block is frame header data, the method further includes: If the received data block is frame header data, the main application clears a buffer, where the buffer is used to splice the video frames; The main application adds the length of the received data block to the current data length, and the method further includes: If the accumulated current data length is less than the total data length of the video frame, the main application determines that the video frame has not been transmitted, and the main application adds the received binarized Y data block, the binarized U data block, and the binarized V data block to the buffer in order; After obtaining the complete video frame, the method further includes: The main application outputs the video frame and clears the buffer.

9. A screen sharing device, characterized in that: Including the main application and extensions; The extension program is used to collect the video stream presented on the screen of the mobile terminal to obtain original video stream data; perform YUV decoding on the original image information included in the video frame in the original video stream data to obtain intermediate data; disassemble the intermediate data to obtain frame header data, basic information, Y data block, U data block and V data block, and convert the frame header data, the basic information, the Y data block, the U data block and the V data block into binary data respectively; Sending the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block, and the binarized V data block to the main application respectively; The main application is used to determine the end of the video frame transmission based on the received binarized frame header data; and to splice the received binarized Y data block, the binarized U data block and the binarized V data block to obtain the complete video frame.

10. The device according to claim 9, characterized in that The extension program is used to obtain basic information, Y data block, U data block and V data block from the intermediate data; allocate memory space for the basic information, Y data block, U data block and V data block, and initialize the memory block; copy the basic information, Y data block, U data block and V data block to the initial position of the memory block, and update the initial position of the memory block; The basic information, the Y data block, the U data block and the V data block are encapsulated into binary data blocks, and indexes are marked; and the memory space occupied by the binarized frame header data, the binarized basic information, the binarized Y data block, the binarized U data block and the binarized V data block is released.

Images