A method, device and medium for transmitting video frames across processes based on Android

Abstract

The present invention discloses a method for transmitting video frames across processes based on Android, comprising: step 1, the server defines a management interface for managing video controls; step 2, the server receives the address and size of the video control sent by the client through the management interface; step 3, the server obtains the video frame and adjusts the size of the video frame according to the size of the video control; step 4, the server converts the current format of the video frame into the format required for rendering; step 5, the server transmits the converted video frame to the address of the video control and then renders it, so that the client plays the video according to the address of the video control. The present invention also provides an electronic device and a computer-readable storage medium, which solves the drawbacks of limited transmission data size and low transmission efficiency, and at the same time, avoids the trouble of the client developing rendering logic by itself.

Description

Technical Field

[0001] The present invention relates to the field of Android technology, and in particular to a method, device and medium for transmitting video frames across processes based on Android. Background Art

[0002] The Android operating system is mainly used in mobile devices, such as smartphones and platform computers. The number of devices using this system has reached 1 billion worldwide. The Android system has become the most popular system in the field of smartphones. The development of Android applications has also flourished. In order to save development costs, more and more software companies have packaged commonly used functions into an SDK (software development kit) application so that other applications can reuse these functions by calling the SDK.

[0003] There are many SDK applications related to cameras on the market, such as face recognition SDK and code scanning SDK. However, these SDKs often solidify the preview interface of camera shots, or only support a small number of interface changes. This makes it difficult for customers to customize and develop the preview screen they want. To achieve a customized preview interface, the customer application must be able to receive video frames shot by the camera SDK application, which requires cross-process transmission of video frames.

[0004] In order to allow the client process to play the video frames of the server process, the traditional cross-process video frame transmission solutions are based on the following idea: the video frames of the server are transmitted to the client, and then the client plays it by itself, renders the video frames by itself, and customizes the preview interface. There are generally the following solutions with this idea:

[0005] 1. If Figure 1 As shown, an AIDL (Android Interface Definition Language, i.e., Android interface definition language, which can realize cross-process access) interface is defined, the client passes the listener to the server, and then the server returns the video frame to the client through the listener;

[0006] 2. If Figure 2 As shown, the client and the server establish a connection through a Socket or a pipe, and then the server transmits the video frame to the client.

[0007] There are drawbacks to both of these solutions:

[0008] 1. First, in solution 1, although the performance of the AIDL interface can transmit data, the size of the transmitted data is limited. Therefore, when transmitting video frames across processes, the video frames generated by some cameras are too large, resulting in the failure of video frame transmission;

[0009] 2. Then, Solution 2 requires at least two copies when using Socket or pipe communication. The transmission efficiency is too low, and the client may experience freezes when rendering video frames, resulting in poor performance.

[0010] 3. In addition, both of the above solutions have a problem, that is, the client needs to write its own rendering logic after receiving the video frame, and different cameras output different video frame formats, which requires different rendering logic. This means that the client must develop different rendering logic on different camera hardware, and it is impossible to achieve "develop once, run everywhere".

[0011] Therefore, it is urgent to develop a new solution for cross-process transmission of video frames to solve the drawbacks of "limited transmission data size" and "low transmission efficiency", while avoiding the trouble of the client developing its own rendering logic. Summary of the invention

[0012] In view of this, the purpose of the present invention is to propose a method for transmitting video frames across processes based on Android. The present invention solves the drawbacks of limited transmission data size and low transmission efficiency. At the same time, it also avoids the trouble of the client developing its own rendering logic, so that the client can define the preview interface of the video frame at will, and "develop once, run everywhere".

[0013] In order to achieve the above technical objectives, the technical solution adopted by the present invention is: a method for transmitting video frames based on Android cross-process, applied to the server, comprises the following steps:

[0014] Step 1, the server defines a management interface for managing video controls;

[0015] Step 2: The server receives the address and size of the video control sent by the client through the management interface;

[0016] Step 3, the server obtains the video frame and adjusts the size of the video frame according to the size of the video control;

[0017] Step 4: The server converts the current format of the video frame into a format required for rendering;

[0018] Step 5: The server transmits the converted video frame to the address of the video control and then renders it, so that the client plays the video according to the address of the video control.

[0019] Furthermore, the step 2 specifically includes:

[0020] The server receives the address of the video control sent by the client through the creation callback of the management interface, wherein the address of the video control is called back to the client by the Android system after the client creates a video control; at the same time, the server receives the address, original width and original height of the video control sent by the client through the size change callback of the management interface; wherein the address, original width and original height of the video control are called back to the client by the Android system after the video control is created and the size of the video control increases from nothing to something.

[0021] Furthermore, after step 1, the server further defines a data storage object; after step 2, the server further includes:

[0022] Step 21: The server stores the obtained address, original width and original height of the video control in a data storage object.

[0023] Furthermore, after step 21, the following steps are further included:

[0024] Step 22. When the size of the video control changes, the server receives the address, new width and new height of the video control sent by the client through the size change callback of the management interface, and updates them to the corresponding positions in the data storage object; wherein the address, new width and new height of the video control are called back to the client by the Android system after the client changes the size of the video control.

[0025] Furthermore, after step 1, the server defines a drawing method, and step 4 specifically includes:

[0026] Each time the server captures a video frame, the server calls the drawing method once to convert the data of the video frame from a byte array into an integer array representing pixels.

[0027] Furthermore, after step 4, the following steps are further included:

[0028] Step 41: When the server calls the drawing method for the first time, it creates a bitmap and stores all converted pixels into the bitmap.

[0029] Furthermore, the step 5 is specifically as follows:

[0030] The server transmits the bitmap to the address of the video control and renders the bitmap, so that the client plays the video according to the address of the video control.

[0031] Furthermore, after step 5, the following steps are further included:

[0032] Step 6. The server receives the address of the video control sent by the client through the destruction callback of the management interface, learns that the video control corresponding to the address of the video control has been destroyed, and stops rendering. At the same time, the data in the data storage object is cleared; wherein the address of the video control is called back to the client by the Android system after the client destroys the video control.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for transmitting video frames across processes based on Android is implemented as described above.

[0034] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for transmitting video frames across processes based on Android.

[0035] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0036] Although the present invention also uses the AIDL interface (the management interface of the video control is one of them), the key is not to use the AIDL interface to directly transfer the video frame, but to transfer the address of the video control. If only the "address" is transferred, there is no need to worry about the problem of the data being too large and exceeding the limit. In addition, the method in which the server directly fills and renders the video frame to the address of the video control does not need to transmit the video frame to the user space of the client, which not only avoids the inefficient copying of the video frame and improves the transmission efficiency; but also encapsulates the cumbersome rendering logic by the server itself, and the client does not need to develop the rendering logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a structural block diagram of AIDL cross-process transmission of video frames in the prior art.

[0039] Figure 2 It is a structural block diagram of Socket cross-process transmission of video frames in the prior art.

[0040] Figure 3 It is a structural block diagram of the transmission video frame in the present invention.

[0041] Figure 4It is a flow chart of a method for transmitting video frames across processes based on Android provided by an embodiment of the present invention.

[0042] Figure 5 It is a schematic diagram of an electronic device provided by an embodiment of the present invention.

[0043] Figure 6 It is a schematic diagram of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0045] The present invention provides a method for transmitting video frames across processes based on Android, which can solve the drawbacks of "limitation on transmission data size" and "low transmission efficiency", and at the same time, avoids the trouble of the client developing rendering logic by itself.

[0046] First, let me introduce the idea of ​​the technical solution of the present invention. The client provides

[0047] SurfaceView (SurfaceView is the video control for playing video frames), or GLSurfaceView or TextureView), then get the SurfaceHolder of the SurfaceView, and then get the Surface through the SurfaceHolder. The Surface is associated with the address of the video control that is actually rendered (or the storage area of ​​the video frame). As long as the "address" is passed to the server process, the server process can fill the video frame into the address of the video control of the client process for rendering. In fact, the ability to support the above operations is also due to the fact that AndroidSDK itself supports cross-process transmission of Surface, and Surface implements the Parcelable interface.

[0048] See also Figure 3 and Figure 4 The present invention provides a method for transmitting video frames across processes based on Android, which is applied to a server and includes the following steps:

[0049] Step 1: The server defines a management interface for managing video controls (such as SurfaceViewMananger interface, GLSurfaceViewMananger interface or

[0050] TextureViewMananger interface), the role of the management interface is to receive the address and size of the video control sent by the client, such as small-capacity data, and does not transmit large-capacity data, solving the problem of "transmission data size limit";

[0051] The management interface defines a creation callback, a size change callback, and a destruction callback. The creation callback is used to receive a surface sent when the client creates a video control. The surface is associated with the address of the video control. The client notifies the server through the creation callback that the video control corresponding to the address of the video control has been created; the size change callback is used to receive a surface and the size of the video control sent when the client changes the size of the video control. The surface is associated with the address of the video control. The client notifies the server through the size change callback that the width and height of the video control corresponding to the address of the video control have changed and the size needs to be updated; the size change callback requires at least one callback. The first size change callback refers to the callback of the size of the video control from "none" to "yes" (the original size after the video control is created); the second and subsequent size change callbacks refer to the callback of the size of the video control from (original size) to (new size); the destruction callback is used to receive a surface sent when the client destroys the video control. The surface is associated with the address of the video control. The client notifies the server through the destruction callback that the video control corresponding to the address of the video control has been destroyed.

[0052] Step 2: The server receives the address and size of the video control sent by the client through the management interface;

[0053] Step 3, the server obtains the video frame and adjusts the size of the video frame according to the size of the video control;

[0054] Step 4: The server converts the current format of the video frame into a format required for rendering;

[0055] Step 5, the server transmits the converted video frame to the address of the video control and then renders it, so that the client plays the video according to the address of the video control. After the client calls the server to render the video frame, the server calls the drawing method every time it captures a video frame, and passes the video frame as a parameter to the address of the video control. Since the entire transmission process is completed in the kernel space and does not pass through the user space of the process, the transmission efficiency is relatively high, solving the problem of "low transmission efficiency". In addition, the server renders the video frame into the client's video control, without the need for client rendering, avoiding the trouble of the client developing its own rendering logic.

[0056] In this embodiment, the step 2 specifically includes:

[0057] The server receives the address of the video control sent by the client through the creation callback of the management interface, wherein the address of the video control is called back to the client by the Android system after the client creates a video control, that is, the client calls the creation callback of the management interface to send a surface (associated with the address of the video control) to inform the server that the video control corresponding to the address of the video control has been created. After receiving the surface, the server can read the address of the video control and know that the video control has been created, and the video frame will be rendered through this address later; at the same time, the server receives the address, original width and original height of the video control sent by the client through the size change callback of the management interface; wherein the address, original width and original height of the video control are called back to the client by the Android system after the video control is created and the size of the video control increases from nothing to something. That is, the client calls the size change callback of the management interface to send surface (the address associated with the video control), width (the original width of the video control) and height (the original height of the video control) to inform the server of the original size of the video control corresponding to the address of the video control. When the server receives surface, width and height, it knows the original width and height of the newly created video control. During the transmission of the video frame, the video frame can be directly transmitted from the user space of the server to the address of the video control without passing through the user space of the client, which has high transmission efficiency.

[0058] The step 1 also includes: binding the server to the client, obtaining the client's calling authority for the management interface, and enabling the client and the server to exchange data through the management interface.

[0059] In this embodiment, after step 1, the server further includes: when implementing the management interface, the server further defines a data storage object (MySurface) for storing the address, width and height of the video control; after step 2, the server further includes:

[0060] Step 21, the server stores the obtained address, original width and original height of the video control in a data storage object. The purpose of storing the data is to facilitate later search;

[0061] MySurface.surface; / / Store the surface sent by the client, which is associated with the address of the video control. Whether to store the surface can be determined according to needs;

[0062] MySurface.width; / / Store the width of the video control;

[0063] MySurface.height; / / Stores the height of the video control.

[0064] In this embodiment, after step 21, the following steps are further included:

[0065] Step 22. When the size of the video control changes, the server receives the address, new width and new height of the video control sent by the client through the size change callback of the management interface, and updates them to the corresponding position in the data storage object; wherein, the address, new width and new height of the video control are called back to the client by the Android system after the client changes the size of the video control. That is, the client calls the size change callback of the management interface to send surface (associated with the address of the video control), width (new width of the video control) and height (new height of the video control) to inform the server that the width and height of the preview control have changed. After receiving the surface, width and height, the server knows that the video control corresponding to the address of the video control has a new width and height, and needs to adjust the size of the video frame before rendering.

[0066] In this embodiment, after step 1, the server further includes: defining a drawing method responsible for rendering video frames; and step 4 specifically includes:

[0067] Each time the server captures a video frame, the server calls the drawing method once to convert the data of the video frame from a byte array into an integer array representing pixels;

[0068] SurfaceViewMananger.draw(byte[]frame); / / Parameter frame represents the video frame data;

[0069] int[] pixels = convertByteToPixels(frame); / / Convert the video frame data from the byte array frame into an integer array pixels representing pixels (change it into a format that can be used for rendering).

[0070] Here, the server needs to perform different conversions on the frame according to the different formats of the video frame in order to obtain the pixel array and complete the subsequent rendering.

[0071] In this embodiment, after step 4, the following steps are further included:

[0072] Step 41, when the server calls the drawing method for the first time, it creates a bitmap as a container of pixels and stores all converted pixels into the bitmap; the bitmap object is only created when the drawing method is called for the first time, and the created bitmap object is reused when the drawing command is called later, which can avoid memory overflow of the server. This is because if a bitmap object is created each time the drawing method is called, when the number of video frames is large, many bitmap objects will inevitably be created, which will cause memory occupation of the server and cause memory overflow.

[0073] In this embodiment, the step 5 is specifically as follows:

[0074] The server transmits the bitmap to the address of the video control and renders the bitmap, so that the client plays the video according to the address of the video control.

[0075] In this embodiment, after step 5, the following steps are further included:

[0076] Step 6. The server receives the address of the video control sent by the client through the destruction callback of the management interface, learns that the video control corresponding to the address of the video control has been destroyed, and stops rendering. At the same time, the data in the data storage object is cleared; wherein, the address of the video control is called back to the client by the Android system after the client destroys the video control. That is, when the preview page is exited or the client does not need to play the video, the client destroys the video control, and the client calls the destruction callback of the management interface to send a surface to inform the server that the video control corresponding to the address of the video control has been destroyed. After receiving the surface, the server knows that the video control corresponding to the address of the video control has been destroyed and can stop rendering.

[0077] In this embodiment, the video control is SurfaceView, GLSurfaceView or TextureView. Taking SurfaceView as the video control and SurfaceViewMananger interface as the management interface of the video control as an example, three levels of nesting are defined on the Android system, namely SurfaceView→SurfaceHolder→Surface (associated with the address of SurfaceView). The client calls the SurfaceViewMananger interface in three life cycles: when the video control is "created", when the size of the video control "changes", and when the video control is "destroyed".

[0078] public void surfaceCreated(SurfaceHolder holder)

[0079] The above is the "creation time" of the SurfaceView life cycle. After the SurfaceView is created, the Android system will call back the SurfaceHolder corresponding to the client's SurfaceView through this life cycle, and the Surface can be obtained through the SurfaceHolder. The client calls the SurfaceViewMananger.surfaceCreated method in this life cycle method, and sends the Surface to the server through the SurfaceViewMananger interface. The server obtains the address of the SurfaceView from the Surface and learns that the client has created a SurfaceView. Call as follows:

[0080] manager.surfaceCreated(holder.getSurface())

[0081] public void surfaceChanged(SurfaceHolder holder,int format,int width,int height)

[0082] The above is the "change" of the life cycle of SurfaceView. When the size of SurfaceView changes, the Android system will use this life cycle to call back the SurfaceHolder corresponding to the client SurfaceView and the new width and height of SurfaceView. It is worth noting that after the Android system calls back that SurfaceView is created, it will immediately call back that the size of SurfaceView has changed, informing the client of the width and height of the newly created SurfaceView, that is, the Android system will call back the client at least once that the size of SurfaceView has changed. And this first size change callback corresponds to a process in which the width and height of SurfaceView change from "nothing" to "something" after SurfaceView is created. The second size change callback is performed when the size of the video control changes. The client calls the SurfaceViewMananger.surfaceChanged method in this life cycle method, and sends Surface, Width, and Height to the server through the SurfaceViewMananger interface. The server obtains the address of SurfaceView from Surface. The server knows that the client changes the width of the SurfaceView corresponding to the address of SurfaceView to W1 and the height to H1 based on the address, Width, and Height of SurfaceView. Call as follows: manager.surfaceChanged(holder.getSurface(),width,height)

[0083] public void surfaceDestroyed(SurfaceHolder holder)

[0084] The above is the "destruction" of the SurfaceView life cycle. When SurfaceView is destroyed, the Android system will call back the SurfaceHolder corresponding to the client's SurfaceView through this life cycle. The Surface can be obtained through the SurfaceHolder. The client calls the SurfaceViewMananger.surfaceDestroyed method in this life cycle method, and sends the Surface to the server through the SurfaceViewMananger interface. The server obtains the address of the SurfaceView from the Surface. The server knows that the SurfaceView corresponding to the address of the SurfaceView has been destroyed based on the address of the SurfaceView. Call as follows: manager.surfaceDestroyed(holder.getSurface())

[0085] like Figure 5 As shown, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the program.

[0086] like Figure 6 As shown, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above method is implemented.

[0087] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0088] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0089] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for transmitting video frames across processes based on Android, applied to a server, characterized in that: The steps include: Step 1, the server defines a management interface for managing video controls, the management interface is responsible for receiving the address and size of the video control sent by the client; the management interface defines a creation callback, a size change callback and a destruction callback, the creation callback is used to receive the surface sent when the client creates the video control, the surface is associated with the address of the video control, and the client informs the server through the creation callback that the video control corresponding to the address of the video control has been created; the size change callback is used to receive the surface and the size of the video control sent when the client changes the size of the video control, the surface is associated with the address of the video control, and the client informs the server through the size change callback The width and height of the video control corresponding to the address of the video control on the server side have changed, and the size needs to be updated; the size change callback requires at least one callback, the first size change callback refers to the callback from the size of the video control from nothing to something; the second and subsequent size change callbacks refer to the callback from the original size to the new size of the video control; the destruction callback is used to receive the surface sent when the client destroys the video control, the surface is associated with the address of the video control, and the client informs the server side through the destruction callback that the video control corresponding to the address of the video control has been destroyed; the server side is bound to the client side, and the client side obtains the calling authority of the management interface, so as to realize data interaction between the client side and the server side through the management interface; Step 2: The server receives the address and size of the video control sent by the client through the management interface; Step 3, the server obtains the video frame and adjusts the size of the video frame according to the size of the video control; Step 4, the server defines a drawing method, and the server converts the current format of the video frame into a format required for rendering; The server creates a bitmap when calling the drawing method for the first time. Each time the server captures a video frame, the server calls the drawing method once to convert the data of the video frame from a byte array to an integer array representing pixels, and stores all the converted pixels into the bitmap. Step 5: The server transmits the converted video frame to the address of the video control and then renders it, so that the client plays the video according to the address of the video control.

2. A method for transmitting video frames across processes based on Android as claimed in claim 1, characterized in that: The step 2 specifically includes: The server receives the address of the video control sent by the client through the creation callback of the management interface, wherein the address of the video control is called back to the client by the Android system after the client creates a video control; at the same time, the server receives the address, original width and original height of the video control sent by the client through the size change callback of the management interface; wherein the address, original width and original height of the video control are called back to the client by the Android system after the video control is created and the size of the video control increases from nothing to something.

3. A method for transmitting video frames across processes based on Android as claimed in claim 2, characterized in that: After step 1, the server further includes: defining a data storage object; after step 2, the server further includes: Step 21: The server stores the obtained address, original width and original height of the video control in a data storage object.

4. A method for transmitting video frames across processes based on Android as claimed in claim 3, characterized in that: After step 21, the following steps are further included: Step 22. When the size of the video control changes, the server receives the address, new width and new height of the video control sent by the client through the size change callback of the management interface, and updates them to the corresponding positions in the data storage object; wherein the address, new width and new height of the video control are called back to the client by the Android system after the client changes the size of the video control.

5. The method for transmitting video frames across processes based on Android as claimed in claim 1, characterized in that: The step 5 is specifically as follows: The server transmits the bitmap to the address of the video control and renders the bitmap, so that the client plays the video according to the address of the video control.

6. The method for transmitting video frames across processes based on Android as claimed in claim 3, characterized in that: After step 5, the following steps are also included: Step 6. The server receives the address of the video control sent by the client through the destruction callback of the management interface, learns that the video control corresponding to the address of the video control has been destroyed, and stops rendering. At the same time, the data in the data storage object is cleared; wherein the address of the video control is called back to the client by the Android system after the client destroys the video control.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for transmitting video frames across processes based on Android is implemented as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a method for transmitting video frames across processes based on Android is implemented as described in any one of claims 1 to 6.

Images