A multi-screen interactive method based on virtual display technology

Abstract

The present invention relates to a multi-screen interactive method based on virtual display technology, comprising: step S1, starting the Android platform, using the surface Java object as a parameter, applying to create a virtual display; step S2, based on the BufferQueue producer-consumer model of the Android platform, the image display end of the surface C++ of the framework component layer receives the available image frame signal and obtains the virtual address using the slot number as a parameter, after the acquisition is completed, inputs it to the kernel driver layer, and at the same time, waits for receiving the kernel driver layer signal to release the available image frame buffer and the slot number; step S3, the kernel driver layer converts the received virtual address into a physical address, and when the conversion is completed, uses the available image frame buffer physical address as input to use the shared memory mechanism and cross-domain transmission, and when the display of the opposite end is completed, notifies the surface C++ of the framework component layer to release the available image frame buffer and the slot number. The BufferQueue mechanism and virtual display technology already available on the Android platform can be used to avoid creating a customized application layer interface, thereby effectively reducing the CPU occupancy rate.

Description

Technical Field

[0001] The present invention relates to the field of multi-screen interaction technology, and in particular to a multi-screen interaction method based on virtual display technology. Background Art

[0002] With the development of electronic technology, more and more electronic devices have multi-screen interactive functions. Multi-screen interactive technology refers to the transmission of digital multimedia content between different terminal devices through wireless network connections. It can synchronize the display content on different screens and control other terminal devices through one terminal device. Multi-screen interactive methods include: one-to-one or one-to-many active interaction, that is, the user needs to open the content to be displayed on the terminal first, and then connect to other terminals through a specific device. In the above multi-screen interactive method, the user needs to understand the presented content in advance, consider whether it is necessary to connect to other settings, and do some necessary preparations before connecting. However, in this way, multi-screen interaction can only be completed manually, and automatic multi-screen interaction between interconnected terminals cannot be realized, which makes multi-screen interaction not smart enough and reduces the experience of multi-screen interaction.

[0003] The technical solutions of the multi-screen interaction method in the prior art include:

[0004] 1. Framework component layer (JAVA):

[0005] 1.1) Record the main screen and obtain the image frame buffer data;

[0006] 1.2) Get an empty buffer from the driver device node;

[0007] 1.3) Transfer the buffer data in step 1.1 to the buffer data in step 1.2 in byte copy mode;

[0008] 2. Kernel driver layer

[0009] 2.1) Taking the physical address as input, use the shared memory mechanism for cross-domain transmission;

[0010] 2.2) The peer end completes the display and notifies the framework component layer to release the buffer, which is then recycled by the kernel layer;

[0011] Among them, the shortcomings are:

[0012] 1. The application interface is strongly related to the selection of technical solutions and the version of the technical solutions. The degree of customization and privacy are high, which leads to high interface learning costs, high development costs, and high maintenance costs.

[0013] 2. When meeting high-definition image resolution and a frame rate of 15fps, the CPU occupancy rate is higher than 2%, and the RAM occupancy rate is higher than 4 times the RAM occupancy rate of the image resolution, which affects the overall system performance or limits the interactive application effects and application scenarios.

[0014] Chinese patent CN201210194276.9 discloses a multi-screen interactive system and a multi-screen interactive method, including multiple client applications searching for a server running a service program, so that a connection is established between the application of each client and the service program of the server; each client sends a control command to the server; the service program of the server executes corresponding operations according to each control command, and displays the corresponding screen generated by executing each operation on the display screen of the server; the service program of the server returns the screen data of the corresponding client after executing the corresponding operation to each client in turn; and the corresponding screen is displayed on the display screen of each client.

[0015] At present, there are some multi-screen interaction methods based on virtual display technology, but they generally require the creation of new customized application layer interfaces, which have high interface learning costs, high development costs, and high maintenance costs; they also have high CPU / RAM resource occupancy rates and low image resolution and frame rates. Summary of the invention

[0016] To this end, the present invention provides a multi-screen interaction method based on virtual display technology, which can effectively solve the technical problem in the prior art that the existing BufferQueue mechanism and virtual display technology of the Android platform cannot be utilized to avoid creating a customized application layer interface, thereby effectively reducing costs and CPU occupancy.

[0017] To achieve the above object, the present invention provides a multi-screen interaction method based on virtual display technology, comprising:

[0018] Step S1, start the Android platform, use the surface Java object of the framework component layer as a parameter, and apply to create a virtual display;

[0019] Step S2, based on the BufferQueue producer-consumer model of the Android platform, the image display end of the surface C++ framework component layer receives the available image frame signal and uses the available image frame buffer slot number as a parameter to obtain the available image frame buffer virtual address, and after the acquisition is completed, inputs it to the kernel driver layer, and at the same time, waits for receiving the kernel driver layer signal to release the available image frame buffer and the available image frame buffer slot number;

[0020] Step S3, the kernel driver layer converts the received available image frame buffer virtual address into an available image frame buffer physical address. When the conversion is completed, the available image frame buffer physical address is used as input to use a shared memory mechanism and cross-domain transmission. When the display on the other end is completed, the framework component layer surface c++ is notified to release the available image frame buffer and the available image frame buffer slot number.

[0021] Furthermore, in step S1, starting the Android platform and using the surface Java object of the framework component layer as a parameter, applying to create a virtual display specifically includes:

[0022] Step S101, during the boot process of the Android platform, the multi-screen interactive service is automatically started during the boot process;

[0023] Step S102, obtaining the surface C++ object of the framework component layer, and after the acquisition is completed, converting the C++ object into a Java object;

[0024] Step S103, using the converted surface Java object as an input parameter, applying to create a virtual display.

[0025] Furthermore, in step S2, when the image display terminal receives the available image frame signal, the specific operations include:

[0026] Step S201 a, requesting to lock the available image frame buffer slot number;

[0027] Step S202a, synchronizing and locking the available image frame buffer using the available image frame buffer slot number as a parameter;

[0028] Step S203a, using the available image frame buffer slot number as a parameter, obtaining an available image frame buffer virtual address;

[0029] Step S204a, taking the available image frame buffer virtual address as a parameter and inputting it into the kernel driver layer;

[0030] Step S205a, waiting to receive the kernel driver layer signal to release the available image frame buffer and the available image frame buffer slot number.

[0031] Furthermore, in step S3, the specific operations of the kernel driver layer after receiving the available image frame buffer virtual address include:

[0032] Step S301, converting the available image frame buffer virtual address into an available image frame buffer physical address;

[0033] Step S302, using the available image frame buffer physical address as input to use a shared memory mechanism and cross-domain transmission;

[0034] Step S303, after the display on the other end is completed, the framework component layer surface C++ is notified to release the available image frame buffer and the available image frame buffer slot number.

[0035] Furthermore, in step S2, the image display terminal includes the following steps in the software initialization process:

[0036] Step S201 b, create a BufferQueue producer object and a consumer object;

[0037] Step S202b, using the producer object and the consumer object as parameters, creating a framework component layer surface C++ object;

[0038] Step S203b, using the producer object and the display device file descriptor as parameters, applying for an image frame buffer from the display driver.

[0039] Furthermore, the supporting features of the kernel driver layer include providing a display driver device node for a peer display.

[0040] Furthermore, the supporting features of the kernel driver layer also include applying for creating an image frame buffer.

[0041] Furthermore, the steps S1-S5 use the existing virtual display technology of the Android platform to avoid creating a customized application layer interface.

[0042] Furthermore, the cross-domain transmission represents zero-copy transmission of cross-domain image data, which utilizes inter-domain memory sharing technology.

[0043] Compared with the prior art, the beneficial effect of the present invention lies in that the present invention starts the Android platform and uses the surface Java object of the framework component layer as a parameter to apply for the creation of a virtual display. Then, based on the BufferQueue producer-consumer model of the Android platform, the image display end of the surface C++ of the framework component layer receives the available image frame signal and obtains the available image frame buffer virtual address using the available image frame buffer slot number as a parameter. After the acquisition is completed, it is input to the kernel driver layer, and at the same time, the kernel driver layer waits for receiving the kernel driver layer signal to release the available image frame buffer and the available image frame buffer slot number. Finally, the kernel driver layer converts the received available image frame buffer virtual address into the available image frame buffer physical address. When the conversion is completed, the available image frame buffer physical address is used as input to use the shared memory mechanism and cross-domain transmission. When the display of the other end is completed, the framework component layer surface C++ is notified to release the available image frame buffer and the available image frame buffer slot number. This enables the application of clever ideas and innovative solutions, effectively reducing component service development and maintenance costs. Through the officially released Android platform, customization is avoided, effectively reducing application development and maintenance costs. It then inherits the existing mechanisms of the Android platform, implements zero-copy technology for image data, and reduces CPU usage.

[0044] In particular, the present invention uses surface java objects, available image frame buffer slot numbers, available image frame buffer virtual addresses, etc. as parameters. Compared with the prior art, the present invention has better technical parameters, with a CPU occupancy rate of less than 1%, a RAM occupancy rate of no more than 3.3 times the image resolution RAM occupancy number, and a frame rate of 25fps.

[0045] In particular, the present invention utilizes the existing BufferQueue mechanism of the Android platform, can stably and efficiently implement the image frame acquisition function without destroying the original framework, and effectively reduces the application-side development cost and maintenance cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a multi-screen interactive method based on virtual display technology according to an embodiment of the present invention;

[0047] Figure 2 This is a structural framework diagram of a multi-screen interactive device based on virtual display technology according to an embodiment of the present invention;

[0048] Figure 3 This is a flowchart of specific implementation steps of step S1 in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the operation flow when the image display terminal receives an available image frame signal according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the operation flow after the kernel driver layer receives the available image frame buffer virtual address according to an embodiment of the present invention;

[0051] Figure 6 The figure is a flowchart of the software initialization process of the image display terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0054] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0055] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] See also Figure 1 and Figure 2 As shown, Figure 1 is a flow chart of a multi-screen interactive method based on virtual display technology according to an embodiment of the present invention, Figure 2 The present invention provides a multi-screen interaction method based on virtual display technology, including:

[0057] Step S1, start the Android platform, use the surface Java object of the framework component layer as a parameter, and apply to create a virtual display;

[0058] Step S2, based on the BufferQueue producer-consumer model of the Android platform, the image display end of the surface C++ framework component layer receives the available image frame signal and uses the available image frame buffer slot number as a parameter to obtain the available image frame buffer virtual address, and after the acquisition is completed, inputs it to the kernel driver layer, and at the same time, waits for receiving the kernel driver layer signal to release the available image frame buffer and the available image frame buffer slot number;

[0059] Step S3, the kernel driver layer converts the received available image frame buffer virtual address into an available image frame buffer physical address. When the conversion is completed, the available image frame buffer physical address is used as input to use a shared memory mechanism and cross-domain transmission. When the display on the other end is completed, the framework component layer surface c++ is notified to release the available image frame buffer and the available image frame buffer slot number.

[0060] Specifically, the present invention starts the Android platform, uses the surface Java object of the framework component layer as a parameter, applies to create a virtual display, and then, based on the BufferQueue producer-consumer model of the Android platform, the image display end of the surface c++ of the framework component layer receives the available image frame signal and uses the available image frame buffer slot number as a parameter to obtain the available image frame buffer virtual address, after the acquisition is completed, inputs it to the kernel driver layer, and at the same time, waits for the kernel driver layer signal to be received to release the available image frame buffer and the available image frame buffer slot number, and finally, the kernel driver layer converts the received available image frame buffer virtual address into the available image frame buffer physical address, when the conversion is completed, uses the available image frame buffer physical address as input to use the shared memory mechanism and cross-domain transmission, and when the display of the opposite end is completed, notifies the surface c++ of the framework component layer to release the available image frame buffer and the available image frame buffer slot number. Thus, clever ideas and innovative solutions can be applied, effectively reducing the component service development cost and maintenance cost, avoiding the introduction of customization through the officially released Android platform, effectively reducing the application end development cost and maintenance cost, and then inheriting the existing mechanism of the Android platform, realizing the image data zero copy technology, and reducing the CPU occupancy rate.

[0061] Specifically, see Figure 3 As shown, it is a schematic diagram of a specific implementation step flow of step S1 of an embodiment of the present invention. In step S1, starting the Android platform, using the surface Java object of the framework component layer as a parameter, applying to create a virtual display specifically includes:

[0062] Step S101, during the boot process of the Android platform, the multi-screen interactive service is automatically started during the boot process;

[0063] Step S102, obtaining the surface C++ object of the framework component layer, and after the acquisition is completed, converting the C++ object into a Java object;

[0064] Step S103, using the converted surface Java object as an input parameter, applying to create a virtual display.

[0065] Specifically, see Figure 4 As shown, it is a schematic diagram of the operation flow when the image display terminal receives the available image frame signal according to an embodiment of the present invention. In the step S2, when the image display terminal receives the available image frame signal, the specific operations include:

[0066] Step S201 a, requesting to lock the available image frame buffer slot number;

[0067] Step S202a, synchronizing and locking the available image frame buffer using the available image frame buffer slot number as a parameter;

[0068] Step S203a, using the available image frame buffer slot number as a parameter, obtaining an available image frame buffer virtual address;

[0069] Step S204a, taking the available image frame buffer virtual address as a parameter and inputting it into the kernel driver layer;

[0070] Step S205a, waiting to receive the kernel driver layer signal to release the available image frame buffer and the available image frame buffer slot number.

[0071] In this embodiment, the image display end refers to the consumer end, and the opposite is the image drawing end, that is, the producer end.

[0072] Specifically, see Figure 5 As shown, it is a schematic diagram of the operation flow after the kernel driver layer receives the available image frame buffer virtual address according to an embodiment of the present invention. In the step S3, the specific operation of the kernel driver layer after receiving the available image frame buffer virtual address includes:

[0073] Step S301, converting the available image frame buffer virtual address into an available image frame buffer physical address;

[0074] Step S302, using the available image frame buffer physical address as input to use a shared memory mechanism and cross-domain transmission;

[0075] Step S303, after the display on the other end is completed, the framework component layer surface C++ is notified to release the available image frame buffer and the available image frame buffer slot number.

[0076] Specifically, see Figure 6 As shown, it is a schematic diagram of the process of software initialization of the image display terminal according to an embodiment of the present invention. In step S2, the image display terminal includes the following steps in the software initialization process:

[0077] Step S201 b, create a BufferQueue producer object and a consumer object;

[0078] Step S202b, using the producer object and the consumer object as parameters, creating a framework component layer surface C++ object;

[0079] Step S203b, using the producer object and the display device file descriptor as parameters, applying for an image frame buffer from the display driver.

[0080] In this embodiment, what is applied for is the image frame buffer, and what is obtained is the available image frame buffer, and the two are not contradictory.

[0081] Specifically, the present invention utilizes the existing BufferQueue mechanism of the Android platform, can stably and efficiently realize the function of acquiring image frames without destroying the original framework, and effectively reduces the development cost and maintenance cost of the application end.

[0082] Specifically, the supporting features of the kernel driver layer include providing a display driver device node for a peer display.

[0083] Specifically, the supporting features of the kernel driver layer also include applying for creating an image frame buffer.

[0084] Specifically, the steps S1-S5 use the existing virtual display technology of the Android platform to avoid creating a customized application layer interface.

[0085] Specifically, the cross-domain transmission refers to zero-copy transmission of cross-domain image data, which utilizes inter-domain memory sharing technology.

[0086] In this embodiment, the cross-domain is between the instrument domain and the entertainment system domain.

[0087] Specifically, the present invention uses surface java objects, available image frame buffer slot numbers, available image frame buffer virtual addresses, etc. as parameters. Compared with the prior art, the present invention has better technical parameters, with a CPU occupancy rate of less than 1%, a RAM occupancy rate of no more than 3.3 times the image resolution RAM occupancy number, and a frame rate of 25fps.

[0088] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A multi-screen interactive method based on virtual display technology, characterized in that: include: Step S1, start the Android platform, use the surface Java object of the framework component layer as a parameter, and apply to create a virtual display; Step S2, based on the BufferQueue producer-consumer model of the Android platform, the image display end of the surface C++ framework component layer receives the available image frame signal and uses the available image frame buffer slot number as a parameter to obtain the available image frame buffer virtual address, and after the acquisition is completed, inputs it to the kernel driver layer, and at the same time, waits for receiving the kernel driver layer signal to release the available image frame buffer and the available image frame buffer slot number; Step S3, the kernel driver layer converts the received available image frame buffer virtual address into an available image frame buffer physical address, and when the conversion is completed, uses the available image frame buffer physical address as input to use a shared memory mechanism and cross-domain transmission, and when the display of the other end is completed, notifies the framework component layer surface c++ to release the available image frame buffer and the available image frame buffer slot number; In step S1, the Android platform is started, and the surface Java object of the framework component layer is used as a parameter to apply for creating a virtual display, which specifically includes: Step S101, during the boot process of the Android platform, the multi-screen interactive service is automatically started during the boot process; Step S102, obtaining the surface C++ object of the framework component layer, and after the acquisition is completed, converting the C++ object into a Java object; Step S103, using the converted surface Java object as an input parameter, applying to create a virtual display; The cross-domain transmission refers to zero-copy transmission of cross-domain image data, which utilizes inter-domain memory sharing technology.

2. The multi-screen interactive method based on virtual display technology according to claim 1, characterized in that: In step S2, when the image display terminal receives the available image frame signal, the specific operations include: Step S201a, requesting to lock the available image frame buffer slot number; Step S202a, synchronizing and locking the available image frame buffer using the available image frame buffer slot number as a parameter; Step S203a, using the available image frame buffer slot number as a parameter, obtaining an available image frame buffer virtual address; Step S204a, taking the available image frame buffer virtual address as a parameter and inputting it into the kernel driver layer; Step S205a, waiting to receive the kernel driver layer signal to release the available image frame buffer and the available image frame buffer slot number.

3. The multi-screen interactive method based on virtual display technology according to claim 1, characterized in that: In step S3, the specific operations of the kernel driver layer after receiving the available image frame buffer virtual address include: Step S301, converting the available image frame buffer virtual address into an available image frame buffer physical address; Step S302, using the available image frame buffer physical address as input to use a shared memory mechanism and cross-domain transmission; Step S303, after the display on the other end is completed, the framework component layer surface C++ is notified to release the available image frame buffer and the available image frame buffer slot number.

4. The multi-screen interactive method based on virtual display technology according to claim 1, characterized in that: In step S2, the image display terminal includes the following steps in the software initialization process: Step S201b, creating a BufferQueue producer object and a consumer object; Step S202b, using the producer object and the consumer object as parameters, creating a framework component layer surface C++ object; Step S203b, using the producer object and the display device file descriptor as parameters, applying for an image frame buffer from the display driver.

5. The multi-screen interactive method based on virtual display technology according to claim 1, characterized in that: The supporting features of the kernel driver layer include providing a display driver device node for a peer display.

6. The multi-screen interactive method based on virtual display technology according to claim 5, characterized in that: The kernel driver layer also supports the creation of an image frame buffer.

7. The multi-screen interactive method based on virtual display technology according to claim 1, characterized in that: The steps S1-S3 use the existing virtual display technology of the Android platform to avoid creating a customized application layer interface.

Images