Method and device for processing video data, equipment and medium
By obtaining video data in the simulator and converting it into a protocol supported by the host computing resource, using the host hardware resources to process video data, the problems of low processing efficiency and inconsistent protocols of traditional simulators are solved, and more efficient video data processing and improvement are achieved. Good versatility.
Patent Information
- Application Number
- CN202411060205.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional simulators are inefficient in processing video data, failing to make full use of the host's hardware computing resources, especially protocol inconsistencies between different operating systems lead to poor universality.
By obtaining the video data in the simulator, converting it into a protocol that satisfies the host's computing resources, and using hardware resources such as the host's graphics processing unit for processing, the protocol conversion and processing of video data is realized.
It improves the speed and versatility of video data processing, enables the simulator to effectively utilize the host's hardware resources and improves the user experience.
Smart Images

Figure CN118945397A_ABST
Abstract
Description
Technical Field
[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for processing video data. Background Art
[0002] A mobile operating system simulator (hereinafter referred to as a "simulator") is a tool used to simulate a mobile operating system on a computing device, which is usually used to develop, test, and run mobile applications without the need for an actual mobile device. These simulators provide users with a convenient platform where they can create, debug, test, and analyze mobile applications.
[0003] When the simulator processes a large amount of information or data, such as video data, the processing speed may be slow and may even fail if the processing power is limited because the software or program usually needs to perform complex calculations and operations when processing such a large amount of information or data, including decoding, encoding, and analysis. Summary of the invention
[0004] In a first aspect of the present disclosure, a method for processing video data is provided. The method includes: obtaining first video data in a simulator for simulating an operating system running on a first device, the first video data satisfying a first protocol; converting the first video data into second video data, the second video data satisfying a second protocol, the second protocol being different from the first protocol; and providing the second video data to a computing resource of the first device for processing video data.
[0005] In a second aspect of the present disclosure, a device for processing video data is provided. The device includes: an acquisition module configured to acquire first video data in a simulator for simulating an operating system running on a first device, the first video data satisfying a first protocol; a conversion module configured to convert the first video data into second video data, the second video data satisfying a second protocol, the second protocol being different from the first protocol; and a providing module configured to provide the second video data to a computing resource of the first device for processing video data. The device also includes other modules configured to implement other steps in the above method.
[0006] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.
[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.
[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;
[0011] Figure 2 A schematic diagram showing a process for processing video data according to some embodiments of the present disclosure;
[0012] Figure 3 A schematic diagram showing a process for processing video data according to other embodiments of the present disclosure;
[0013] Figure 4 A schematic diagram showing an example structure of a device for processing video data according to some embodiments of the present disclosure;
[0014] Figure 5 A schematic diagram showing an example structure of a device for processing video data according to other embodiments of the present disclosure;
[0015] Figure 6 A flowchart of a method for processing video data according to some embodiments of the present disclosure is shown;
[0016] Figure 7 A block diagram showing an apparatus for processing video data according to some embodiments of the present disclosure; and
[0017] Figure 8 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.
[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0021] As used herein, the term "Hardware Abstraction Layer" (HAL) is a software layer that is an interface layer between the operating system kernel and the hardware circuit, used to abstract the hardware. The hardware abstraction layer provides a standard interface for exposing device hardware functions to higher-level frameworks. The hardware abstraction layer hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on multiple platforms.
[0022] "QEMU" is a processor simulation software that distributes source code under the General Public License (GPL). QEMU supports multiple architectures and is also extensible, open source, portable, can customize new instruction sets, and can store and restore running states. However, QEMU does not fully support some operating systems, has a low simulation speed, and is difficult to use.
[0023] "Goldfish" is a virtual ARM processor used in the emulation environment of the Android operating system. The Android emulator runs the instruction set by running it.
[0024] "Video Acceleration Application Programming Interface" (VA-API) is an open source library and application programming interface (API) standard that can be used for video data processing using graphics processing hardware.
[0025] "Mesa 3d" is a cross-platform graphics API that can be used to render 2D and 3D graphics. Mesa 3d enables graphics processing units to be used for video data processing to increase processing speed.
[0026] The "virgl video driver" is a video driver applied to an emulator or a virtual machine, which is used to convert video data into data that can be parsed by an application and enable the application to process or display the video data.
[0027] "virglrenderer" is an interface for a virtualized graphics processing unit, used to provide an interface for processing video data for an emulator or virtual machine to use a graphics processing unit for video data processing.
[0028] Traditionally, simulators run on a computing device (e.g., a host) and simulate a certain operating system. Since simulators usually do not use the host's hardware computing resources, such as a graphics processing unit (GPU), to process data, simulators usually do not involve such hardware computing resources when processing video data. However, since video data is usually large in size, traditional simulators have low processing efficiency for video data.
[0029] In addition, the methods and standards for video data processing are not exactly the same in different operating systems, which makes it impossible for the simulator running on the host to fully utilize the computing resources of the host. In the system simulator, the software computing resources within the Android system are mainly used for video encoding and decoding. If you want to use the host's hardware computing resources (such as GPU) to process video data on the simulator, you need to use a code base or specific software based on the Android system. However, these unique code bases or specific software have poor versatility and do not support many APIs. For example, the Android simulator may need to use a virtual processor, such as Goldfish, for video encoding and decoding. In the case of video encoding and decoding based on Goldfish, you may also need to use the Android Open Source Project (ASOP) library and specific QEMU patches, which will result in poor versatility of such solutions.
[0030] In response to the above and other potential problems, an embodiment of the present disclosure proposes a scheme for processing video data. In this scheme, first video data that meets the first protocol in a simulator running on a host (hereinafter also referred to as the first device) is converted into second video data that meets the second protocol, and the second video data is provided to the host's computing resources for processing video data. In this way, the host's computing resources, such as a GPU, a graphics card, or other appropriate computing resources for processing video data, can be used to process the simulator's video data. The scheme of the present disclosure introduces hardware computing resources for processing video data into the simulator's processing of video data, thereby improving the processing speed of video data and also improving the versatility of video data processing.
[0031] Some example embodiments of the present disclosure will be described below with reference to the accompanying drawings.
[0032] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the example environment 100, a video data processing device 101 is used to control the transmission and processing of video data. A first device 130 provides a computing resource 120 for processing video data and hosts an operating environment of a simulator 110.
[0033] Specifically, the video data processing device 101 may provide the video data in the simulator 110 running on the first device 130 to the computing resource 120 of the first device 130 for processing the video data, and may also provide the processed video data from the computing resource 120 to the simulator 110 .
[0034] The video data processing device 101 can perform various processing on the video data obtained from the simulator 110. For example, the video data processing device 101 can adjust or convert the video data so that the video data meets different protocols. In addition, the video data processing device 101 can also control and manage other data or operations of the simulator 110, which is not limited in the embodiments of the present disclosure.
[0035] It should be understood that the video data processing device 101 and the first device 130 may be integrated or independent. In some embodiments, the video data processing device 101 may be the same device as the first device 130. That is, the video data processing device 101 may control the simulator on the local machine to use specific computing resources on the local machine, such as a GPU. As an alternative, in some embodiments, the video data processing device 101 may be a different device from the first device 130, and the video data processing device 101 may perform data transmission with the first device 130.
[0036] In some embodiments, the video data processing device 101 may be a computer, a cloud server, a data center, or other appropriate devices. In some embodiments, the first device 130 may be a computer, a cloud server, a data center, or other appropriate devices. It should be understood that the video data processing device 101 may be configured to implement any or all of the techniques of the embodiments of the present disclosure, and the embodiments of the present disclosure do not impose any limitation on this.
[0037] The following Figures 2 to 8 The technical solution of the present disclosure is further described.
[0038] Figure 2 FIG. 2 is a schematic diagram of a process 200 for processing video data according to some embodiments of the present disclosure. The process 200 may be performed by, for example Figure 1 The video data processing device 101 or other suitable device in the embodiment is executed. In the following, for the purpose of explanation, reference is made to Figure 1 The process 200 is described below.
[0039] like Figure 2 As shown, the video data processing device 101 obtains the first video data 210 satisfying the first protocol in the simulator 110, and converts the first video data 210 into the second video data 220. The second video data 220 satisfies the second protocol, and the second protocol is different from the first protocol. Then, the video data processing device 101 provides the second video data 220 to the computing resource 120 for processing video data of the first device 130.
[0040] The computing resource 120 for processing video data may include a physical machine device of the first device, such as but not limited to a GPU, a graphics card, a video encoding card, etc. It should be understood that the above examples of computing resources 120 are only for the purpose of description and are not intended to limit the embodiments of the present disclosure. In other embodiments of the present disclosure, the computing resource 120 may also include other hardware processing units on the first device, etc.
[0041] In some embodiments, the video data processing device 101 can obtain the first video data in the simulator 110 through at least one simulator video data interface, and can also convert the first video data 210 into the second video data 220 based on the protocol conversion module. In addition, the video data processing device 101 can provide the video data to the computing resource 120 through at least one host video data port. The detailed process related to this will be described below in conjunction with Figure 5 Give a description.
[0042] The simulator 110 may be used to simulate any appropriate operating system, for example, it may be an Android simulator, etc. It should be understood that the above examples of simulators are merely exemplary, and the present disclosure does not impose any limitation thereto.
[0043] In addition, the present disclosure does not impose any limitation on the number of simulators 110. For example, at least one simulator 110 may be run on the first device 130. When multiple simulators 110 are run on the first device 130, the operating systems of these simulators 110 may be different or the same.
[0044] In some embodiments, the operating system of the first device 130 and the operating system of the simulator 110 may be different or the same, and the embodiments of the present disclosure are not limited to this.
[0045] The computing resource 120 of the first device 130 may be a graphics processing unit of the first device 130 , such as a GPU, a GPU cluster, an accelerated processing unit (APU), or other appropriate hardware processing units.
[0046] In the above-mentioned process of converting the first video data 210 into the second video data 220, the video data processing device 101 can transmit encoding and decoding information to the first video data 210 or perform other appropriate processing. In some embodiments, the second protocol can be, for example, a VA-API protocol, etc. It should be understood that the above examples of protocols are only for the purpose of description and do not limit the embodiments of the present disclosure. In other embodiments according to the present disclosure, other appropriate protocols are also applicable, and the embodiments of the present disclosure are not limited to this.
[0047] In some embodiments, the computing resource 120 may be used to decode or otherwise properly process the second video data 220. The video data decoded by the computing resource 120 may be provided to the simulator 110 for transmitting, storing, or displaying the video data through a display interface.
[0048] Utilizing the exemplary embodiments discussed above, the protocol satisfied by the video data can be changed by converting the video data so that the video data can be processed in an operating system that supports different protocols. Thus, the graphics processing unit in the first device 130 can be used to process the video data in the simulator, thereby improving the processing speed of the video data.
[0049] Figure 3 FIG. 3 is a schematic diagram of a process 300 for processing video data according to some other embodiments of the present disclosure. The process 300 may be, for example, Figure 1The video data processing device 101 or other suitable device in the embodiment is executed. In the following, for the purpose of explanation, reference is made to Figure 1 and Figure 2 Process 300 is described with reference to an embodiment of the present invention.
[0050] like Figure 3 As shown, in the video data processing process 300, the video data processing device 101 obtains the first video data 210 in the simulator 110 for simulating the operating system running on the first device 130, and converts the first video data 210 into the second video data 220. Here, the first video data 210 satisfies the first protocol, and the second video data 220 satisfies the second protocol different from the first protocol. Then, the video data processing device 101 provides the second video data 220 to the computing resource 120 of the first device 130 for processing video data.
[0051] After the processing of the computing resource 120 is completed, the video data processing device 101 can convert the processed second video data, that is, the third video data 310, into the fourth video data 320. Since the third video data 310 is obtained by processing the second video data 220 by the computing resource 120, the third video data 310 and the second video data 220 can be similar and both meet the second protocol. After the third video data 310 is converted, the fourth video data 320 obtained can meet the first protocol, and thus can be directly used by the simulator 110. In this way, when the fourth video data 320 is provided to the simulator 110, the simulator 110 can directly use the fourth video data 320, for example, play it.
[0052] In some embodiments, the video data processing device 101 obtains the first video data 210 from the simulator application in the simulator 110 through the simulator video data interface. For example, the simulator application may be an application installed in the simulator, and the first video data 210 may be video data downloaded from the Internet by the simulator application. Then, the video data processing device 101 may convert the first video data 210 to obtain the second video data 220, for example, converting the first video data 210 from the Android system that does not meet the VA-API protocol into the second video data 220 that meets the VA-API.
[0053] Next, the video data processing device may provide the second video data 220 to the computing resource 120, so that the computing resource 120 may decode the second video data 220, thereby obtaining the third video data 310. Then, the video data processing device 101 converts the third video data 310 into the fourth video data 320, for example, converts the third video data 310 satisfying the VA-API protocol into the fourth video data 320 satisfying the video data processing protocol of the Android system. In addition, the fourth video data 320 may be provided to the simulator application in the simulator 110 for transmission, storage, or video presentation through a display interface, etc.
[0054] In some embodiments, after the fourth video data 320 is provided to the simulator 110, the simulator 110 may store or transmit the fourth video data 320 or provide the fourth video data 320 to a display interface of the simulator 110 to display video content associated with the fourth video data 320. It should be understood that other processing performed by the simulator 110 on the fourth video data 320 is also applicable in other embodiments, and the present disclosure does not impose any limitation on this.
[0055] pass Figure 3 The discussed embodiment can provide video data processed by the graphics processing unit in the host to the simulator to achieve a closed loop of video transmission, thereby improving the video data processing speed experienced by users of the simulator.
[0056] Figure 4 A schematic diagram of an example structure 400 of an apparatus for processing video data according to some embodiments of the present disclosure is shown. Figures 1 to 3 The example structure 400 is described with reference to an embodiment of the present invention. The example structure 400 can be implemented on the first device 130, and can also be implemented on other suitable devices.
[0057] like Figure 4 As shown, the example structure 400 includes a simulator 402, a protocol conversion module 404, and a graphics processing unit 406. The simulator 402 is similar to Figure 1 The simulator 110 shown is not described in detail here.
[0058] The protocol conversion module 404 is used to convert the codec protocol that the video data satisfies. The protocol conversion module 404 can convert the first video data 210 that satisfies the first protocol into the second video data 220 that satisfies the second protocol. The protocol conversion module 404 can be implemented based on the second protocol and a video codec tool that supports the second protocol. Bidirectional data transmission can be performed between the simulator 402 and the protocol conversion module 404.
[0059] The graphics processing unit 406 may be a part or the entirety of the computing resource 120 , and bidirectional data transmission may be performed between the protocol conversion module 404 and the graphics processing unit 406 .
[0060] Specifically, the protocol conversion module 404 may obtain the first video data 210 from the simulator 402, then convert the first video data 210 into the second video data 220, and provide the second video data 220 to the graphics processing unit 406. The protocol conversion module 404 may also obtain the third video data 310 from the graphics processing unit 406, then convert the third video data 310 into the fourth video data 320, and provide the fourth video data 320 to the simulator 402.
[0061] In some embodiments, the protocol conversion module 404 may call the hardware abstraction layer of the simulator 402. For example, the implementation of the protocol conversion module 404 may be based on some or all functions of the hardware abstraction layer connecting the video data processing framework in the simulator 110 and the virtual video data processing computing resources.
[0062] In the above-mentioned embodiment, the first protocol is a protocol for processing video data supported by the simulator 402. For example, if the operating system of the simulator 110 is an Android operating system, the first protocol may be a video codec protocol supported by the Android operating system. In some embodiments, the second protocol is a protocol for processing video data supported by the first device 130. For example, if the operating system of the first device 130 is operating system, the first protocol can be Video codec protocols supported by the operating system.
[0063] The second protocol may be, for example, a protocol supported by a video acceleration application programming interface, or other appropriate protocols. Regarding the second protocol, the simulator 402 may or may not support it, and the embodiments of the present disclosure do not impose any limitation on this.
[0064] In the embodiments of the present disclosure described above, video data processing involves encoding or decoding of video data, and may also involve other processing methods, which are not limited in any way by the present disclosure.
[0065] Figure 3 The illustrated embodiment converts the video data to change the protocol that the video data satisfies by converting the video data through the protocol conversion module 404. The protocol conversion module 404 can be implemented flexibly and quickly, so that intermediate processing such as protocol conversion of the video data can be completed at low cost and with high flexibility.
[0066] The following will refer to Figure 5 An example structure 500 of an apparatus for processing video data according to some embodiments of the present disclosure is described. Figure 5 Shown is based on Figure 4 The specific form of the exemplary structure 400 of the apparatus for processing video data of the embodiment of the present invention. Figures 1 to 4 The example structure 500 is described with reference to an embodiment of the present invention.
[0067] like Figure 5 As shown, the example structure 500 of the device for processing video data may include a simulator application 502, a first interface 504, a hardware abstraction layer 508, a third interface 510, a fourth interface 512, a simulator controller 415, a fifth interface 516, and a host graphics processing unit 518. It should be understood that the various units / modules in the above example structure 500 are exemplary rather than restrictive, and the embodiments of the present disclosure do not impose any specific limitations on their specific implementation.
[0068] In some embodiments, the simulator application 502 may be an application in the simulator 110 for initiating processing of video data, and may also be used to transmit, store, display, or process the video data in any other manner that may implement the embodiments of the present disclosure.
[0069] The first interface 504 can perform bidirectional data transmission with the simulator application 502. The first interface 504 is associated with the operating system of the simulator 110. For example, if the operating system of the simulator 110 is an Android system, the first interface 504 can be a part or all of the multimedia framework of the Android system, such as MediaCodec. The first interface 504 can support a first protocol.
[0070] The second interface 506 can perform bidirectional data transmission with the first interface 504. The second interface 506 is associated with the operating system of the simulator 110. For example, if the operating system of the simulator 110 is an Android system, the second interface 506 can be a part or all of the multimedia framework of the Android system, such as Media Codec2. The second interface 506 can support the first protocol.
[0071] The hardware abstraction layer 508 can perform bidirectional data transmission with the second interface 504. The hardware abstraction layer 508 is associated with the first protocol and the second protocol. The hardware abstraction layer 508 can implement part or all of the functions of the protocol conversion module 404. In some embodiments, the hardware abstraction layer 508 can be implemented based on the Fast Forward Moving Picture Expert Group (FFMPEG) library and the VA-API protocol. Ffmpeg is a set of open source libraries and a collection of command line tools that provide a cross-platform open source audio and video framework. The Ffmpeg library can provide a variety of codec tools and development kits for processing video data. In some embodiments, the hardware abstraction layer 508 based on the FFMPEG library and the VA-API protocol can be registered in the operating system of the simulator 110.
[0072] In some embodiments, the hardware abstraction layer 508 may be injected into the operating system of the simulator 110 before performing video data processing.
[0073] Table 1 shows an example of registering the hardware abstraction layer 508 in the Android operating system of the emulator 110 .
[0074] Table 1
[0075]
[0076] In the example shown in Table 1, the Android system configures the codec to be used through an eXtensible Markup Language (XML) file.
[0077] Table 2 shows an example of making the Android system select a codec based on the ffmpeg library and the VA-API protocol.
[0078] Table 2
[0079]
[0080]
[0081]
[0082] Table 3 shows an example of compiling an executable program in an Android system and adding XML to an Android system image.
[0083] Table 3
[0084]
[0085] Table 4 shows an example of adding services in the Android system.
[0086] Table 4
[0087]
[0088]
[0089] In the embodiments shown in Tables 2 to 4, when the Android system is turned on, it will traverse android.hardware.media.c2@1.2-service-ffmpeg.rc and then start the android.hardware.media.c2@1.2-service-ffmpeg program according to the description of the rc file. In addition, android.hardware.media.c2@1.2-service-ffmpeg runs in the system as a system service and plays a part of the function of the hardware abstraction layer 508.
[0090] Table 5 shows an example of the ffmpeg library in the hardware abstraction layer 508.
[0091] Table 5
[0092]
[0093] In the above Table 5, libva is the VA-API implementation library.
[0094] The third interface 510 can perform bidirectional data transmission with the hardware abstraction layer 508. The third interface 510 can support the second protocol. In some embodiments, the third interface 510 is an API called by the hardware service layer 508. In some embodiments, the third interface 510 is a VA-API called by the hardware service layer 508.
[0095] The fourth interface 512 can perform bidirectional data transmission with the third interface 510. The fourth interface 512 can support the second protocol. In some embodiments, the fourth interface 512 can be a video rendering interface, such as mesa 3d. The fourth interface 512 can call a kernel driver, such as a virgl video driver, to process video data.
[0096] The simulator controller 514 can perform bidirectional data transmission with the fourth interface 512. The simulator controller 514 is a control program in the first device 130 for controlling the simulator 110.
[0097] The fifth interface 516 can perform bidirectional data transmission with the emulator controller 514 and can support the second protocol. In some embodiments, the fifth interface 516 can be a host video processing interface supporting VA-API, for example, virglrenderer.
[0098] The host graphics processing unit 518 can perform bidirectional data transmission with the fifth interface 516. The host graphics processing unit 518 can support the second protocol. The host graphics processing unit 518 can be the computing resource 120 of the first device 130. In some embodiments, the host graphics processing unit 518 can be a GPU of a computer or a GPU cluster of a data center and other appropriate hardware. The host graphics processing unit 518 can encode and decode video data.
[0099] refer to Figure 5 In some examples, during the process of processing video data, first, the simulator application 502 can call the first interface 504 to obtain the first video data 210 from the simulator application 502, and the second interface 506 can obtain the first video data 210 from the first interface 504 and provide it to the hardware abstraction layer 508.
[0100] The hardware abstraction layer 508 can convert the first video data 210 into the second video data 220. In some embodiments, the reference Figure 5 The hardware abstraction layer 508 can implement an interface that supports ffmpeg by inheriting an abstract class in the Android multimedia framework, so that the hardware abstraction layer 508 can be called by the second interface 506 and can implement the transmission of codec parameters through the inherited class, so that the codec parameters are passed from the second interface 506 to the buffer in the hardware abstraction layer 508. In addition, other information such as the length, width, height and compression format of the codec can be passed. The hardware abstraction layer 508 can call the VA-API of the third interface 510 to transmit the above information and the second video data 220.
[0101] Table 6 shows an example of implementing a user-mode VA-API driver in an Android system.
[0102] Table 6
[0103]
[0104]
[0105] In the example shown in Table 6, the gallium-va parameter is used in the Android system to enable the VA-API function and compile the VA-API library.
[0106] Specifically, in the process of the hardware abstraction layer 508 converting the first video data 210 into the second video data 220, the video data processing device 101 can obtain the codec standard supported by the simulator 110. Then, the video data processing device 101 can perform codec format initialization based on the codec standard supported by the simulator 110. Next, the video data processing device 101 can transfer codec parameters and codec information as described above.
[0107] Table 7 shows an example of the process of initializing the codec format.
[0108] Table 7
[0109]
[0110] In the codec format initialization process shown in Table 7, the hardware abstraction layer 508 may perform codec format initialization after querying the codec standards supported by the simulator 110 .
[0111] Continue to refer Figure 5 Next, the third interface 510 may call the fourth interface 512, thereby calling the virgl video driver to process the second video data 220. Then, the video data processing device 101 may transmit the second video data from the fourth interface 512 to the simulator controller 514. The simulator controller 514 may call the fifth interface 516 to provide the second video data 220 to the host graphics processing unit for video data processing.
[0112] like Figure 5 As shown, the video data processing device 110 can process the second video data 220 through the host graphics processing unit 518 to obtain the third video data 310 and call the fifth interface 516 to send the third video data 310 to the simulator controller 514. Then the simulator controller 514 can call the fourth interface 512 to send the third video data 310. Next, the third interface 510 can call the fourth interface 512 to send the third video data 310 to the hardware abstraction layer 508. At the hardware abstraction layer 508, the third video data 310 can be converted into the fourth video data 320. Then, the hardware abstraction layer 508 can call the second interface 506 to send the fourth video data 320. The second interface 506 can call the first interface 504 to send the fourth video data 320 to the simulator application 502.
[0113] As an alternative, the third video data 310 described above may be directly sent to the second interface 506 through the simulation controller 514 . Then the second interface 506 may call the first interface 504 to send the third video data 310 to the simulator application 502 .
[0114] Table 8 shows an example of sending the video data processed by the computing resource 120 to the second interface 506 .
[0115] Table 8
[0116]
[0117] use Figure 5The exemplary embodiment discussed can convert the protocol of the video data according to the interface protocol library and the codec tool in the hardware abstraction layer, so that the video data can be processed on the graphics processing unit of the host, effectively improving the speed of processing the video data.
[0118] It should be understood that the data transmission path in the video data processing process can be any other way that can implement the present disclosure, and the embodiments of the present disclosure have no limitation to this.
[0119] From the above discussion, it can be seen that the embodiments of the present disclosure provide a video data protocol conversion system based on the codec standard supported by the operating system. The method of the present disclosure can convert video data according to the supported video data processing protocol so that it can be processed by the computing resources of the video data processing, thereby improving the efficiency and versatility of video data processing.
[0120] Figure 6 FIG. 6 is a flowchart of a method 600 for processing video data according to some embodiments of the present disclosure. The method 600 may be, for example, Figure 1 The video data processing device 101 or other appropriate device is used for execution.
[0121] At 610, the video data processing device 101 obtains first video data in a simulator for simulating an operating system running on a first device, and the first video data satisfies a first protocol. At 620, the video data processing device 101 converts the first video data into second video data, and the second video data satisfies a second protocol, which is different from the first protocol. At 630, the video data processing device 101 provides the second video data to a computing resource of the first device for processing video data.
[0122] In some exemplary embodiments, the video data processing device 101 may convert the third video data satisfying the second protocol into the fourth video data satisfying the first protocol. The third video data is obtained by processing the second video data by computing resources. The video data processing device 101 may also provide the fourth video data to the simulator.
[0123] In some exemplary embodiments, the video data processing device 101 may convert the first video data into the second video data through a protocol conversion module, and the protocol conversion module may be implemented based on the second protocol and a video codec tool supporting the second protocol.
[0124] In some exemplary embodiments, the protocol conversion module comprises a hardware abstraction layer of the simulator.
[0125] In some exemplary embodiments, the first protocol comprises a protocol supported by the simulator for processing video data, and the second protocol comprises a protocol supported by the first device for processing video data.
[0126] In some exemplary embodiments, the second protocol is a protocol supported by a video acceleration application programming interface.
[0127] In some exemplary embodiments, video data processing may include encoding or decoding video data.
[0128] In some example embodiments, the computing resource may include a graphics processing unit of the first device.
[0129] Figure 7 FIG. 7 is a block diagram of an apparatus 700 for processing video data according to some embodiments of the present disclosure. The apparatus 700 may be implemented as or included in Figure 1 The video data processing device 101 of FIG. 700 includes various modules / components in the apparatus 700 that can be implemented by hardware, software, firmware, or any combination thereof.
[0130] As shown in the figure, the device 700 includes an acquisition module 710, which is configured to acquire first video data in a simulator running on a first device for simulating an operating system, the first video data satisfying a first protocol; a conversion module 720, which is configured to convert the first video data into second video data, the second video data satisfying a second protocol, and the second protocol is different from the first protocol; and a providing module 730, which is configured to provide the second video data to a computing resource of the first device for processing video data.
[0131] In some exemplary embodiments, the conversion module 720 may be further configured to convert third video data satisfying the second protocol into fourth video data satisfying the first protocol, wherein the third video data is obtained by processing the second video data through computing resources; and provide the fourth video data to the simulator.
[0132] In some exemplary embodiments, the conversion module 720 may be further configured to convert the first video data into the second video data through a protocol conversion module, where the protocol conversion module is implemented based on the second protocol and a video codec tool supporting the second protocol.
[0133] In some exemplary embodiments, the protocol conversion module may include a hardware abstraction layer of the simulator.
[0134] In some exemplary embodiments, the first protocol may include a protocol supported by the simulator for processing video data, and the second protocol may include a protocol supported by the first device for processing video data.
[0135] In some exemplary embodiments, the second protocol may be a protocol supported by a video acceleration application programming interface.
[0136] In some exemplary embodiments, video data processing may include encoding or decoding video data.
[0137] In some example embodiments, the computing resource may include a graphics processing unit of the first device.
[0138] Figure 8 8 is a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 8 The electronic device 800 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 8 The electronic device 800 shown can be used to implement Figure 1 The video data processing device 101 or Figure 7 An apparatus 700 for processing video data.
[0139] like Figure 8 As shown, the electronic device 800 is in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the electronic device 800.
[0140] The electronic device 800 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., register, cache, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. The storage device 830 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 800.
[0141] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 8As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.
[0142] The communication unit 840 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.
[0143] The input device 850 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 860 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 800, or communicate with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).
[0144] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.
[0145] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0146] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0147] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0148] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.
[0149] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.
Claims
1. A method for processing video data, include: Acquire first video data in a simulator for simulating an operating system running on a first device, wherein the first video data satisfies a first protocol; Converting the first video data into second video data, where the second video data satisfies a second protocol, and the second protocol is different from the first protocol; as well as The second video data is provided to a computing resource of the first device for processing video data.
2. The method according to claim 1, further comprising: include: converting third video data satisfying the second protocol into fourth video data satisfying the first protocol, wherein the third video data is obtained by processing the second video data by the computing resource; as well as The fourth video data is provided to the simulator.
3. The method according to claim 1, wherein the first video data is converted into the second video data through a protocol conversion module, and the protocol conversion module is implemented based on the second protocol and a video codec tool supporting the second protocol. The method according to claim 3 , wherein the protocol conversion module comprises a hardware abstraction layer of the simulator.
5. The method according to any one of claims 1-4, wherein the first protocol comprises a protocol supported by the simulator for processing video data, and the second protocol comprises a protocol supported by the first device for processing video data. The method of claim 5 , wherein the second protocol is a protocol supported by a video acceleration application programming interface. The method according to claim 1 , wherein the video data processing comprises encoding or decoding the video data.
8. The method of claim 1, wherein the computing resource comprises a graphics processing unit of the first device.
9. A device for processing video data, include: An acquisition module is configured to acquire first video data in a simulator for simulating an operating system running on a first device, wherein the first video data satisfies a first protocol; a conversion module, configured to convert the first video data into second video data, the second video data satisfies a second protocol, and the second protocol is different from the first protocol; as well as The providing module is configured to provide the second video data to a computing resource of the first device for processing video data.
10. An electronic device, include: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.
11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.
Citation Information
Patent Citations
Using Multiple Protocols In A Virtual Desktop Infrastructure
CN107683461A
Simulation including multiple simulators
CN107784152A
Virtualization-based hardware decoding method, decoding device and storage medium
CN107911700A
Implementation method and implementation system of security desktop transport protocol based on virtualization environment
CN111901414A
Android video decoding hardware acceleration method based on Linux desktop video card
CN114125558A
