Cloud mobile phone real-time screen video generation method and system based on end-cloud collaboration, computer equipment and storage medium

Abstract

The invention discloses a cloud mobile phone real-time screen video generation method based on end-cloud collaboration. The method comprises the following steps: a cloud mobile phone creates and initializes a video object; the client obtains basic audio data and basic video data and sends the basic audio data and the basic video data to the cloud mobile phone; the cloud mobile phone analyzes the basic audio data and the basic video data to obtain material data; the cloud mobile phone processes the video object step by step based on the material data and the video demand, generates a real-time screen video and sends the real-time screen video to the client; the step-by-step processing method comprises fuzzy processing and / or watermark processing. According to the cloud mobile phone real-time screen video generation method and system based on end-cloud cooperation, the computer equipment and the storage medium provided by the invention, based on the cooperation of the cloud mobile phone and the client, the client can control the generation of a real-time screen of the cloud mobile phone and the same-screen display of different clients; and the effects of enhancing the video identifier and protecting the video content are realized.

Description

Technical Field

[0001] The present invention relates to the field of cloud mobile phones, and in particular to a method, system, computer equipment and storage medium for generating real-time screen video of a cloud mobile phone based on end-cloud collaboration. Background Art

[0002] Cloud phones are a type of smart phone based on cloud computing technology. In some scenarios where screen video recording is required for cloud phones, the existing technology mainly uses MediaRecorder in cloud phones to record the VirtualDisplay output Surface created by MediaProject. This method can only achieve simple screen recording, and cannot achieve complex functions such as adding real-time watermarks. Moreover, during the entire process, the client can only control the process of the cloud phone generating real-time screen video, but cannot directly participate in the generation process, which means that it is impossible to display the client status in the real-time screen video and to achieve the same-screen interaction between multiple clients. Summary of the invention

[0003] In order to address the deficiencies in the prior art, the present invention provides a method, system, computer device and storage medium for generating real-time screen video of a cloud phone based on end-cloud collaboration. Based on the collaboration between the cloud phone and the client, it can realize the client-controlled generation of the real-time screen of the cloud phone and the same-screen display of different clients, and achieves the effect of enhancing video identification and protecting video content.

[0004] In order to achieve the above object, the specific scheme adopted by the present invention is:

[0005] The method for generating real-time screen video of a cloud phone based on end-cloud collaboration includes the following steps:

[0006] The cloud phone creates and initializes the video object;

[0007] The client obtains basic audio data and basic video data and sends them to the cloud phone;

[0008] The cloud phone analyzes the basic audio data and basic video data to obtain material data;

[0009] The cloud phone processes the video object step by step based on the material data and video requirements, generates real-time screen video and sends it to the client;

[0010] The step-by-step processing method includes blurring and / or watermarking.

[0011] Preferably, the method for the cloud phone to create and initialize a video object includes:

[0012] The cloud phone initializes EGL;

[0013] The cloud phone creates and initializes a video object, which includes a screen texture object and a watermark texture object;

[0014] The cloud phone creates a vertex buffer object corresponding to the video object and initializes it.

[0015] Preferably, the specific method for the client to obtain basic audio data and basic video data includes:

[0016] The client uses a microphone to collect original sound and a camera to collect original video;

[0017] The client performs pre-quality processing on the original sound and original video;

[0018] The client encodes the original audio data and original video data that have undergone pre-quality processing based on a preset encoding method to obtain basic audio data and basic video data.

[0019] Preferably, when the cloud phone parses the basic audio data and the basic video data to obtain the material data, a weight is assigned to the material data.

[0020] Preferably, when the cloud phone parses the basic audio data and the basic video data, data fusion and data association are performed based on timestamps, and the data association is used to associate different basic audio data and / or basic video data.

[0021] Preferably, the specific method for the cloud phone to gradually process the video object based on the material data and video requirements includes:

[0022] The cloud phone receives a drawing instruction, which includes a fuzzy control instruction and a watermark control instruction;

[0023] The cloud phone determines whether the screen texture is available, and if so, performs fuzzy drawing according to the fuzzy control instruction;

[0024] The cloud phone draws the watermark according to the watermark control instructions.

[0025] Preferably, after the cloud phone generates the real-time screen video, the resources occupied by the video object are released.

[0026] A cloud phone real-time screen video generation device based on end-cloud collaboration, used to implement the above-mentioned cloud phone real-time screen video generation method based on end-cloud collaboration, the device comprising:

[0027] A data transmission module, used to transmit basic audio data and basic video data to the cloud phone, and to transmit real-time video to the client;

[0028] The human-computer interaction module is used to obtain and forward drawing instructions and weight setting instructions to the cloud phone.

[0029] Computer equipment, including:

[0030] a memory storing a computer program;

[0031] A processor is used to execute a computer program to implement the above-mentioned method for generating real-time screen video of a cloud phone based on end-cloud collaboration.

[0032] The storage medium stores a computer program, which, when executed, implements the above-mentioned cloud phone real-time screen video generation method based on end-cloud collaboration.

[0033] The present invention is based on the collaboration between the cloud phone and the client, and can realize the OpenGLES video processing, watermark addition, and recording of the real-time screen of the cloud phone controlled by the client, and can realize the same-screen interaction of multiple clients on a cloud phone. In addition, the present invention achieves the effect of enhancing video identification and protecting video content on the basis of completing screen recording. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is a schematic diagram of the cooperation mode between the cloud phone and the client in the present invention;

[0036] Figure 2 is a flow chart of a method for generating real-time screen video according to the present invention;

[0037] Figure 3 It is a schematic diagram of the specific generation process of the real-time screen video in the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figures 1 to 3 The present invention first provides a method for generating real-time screen video of a cloud phone based on end-cloud collaboration, including S1 to S4.

[0040] S1. The cloud phone creates and initializes a video object. The method for the cloud phone to create and initialize a video object includes S11 to S13.

[0041] S11, the cloud phone initializes EGL. EGL is an interface between rendering APIs such as OpenGLES and the cloud phone operating system, and acts as a bridge. It is a conventional technology in this field. The specific setting and initialization process is known to those skilled in the art and will not be repeated here.

[0042] S12. The cloud phone creates and initializes a video object, where the video object includes a screen texture object and a watermark texture object.

[0043] S13, the cloud phone creates a vertex buffer object corresponding to the video object and initializes it. The creation and initialization methods of the screen texture object, watermark texture object and vertex buffer object are also conventional techniques in the art and will not be repeated here.

[0044] S2, the client obtains basic audio data and basic video data, and sends them to the cloud phone. In the present invention, there can be multiple clients, and they independently collect basic audio data and basic video data, and then send them to the cloud phone respectively, so as to realize the collaboration between one cloud phone and multiple clients. The specific method for the client to obtain basic audio data and basic video data includes S21 to S23. In the present invention, the client can be a common terminal device such as a mobile phone, a tablet computer, a laptop computer and a desktop computer.

[0045] S21, the client uses a microphone to collect original sound and a camera to collect original video. It should be noted that the original video here refers to a video composed of frame images without audio data, and the original sound is coordinated with the original video.

[0046] S22, the client performs pre-quality processing on the original sound and original video. The client is limited by the environment in which it is located, and the collected original sound and original video may have poor quality. For example, the original sound has noise and the original video has jitter. These poor quality parts will cause serious interference to the subsequent process and ultimately affect the quality of the real-time screen video. Therefore, it is necessary to perform pre-quality processing first to improve the quality of the original sound and original video. In order to avoid the excessive load of the cloud phone, resulting in slow or even erroneous generation of real-time screen video, the process of pre-quality processing in the present invention is performed by the client. For the original sound, the client can process it by methods such as audio noise reduction and ambient sound weakening. For the original video, the client can process it by methods such as clarity and deletion of some clips, which are all conventional technologies in the field. The amount of data of the original sound and original video obtained by the client is limited, and the corresponding load for pre-quality processing is also low, which can effectively avoid the problem of excessive load on the cloud phone caused by concentrated processing in the cloud phone.

[0047] S23. The client encodes the original audio data and the original video data that have been pre-quality processed based on a preset encoding method to obtain basic audio data and basic video data. The basic audio data and the basic video data are processed, encoded, and transmitted separately, and can also be processed separately in the cloud phone. The cloud phone does not need to repeatedly separate audio and video, which can reduce the load of the cloud phone and improve the overall efficiency.

[0048] S3. The cloud phone parses the basic audio data and basic video data to obtain material data. Considering that the cloud phone needs to cooperate with multiple clients, there may be differences between different clients. This difference may be determined by the device characteristics of the client. For example, the resolution of the basic video data collected by some clients is low, or it may be determined by the importance of the basic audio data and basic video data collected by the client. In order to better reflect the differences between different clients in the implementation of screen video, the cloud phone parses the basic audio data and basic video data to obtain material data and assigns weights to the material data. The specific weight setting method can be determined according to the differences between the clients. For example, when the difference is manifested in the inconsistent resolution of the basic video data caused by the performance of the client device, a higher weight can be assigned to the material data corresponding to the basic video data with a higher resolution, and a lower weight can be assigned to the material data corresponding to the basic video data with a lower resolution. When processing the material data subsequently, the weights are combined for processing.

[0049] Furthermore, when the difference between different material data is manifested in the inconsistent resolution of basic video data caused by the performance of the client device, when the real-time screen video is subsequently generated, the real-time screen video can be divided into multiple regions, and the areas of different regions are different. The larger area corresponds to the material data with higher weight, and the smaller area corresponds to the material data with lower weight. When the difference between different material data is manifested in the different importance of the basic audio data and basic video data collected by the client, the real-time screen video can also be divided into multiple regions, and the larger area corresponds to the material data with higher importance, and the smaller area corresponds to the material data with lower importance.

[0050] S4. The cloud phone processes the video object step by step based on the material data and video requirements, generates a real-time screen video and sends it to the client, wherein the step-by-step processing method includes blur processing and / or watermark processing.

[0051] The specific method for the cloud phone to gradually process the video object based on the material data and video requirements includes S41 to S43.

[0052] S41, the cloud phone receives a drawing instruction, which includes a fuzzy control instruction and a watermark control instruction. The drawing instruction can be sent to the cloud phone by one of the clients, or generated by a dedicated upper control device and sent to the cloud phone.

[0053] S42, the cloud phone determines whether the screen texture is available, if the screen texture is available, and the fuzzy control instruction is to set the fuzzy, then the fuzzy drawing is performed according to the fuzzy control instruction. If the screen texture is not available, or the fuzzy control instruction is not to set the fuzzy, then the fuzzy processing process is directly skipped.

[0054] S43, the cloud phone draws the watermark according to the watermark control instruction. Specifically, if the watermark control instruction is to add a watermark, the watermark texture is drawn to achieve the addition of the watermark, and if the watermark control instruction is not to add a watermark, the watermark texture processing process is directly skipped.

[0055] Finally, after the cloud phone generates the real-time screen video and sends it to the client, it releases the resources occupied by the video object and stores the real-time screen video locally as a backup. After the client receives the real-time screen video, the client decodes the real-time screen video according to the decoding method corresponding to the encoding method of the aforementioned encoding process, and can play it after decoding.

[0056] The present invention further provides a cloud phone real-time screen video generation device based on end-cloud collaboration, which is used to implement the above-mentioned cloud phone real-time screen video generation method based on end-cloud collaboration. The device includes a data transmission module and a human-computer interaction module.

[0057] The data transmission module is used to transmit basic audio data and basic video data to the cloud phone, and to transmit real-time video to the client.

[0058] The human-computer interaction module is used to obtain and forward drawing instructions and weight setting instructions to the cloud phone.

[0059] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0060] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0061] The present invention also provides a computer device, including a memory and a processor.

[0062] A memory storing a computer program.

[0063] A processor is used to execute a computer program to implement the above-mentioned method for generating real-time screen video of a cloud phone based on end-cloud collaboration.

[0064] Finally, the present invention provides a storage medium storing a computer program, which implements the above-mentioned cloud phone real-time screen video generation method based on end-cloud collaboration when the computer program is executed. Based on the storage medium, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Readom Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk, and other media that can store program codes.

[0065] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0066] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating real-time screen video of a cloud phone based on end-cloud collaboration, characterized in that: The steps include: The cloud phone creates and initializes the video object; The client obtains basic audio data and basic video data and sends them to the cloud phone; The cloud phone analyzes the basic audio data and basic video data to obtain material data; The cloud phone processes the video object step by step based on the material data and video requirements, generates real-time screen video and sends it to the client; The step-by-step processing method includes blurring and / or watermarking.

2. The method for generating real-time screen video of a cloud phone based on end-cloud collaboration according to claim 1, characterized in that: The methods for cloud phones to create and initialize video objects include: The cloud phone initializes EGL; The cloud phone creates and initializes a video object, which includes a screen texture object and a watermark texture object; The cloud phone creates a vertex buffer object corresponding to the video object and initializes it.

3. The method for generating real-time screen video of a cloud phone based on end-cloud collaboration according to claim 1, characterized in that: The specific methods for the client to obtain basic audio data and basic video data include: The client uses a microphone to collect original sound and a camera to collect original video; The client performs pre-quality processing on the original sound and original video; The client encodes the original audio data and original video data that have undergone pre-quality processing based on a preset encoding method to obtain basic audio data and basic video data.

4. The method for generating real-time screen video of a cloud phone based on end-cloud collaboration according to claim 3, characterized in that: When the cloud phone parses the basic audio data and the basic video data to obtain the material data, it assigns weights to the material data.

5. The method for generating real-time screen video of a cloud phone based on end-cloud collaboration according to claim 3, characterized in that: When the cloud phone parses the basic audio data and the basic video data, data fusion and data association are performed based on the timestamp, and the data association is used to associate different basic audio data and / or basic video data.

6. The method for generating real-time screen video of a cloud phone based on end-cloud collaboration according to claim 1, characterized in that: The specific methods for cloud phones to gradually process video objects based on material data and video requirements include: The cloud phone receives a drawing instruction, which includes a fuzzy control instruction and a watermark control instruction; The cloud phone determines whether the screen texture is available, and if so, performs fuzzy drawing according to the fuzzy control instruction; The cloud phone draws the watermark according to the watermark control instructions.

7. The method for generating real-time screen video of a cloud phone based on end-cloud collaboration according to claim 1, characterized in that: After the cloud phone generates a real-time screen video, it releases the resources occupied by the video object.

8. A cloud phone real-time screen video generation device based on end-cloud collaboration, characterized in that: The device is used to implement the method for generating real-time screen video of a cloud phone based on end-cloud collaboration as described in any one of claims 1 to 7, the device comprising: A data transmission module, used to transmit basic audio data and basic video data to the cloud phone, and to transmit real-time video to the client; The human-computer interaction module is used to obtain and forward drawing instructions and weight setting instructions to the cloud phone.

9. Computer device, characterized in that include: a memory storing a computer program; A processor, used to execute a computer program to implement the method for generating real-time screen video of a cloud phone based on end-cloud collaboration as described in any one of claims 1-7.

10. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed, a method for generating real-time screen video of a cloud phone based on end-cloud collaboration as described in any one of claims 1 to 7 is implemented.

Images