Data processing method and system, cloud terminal, server and computing device

By sharing the video encoding and decoding tasks of cloud terminals to the server, using the server's computing resources, the problem of high CPU resource occupancy of cloud mobile phones is solved, the cost is reduced and the encoding and decoding efficiency is improved, and the cloud mobile phones are enhanced in live broadcast and mobile office Competitiveness in the field.

CN114157919BActive Publication Date: 2024-11-15HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010990303.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-09-07
Filing Date
2020-09-18
Publication Date
2024-11-15
Estimated Expiration
2040-09-18

AI Technical Summary

Technical Problem

In live video broadcasts and mobile office scenarios, cloud mobile phones have high requirements for CPU performance, resulting in increased costs and wasted CPU resources on a single cloud mobile phone, which affects competitive advantages and efficiency.

Method used

The encoding and decoding tasks of cloud terminals are shared on the server, and the powerful computing resources of the server are used for hardware or software encoding and decoding, reducing the processor resource usage of cloud terminals, and dynamically adjusting the encoding and decoding methods according to application needs.

Benefits of technology

It reduces the cost and specifications of a single cloud terminal, improves encoding and decoding efficiency, avoids waste of CPU resources, and enhances the competitive advantage of cloud mobile phones in the fields of live broadcast and mobile office.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114157919B_ABST
    Figure CN114157919B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a method, system, cloud terminal, server, computing device and computer program product for processing data, which can solve the problem of insufficient CPU resources and software compatibility of cloud terminals. The method includes: the cloud terminal sends data; the server receives the data; the server processes the data, wherein the processing is encoding the data or decoding the data; and the server sends the processed data to the cloud terminal.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims priority to Chinese patent application No. 202010928653.1, filed on September 7, 2020, with invention name “Encoding / decoding method, device, server and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of cloud technology, and mainly to a data processing method and system, a cloud terminal, a server, a computing device and a computer program product. Background Art

[0003] As a new type of service, cloud phone plays a very good role in extending and supplementing physical mobile phones, and has begun to emerge in cloud phone games, cloud phone live broadcasts, mobile office and other scenarios. In all of the above scenarios, video encoding and decoding are involved, for example: (1) In cloud phone games, the mobile phone screen image (or game image) needs to be video encoded and then transmitted to the physical phone through the network; (2) In the live broadcast scenario, the received camera image data of the physical phone needs to be video decoded and displayed on the cloud phone. When the live broadcast application is started on the cloud phone, the cloud phone encodes the camera image data and sends it to the live broadcast platform; (3) In the mobile office scenario, when employees in industries with high information security requirements such as government and finance make video calls through instant messaging applications, the camera image on the cloud phone needs to be video encoded, and video decoding is required when sharing and playing videos in video conferences.

[0004] However, since video encoding and decoding have relatively high requirements on the CPU performance of cloud phones, in order to meet the needs of cloud phone live streaming and mobile office, the CPU specifications of a single cloud phone must be improved. This will have two disadvantages: (1) It will increase the cost of a single cloud phone, which is not conducive to building the competitive advantage of cloud phones in the fields of live streaming and mobile office; (2) In addition, the use time of cloud phones is time-limited. For example, live streaming and mobile office are concentrated in the daytime working time and the evening leisure time after get off work, which will cause a waste of CPU resources of cloud phones. Summary of the invention

[0005] The following is a detailed description of a data processing method and system, cloud terminal, server, computing device and computer program product provided in an embodiment of the present application, which can reduce the occupancy of CPU resources of cloud phones by processing data in conjunction with the accompanying drawings.

[0006] In the first aspect of the present disclosure, a data processing method is provided, which is applied between a cloud terminal and a server. The method comprises: the cloud terminal sends data; the server receives the data; the server processes the data, wherein the processing is encoding the data or decoding the data; and the server sends the processed data to the cloud terminal.

[0007] The data processing method provided by the present application shares the encoding and decoding tasks of the cloud terminal to the server, thereby saving the processor resources of the cloud terminal and reducing the cost and specifications of a single cloud terminal. The powerful computing resources possessed by the server further improve the efficiency of the encoding and decoding tasks on the cloud terminal.

[0008] In a possible design of the first aspect, the server encoding the data includes: performing hardware encoding or software encoding on the data.

[0009] The server's encoder module is responsible for the encoding task. Among them, the software encoding task runs on the CPU; the hardware encoding task runs on the hardware codec chip. When the encoding task is executed on the server, the server can provide higher encoding efficiency.

[0010] In a possible design of the first aspect, the server decoding the data includes: performing hardware decoding or software decoding on the data.

[0011] The server's decoder module is responsible for the decoding task. Among them, the software decoding task runs on the CPU, and the hardware decoding task runs on the hardware codec chip. When the decoding task is executed on the server, the server can provide higher decoding efficiency.

[0012] In a possible design of the first aspect, the data processing method further includes: the cloud terminal sends a processing rule to the server; the processing rule is used to determine whether to perform hardware encoding or software decoding on the data, or the processing rule is used to determine whether to perform software encoding or software decoding on the data.

[0013] The cloud terminal sends instructions and data to the server. The instructions include processing rules, and the processing rules include encoding methods or decoding methods: encoding methods include hardware encoding or software encoding, and decoding methods include hardware decoding or software decoding. The server calls different encoding modules and decoding modules to perform operations according to different encoding and decoding methods. Please refer to the previous description.

[0014] In a possible design of the first aspect, the data processing method further includes: before the cloud terminal sends the data, the cloud terminal obtains a request from the terminal to process the data.

[0015] Before the cloud terminal sends data to the server, it must first obtain a data processing request from the terminal. This request comes from a physical terminal and is triggered by the user of the terminal.

[0016] In a possible design of the first aspect, the data processing method further includes: before the cloud terminal sends the data, the cloud terminal obtains a request to process the data, and the request is triggered by an application on the cloud terminal.

[0017] Before the cloud terminal sends data to the server, the cloud terminal obtains a data processing request from the application, which is related to the encoding or decoding requirements of the application layer. If the application needs encoding, an encoding request is issued, and if the application needs decoding, a decoding request is issued.

[0018] Among them, the processing rules in the request sent by the application are not the same as the processing rules sent by the cloud terminal to the server in the third possible implementation method of the first aspect of the present invention, that is, the cloud terminal can modify the processing rules and send the modified processing rules to the server. There are four ways of modification: software encoding to hardware encoding, hardware encoding to software encoding, software decoding to hardware decoding, and hardware decoding to software decoding. Among them, changing software encoding to hardware encoding and software decoding to hardware decoding can improve the efficiency of encoding and decoding, reduce the efficiency of cloud terminal CPU resources, and solve the incompatibility problem when some applications call hardware encoding or hardware decoding on the cloud terminal.

[0019] In a possible design of the first aspect, the data processing method further includes: the cloud terminal sends a virtual desktop obtained according to the data to the terminal.

[0020] The application of the cloud terminal obtains a virtual desktop based on the processed data, and sends the virtual desktop to the terminal so that the terminal displays the virtual desktop to the user.

[0021] In a second aspect, the present invention provides a system for processing data, the system comprising:

[0022] Cloud terminal, used to send data to the server;

[0023] The server is used to receive the data, process the data, and then send the processed data to the cloud terminal, wherein the processing is encoding or decoding the data.

[0024] The data processing method provided by the present application shares the encoding and decoding tasks of the cloud terminal to the server, thereby saving the processor resources of the cloud terminal and reducing the cost and specifications of a single cloud terminal. The powerful computing resources possessed by the server further improve the efficiency of the encoding and decoding tasks of the cloud terminal.

[0025] In a possible design of the second aspect, the server further includes: a device for hardware encoding or software encoding the data.

[0026] The server's encoder module is responsible for the encoding task. Among them, the software encoding task runs on the CPU; the hardware encoding task runs on the hardware codec chip. When the encoding task is executed on the server, the server can provide higher encoding efficiency.

[0027] In a possible design of the second aspect, the server further includes: a device for performing hardware decoding or software decoding on the data.

[0028] The server's decoder module is responsible for the decoding task. Among them, the software decoding task runs on the CPU; the hardware decoding task runs on the hardware codec chip. When the decoding task is executed on the server, the server can provide higher decoding efficiency.

[0029] In a possible design of the second aspect, the data processing system further includes:

[0030] The cloud terminal is used to send a processing rule to a server, wherein the processing rule is used to determine whether to perform hardware encoding or software decoding on the data, or the processing rule is used to determine whether to perform software encoding or software decoding on the data.

[0031] The cloud terminal sends instructions and data to the server. The instructions include processing rules, and the processing rules include encoding methods or decoding methods: encoding methods include hardware encoding or software encoding, and decoding methods include hardware decoding or software decoding. The server calls different encoding modules and decoding modules to perform operations according to different encoding and decoding methods. Please refer to the previous description.

[0032] In a possible design of the second aspect, the data processing system further includes:

[0033] The cloud terminal is used to obtain a request from the terminal to process the data.

[0034] Before the cloud terminal sends data to the server, it must first obtain a data processing request from the terminal. This request comes from a physical terminal and is triggered by the user of the terminal.

[0035] In a possible design of the second aspect, the data processing method further includes:

[0036] The cloud terminal is used to obtain a request for processing the data, where the request is triggered by an application on the cloud terminal.

[0037] Before the cloud terminal sends data to the server, the cloud terminal obtains a data processing request from the application, which is related to the encoding or decoding requirements of the application layer. If the application needs encoding, an encoding request is issued, and if the application needs decoding, a decoding request is issued.

[0038] Among them, the processing rules in the request sent by the application are not the same as the processing rules sent by the cloud terminal to the server in the third possible implementation method of the second aspect of the present invention, that is, the cloud terminal can modify the processing rules and send the modified processing rules to the server. There are four ways of modification: software encoding to hardware encoding, hardware encoding to software encoding, software decoding to hardware decoding, and hardware decoding to software decoding. Among them, changing software encoding to hardware encoding and software decoding to hardware decoding can improve the efficiency of encoding and decoding, reduce the efficiency of cloud terminal CPU resources, and solve the problem of incompatibility when some applications call hardware encoding or decoding on the cloud terminal.

[0039] In a possible design of the second aspect, the data processing system further includes:

[0040] The cloud terminal is used to send the virtual desktop obtained according to the data to the terminal.

[0041] The application of the cloud terminal obtains a virtual desktop based on the processed data, and sends the virtual desktop to the terminal so that the terminal displays the virtual desktop to the user.

[0042] In a possible design of the second aspect, the data processing system also includes: the cloud terminal is a cloud phone.

[0043] In a third aspect, the present application provides a cloud terminal, which is a cloud terminal provided by the second aspect of the present application or any possible design of the second aspect.

[0044] In a fourth aspect, the present application provides a server, which is a server provided by the second aspect of the present application or any possible design of the second aspect.

[0045] In a fifth aspect, the present application provides a computing device, the computing device comprising a processor and a memory. The processor executes instructions in the memory, so that the computing device implements the cloud terminal or server provided by the second aspect of the present application or any possible design of the second aspect.

[0046] In a sixth aspect, the present application provides a computer program product, which includes instructions, and the instructions are used to implement the steps executed by the cloud terminal in the method provided by the first aspect of the application or any possible design of the first aspect, and / or to implement the steps executed by the server in the method provided by the first aspect of the application or any possible design of the first aspect.

[0047] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is an example schematic diagram of the system structure of data processing provided in an embodiment of the present application.

[0049] Figure 2 It is an example schematic diagram of the system structure of data processing provided in the embodiment of the present application.

[0050] Figure 3 It is an example schematic diagram of the system structure of data processing provided in an embodiment of the present application.

[0051] Figure 4 This is an example schematic diagram of the system structure of data processing provided by way of example in this application.

[0052] Figure 5 It is a flow chart of the data processing method provided in the embodiment of the present application.

[0053] Figure 6 It is an example schematic diagram of the system structure of data processing provided in an embodiment of the present application.

[0054] Figure 7 This is an example schematic diagram of the system structure of the data processing method provided in the embodiment of the present application.

[0055] Figure 8 It is an illustrative diagram of the data processing method framework provided in the embodiments of the present application.

[0056] Fig. 9 It is an example schematic diagram of the method involved in data processing in the cloud phone shopping live broadcast scenario provided in the embodiment of the present application.

[0057] Fig.10 It is an example structural diagram of the device 300 provided in an embodiment of the present application.

[0058] Fig.11 It is an example structural diagram of a computing device 600 provided in an embodiment of the application. DETAILED DESCRIPTION

[0059] Figure 1 is a schematic diagram of a data processing system structure provided in an embodiment of the present application, such as Figure 1 As shown, the system may include a cloud terminal 001 and a server 002 .

[0060] In one embodiment, the cloud terminal 001 may be a virtual device that provides a virtual desktop, for example, the virtual device may be an instance, and the instance may be a virtual machine or a container. In one possible implementation, the cloud terminal 001 is a cloud phone.

[0061] Server 002 may be a device with computing capabilities, and server 002 has a memory and a processor. The processor of server 002 may be a central processing unit (CPU). Optionally, server 002 may also have one or more of a graphics processing unit (GPU), a neural network processor (NPU), and a field programmable gate array (FPGA). The memory of server 002 may be a random access memory (RAM) or a solid-state drive (SSD) or other devices or memory instances with storage capabilities. A solid-state drive may also be called a solid-state drive (SSD).

[0062] In an embodiment of the present application, applications may be installed on the cloud terminal 001, and the applications may include live video broadcast, video call, video conference, video player, audio player, mobile game, etc. Optionally, the application on the cloud terminal 001 may issue a request for encoding or decoding, or may issue both encoding and decoding requests at the same time.

[0063] In one embodiment, the server 002 may be a physical device deployed in the network, and the cloud terminal 001 and the server 002 communicate based on a communication protocol or a call protocol, so that the cloud terminal 001 can call the service provided by the server 002 or exchange data with the server 002. For example, the communication protocol may be a socket. For example, the call protocol may be a remote procedure call (RPC).

[0064] In one implementation scenario, for example, the server 002 may be deployed with a software encoding module and a software decoding module. The software encoding module is used to encode data. The software decoding module is used to decode data. In one possible implementation, the encoding or decoding operation is run on the CPU of the server 002. In one possible implementation, the software encoding module and the software decoding module may be directly deployed in the server 002, or deployed in an instance running on the server 002.

[0065] In an implementation scenario example, the server 002 may be a device with hardware encoding capability, and the server 002 is deployed with a hardware encoding module and a hardware decoding module. The hardware encoding module is used to encode data. The hardware decoding module is used to decode data. In a possible implementation method, the encoding and decoding operations are run on a hardware encoding chip of the server 002, for example, the hardware encoding chip may be a GPU or an NPU or other chip with hardware encoding capability.

[0066] Figure 2 is a schematic diagram of the system structure of the second data processing provided in the embodiment of the present application, such as Figure 2 As shown, the server 002 can also be deployed in a cloud server cluster 003. The cloud server cluster 003 has multiple servers similar to the server 002. The server 002 of the cloud server cluster 003 can also be composed of a certain number of elastic cloud servers (ElasticCloud Server, ECS), that is, the number of servers 002 in the cloud server cluster 003 can be elastically scalable.

[0067] Figure 2 is a schematic diagram of the system structure of the third data processing provided by the embodiment of the present application, such as Figure 3 As shown, the server 002 can also be deployed in a data center 004.

[0068] In a possible embodiment, one or more servers may be included in the cloud server cluster 003 (or data center 004), and the server may provide encoding capabilities, or the server may provide decoding capabilities, or the server may provide both encoding and decoding capabilities. In one scenario, for example, the server used for encoding and the server used for decoding belong to different clusters or data centers. That is, the cluster composed of servers used for encoding is called an encoding cluster, the cluster composed of servers used for decoding is called a decoding cluster, the data center where the servers used for encoding are located is called an encoding data center, and the data center where the servers used for decoding are located is called a decoding data center.

[0069] Figure 4 This is another data processing system structure diagram provided as an example. Figure 4 As shown, based on Figures 1 to 3The system may include terminal 005 as described in any system structure diagram. In some embodiments, terminal 005 may be a mobile phone, tablet or other physical device with display and communication functions. Terminal 005 may communicate with cloud terminal 001 through the network. Terminal 005 uploads local data and instructions to cloud terminal 001. Cloud terminal 001 may send virtual desktop to terminal 005. Users may control the virtual desktop operating system running on cloud terminal 001 on terminal 005. For example, Figure 4 Server 002 in can also be Figure 3 and Figure 4 The cloud server cluster 003 or data center 004, for the implementation of the cloud server cluster 003 and the data center 004, please refer to the previous description.

[0070] For example, Figure 4 As shown, terminal 005 can collect video data locally and upload the collected video data to the software of cloud terminal 001. Cloud terminal 001 can send a service request (such as an encoding request or a decoding request) to server 002 according to the needs of the application, and forward the video data collected locally by terminal 005 to server 002; server 002 returns the processed video data to the application of cloud terminal 001. The application of the cloud terminal obtains a virtual desktop based on the processed video data, for example, generates a live broadcast interface based on the video data, and the live broadcast interface belongs to part or all of the content displayed on the virtual desktop. Cloud terminal 001 sends the virtual desktop to the display of terminal 005.

[0071] Figure 5 This is a flow chart of a data processing method provided by an embodiment of the present application. The method can be applied to Figures 1 to 4 in any of the systems shown. Figure 5 This method is applied to Figure 1 An embodiment of the system architecture shown, Figure 1 The system includes a cloud terminal and a server. Figures 2 to 4 System architecture ratio Figure 1 Complex, the steps of the flowchart of the method embodiment applied to them will be more Figure 5 More, but their core steps are still the same Figure 5 The flowcharts of the embodiments shown are similar. Figure 1 and Figure 5 The process of the method for processing data in the present application is described by way of example. The process steps include step 201 to step 205 .

[0072] Step 201: The cloud terminal sends data to the server.

[0073] The cloud terminal sends data to the server through the network, and the data is data to be decoded or encoded.

[0074] In a possible implementation manner, the cloud terminal sends an instruction indicating encoding and an instruction indicating decoding to the server, and the instruction includes an encoding or decoding rule.

[0075] Optionally, the instruction carries data to be encoded or decoded.

[0076] Optionally, the instruction and data (data to be encoded or decoded) are sent independently.

[0077] In a possible embodiment, the cloud terminal may be continuously sending a video stream data, and at the same time as starting to send the video stream, send an instruction to the server to start encoding. For example, after the video stream stops or the application sends a stop request, the cloud terminal will send an instruction to the server to close the encoding service.

[0078] For example, there are some applications installed on the cloud terminal. The applications can issue encoding or decoding requests according to their own business needs. For example, the live broadcast application can issue an encoding request, and the video playback application can issue a decoding request. The encoding method in the encoding request can be hardware encoding or software encoding. The decoding method in the decoding request can be hardware decoding or software decoding.

[0079] In a possible implementation, the cloud terminal determines the encoding or decoding rule according to the request sent by the application.

[0080] In a possible implementation, the encoding or decoding method in the determined rule may be different from the encoding or decoding rule in the request issued by the application. For example, the application issues a request for software encoding, and the determined encoding rule is hardware encoding, or the application issues a request for hardware encoding, and the determined encoding rule is software encoding, that is, the cloud terminal can modify the encoding method issued by the application.

[0081] In a possible implementation, the encoding or decoding method in the determined rule may also be the same as the encoding or decoding rule in the request issued by the application, that is, the cloud terminal may not modify the encoding method issued by the application.

[0082] Step 202: The server receives data.

[0083] For example, after receiving the data sent by the cloud terminal, the server needs to start the encoding or decoding task according to the instruction sent synchronously by the cloud terminal, and return the corresponding service response to the cloud terminal. In a possible embodiment, the server can be continuously receiving a video stream data. After receiving the start command, the server will start encoding or decoding the continuously received video frames, and return the service response of starting encoding or decoding to the cloud terminal, until the server receives the command to close the encoding or decoding task, and then returns the service response of ending encoding or decoding to the cloud terminal.

[0084] Step 203: The server performs encoding or decoding services.

[0085] For example, the server encodes or decodes the data according to the received encoding rules or decoding rules, wherein the encoding rules may include encoding standards, encoding parameters, and encoding methods, and the decoding rules may include decoding standards, decoding parameters, and decoding methods.

[0086] In an example implementation of encoding, the server receives an encoding request from a cloud terminal. The encoding request contains encoding standards such as H.264, H.264, etc. The encoding parameters may include resolution, bit rate, frame rate, DTS and PTS, etc. The encoding method may be software encoding or hardware encoding. For example, video encoding is the process of converting an original video format file into another video format file. The main purpose is to compress the original video for storage and transmission. The encoded data is a compressed frame, which has a smaller data volume than the original frame.

[0087] For example, if the encoding method is hardware encoding and decoding, the server calls a hardware encoding chip (such as a GPU chip) to perform encoding calculations on the original frame. If the encoding method is software encoding, the server calls a CPU to perform encoding calculations on the original frame.

[0088] The encoding method used by the server and the encoding method in the received instruction can be the same, but in some embodiments, the encoding method used by the server may be different from the encoding method in the request issued by the cloud terminal application. The cloud terminal performs some processing on the content in the request, which may include modifying the encoding method. Please refer to the description in step 201.

[0089] An example implementation of decoding, the server receives a decoding request from the cloud terminal, the decoding standard, decoding parameters, and decoding method can refer to the description of the encoding rules in the previous text. For example, video decoding is the reverse process of video encoding. It is the process of converting compressed video format files into original video format files through decompression technology. The main purpose is to decompress the compressed video frames to support the playback and display of the screen. The decoded data is the decoded frame (which may be lossy compared to the original frame). The decoded frame has a larger data volume than the compressed frame, and the visual effect after visualization is better.

[0090] For example, if the encoding method is hardware encoding and decoding, the server calls a hardware decoding chip, such as a GPU chip, to decode and calculate the compressed frame. If the decoding method is software decoding, the server calls a CPU to decode and calculate the compressed frame.

[0091] In some embodiments, the decoding method used by the server and the decoding method in the received instruction may be the same.

[0092] In some embodiments, the decoding method used by the server may be different from the decoding method in the request sent by the cloud terminal application. The cloud terminal performs some processing on the content in the request, which may include modifying the decoding method. For example, the decoding method indicated by the instruction is software decoding, but the decoding method used by the server is hardware decoding.

[0093] Step 204: The server sends the processed data.

[0094] The server sends the processed data to the cloud terminal through the network.

[0095] In an application scenario, the server sends the encoded frame to the cloud terminal, wherein the encoded frame is generated by a module responsible for encoding inside the server.

[0096] In an application scenario, the server sends a decoded frame to a cloud terminal, wherein the decoded frame is generated by a module responsible for decoding within the server.

[0097] In a possible implementation, the server continuously sends a processed video stream. In some possible embodiments, after receiving the stop codec instruction or the video stream transmission stops, the server will send a response to the cloud terminal at the same time after stopping the codec service.

[0098] Step 205: The cloud terminal receives the processed data.

[0099] The cloud terminal receives data returned by the server from the network. For example, the data may include processed decoded frames or compressed frames.

[0100] For example, in an application scenario, a video player application previously sent a decoding request. After receiving the decoded stream returned by the server, the cloud terminal forwards the data to the video player application at the application layer through the framework layer. The video player application presents the decoded video stream in the user interface of the application. In a possible implementation, the desktop content where the application user interface is located will be sent to a terminal.

[0101] As an example of an application scenario, a video call application has previously issued an encoding request. After receiving the encoded stream returned by the server, the cloud terminal forwards the data to the video playback application at the application layer through the framework layer. The video call application sends the encoded video stream to one or more other terminals through the network or stores it in the cloud. For example, the terminal used by the person who is having a video call with the current user will receive the encoded video stream.

[0102] Figure 6 It is a system structure diagram of a data processing method provided in an embodiment of the present application, including an internal structure diagram of a cloud terminal 001 and a server 002. The cloud terminal includes an application layer and an architecture layer, wherein the application layer includes various types of applications, and this embodiment mainly involves two types of applications 101 and 102. Application 101 requires the architecture layer to provide video encoding services, and application 102 requires the architecture layer to provide video decoding services. The architecture layer includes a codec management module 111, which is composed of a media codec interface, a software codec plug-in or a hardware codec plug-in and a corresponding control module. It is responsible for receiving encoding or decoding requests for applications such as live broadcast, video call, and video playback, and calling corresponding software or hardware plug-ins according to the differences in the requirements of different applications, and returning target video frame data. The communication module 114 of the cloud terminal and the communication module 121 of the server are used to transmit data and control instructions for the cloud terminal and the transcoding cluster in the network through a transmission protocol. The server transcoding management module 122 is responsible for receiving the encoding or decoding request of the cloud phone encoding or decoding module, and issuing the encoding or decoding task to the corresponding external encoding or decoder module, and is also responsible for receiving the encoded or decoded data and forwarding it to the communication module 121. The encoder module 123 is responsible for the actual video encoding work, receiving the original video frame, encoding it by the hardware encoding chip, and returning the compressed frame; the decoder set module 124 is responsible for the actual video decoding work, receiving the original compressed frame, decoding it by the hardware decoding chip, and returning the decoded frame. Optionally, the encoder module 123 and the decoder module 124 can also belong to the same functional module, which does not affect the solution of this embodiment.

[0103] In another embodiment, if Figure 7 FIG. 1 is a schematic diagram of the system structure of another data processing method provided by the present application. Figure 6, the codec service of the cloud terminal is carried by a cloud server cluster or a data center, rather than a physical server. In some possible embodiments, such a deployment method can enhance the service efficiency of the external codec. Figure 7 , the cloud server cluster 003 or data center 004 may include one or more management nodes 131 and two sub-clusters, namely, the encoding cluster 132 and the decoding cluster 133, which are responsible for the actual encoding work and the decoding work respectively. Optionally, the encoding cluster 132 and the decoding cluster 133 may also be merged or distributed in one or more service clusters. Figure 6 The codec management module of server 002, Figure 7 The management node 131 in the system may include two functions: communication and cluster management. Each management node may be composed of a server, which may not include an encoding or decoding module.

[0104] Figure 8 A data processing method framework example diagram provided in an embodiment of the present application is given. Figure 7 Further explanation of the system in. In some embodiments, the transcoding management module 111 is divided into a media encoding and transcoding interface, a software or hardware encoding and decoding plug-in, and a control unit according to the function, wherein the software or hardware transcoding plug-ins 115 and 116 are responsible for creating threads and calling the hardware transcoding control module 113 or the software transcoding module 112 to implement software hardware encoding and decoding. The hardware transcoding module 113 is used to implement specific control of hardware encoding or decoding, including starting the encoder or decoder, sending encoding or decoding parameters, sending video frames to be encoded or decoded, receiving encoded or decoded video frames, closing the encoder or decoder, etc. The software transcoding module 112 is used to implement specific control of software encoding or decoding, including starting the encoder or decoder, sending encoding or decoding parameters, sending video frames to be encoded or decoded, receiving encoded or decoded video frames, closing the encoder or decoder, etc.

[0105] Figure 8 401-421 shown in are some key steps involved in this embodiment. The size of the serial number does not represent the actual execution order, but only serves as an identifier to represent a certain action and step. If a certain identifier appears more than once, it means that the same or similar actions and steps are executed at the location of the identifier. Figure 8 The process of the method for processing data in the present application is described by way of example. The process steps include step 401 to step 420.

[0106] Step 401: Request video encoding or decoding.

[0107] The video encoding or decoding request is usually initiated by the application, and different types of applications will issue video encoding or decoding requests. For example, mobile live broadcast applications may issue encoding requests, video playback applications may issue software decoding requests, and cloud mobile games and video calls in mobile office scenarios may issue encoding and decoding requests at the same time.

[0108] Step 402: Call the software codec plug-in.

[0109] The media codec interface calls the software codec plug-in. In some embodiments, the media codec interface can be a native interface of the cloud terminal operating system. In order not to affect the call of the application layer, the native media codec interface can be retained.

[0110] In some embodiments, the software codec plug-in may be some existing plug-in, or may be a plug-in developed by the inventor himself.

[0111] Step 403: Call the hardware codec plug-in.

[0112] The media codec interface calls the hardware codec plug-in. The cloud phone in the prior art only supports step 402, which makes the software functions that some applications need to call the hardware codec unavailable. For example, the inventor can develop a hardware codec plug-in according to the needs. The embodiment of this method can realize the function of calling the hardware chip for encoding or decoding, and improve the compatibility of the cloud phone application software.

[0113] Step 404: The software codec plug-in calls the corresponding software codec control unit, creates threads, manages multiple streams, and implements overall control of hardware encoding or decoding work.

[0114] Step 405: The hardware codec plug-in calls the corresponding hardware codec control unit to create a thread and manage multiple streams to achieve overall control of the hardware encoding or decoding work.

[0115] Step 406: Send the original frame or the compressed frame.

[0116] The software codec control unit 112 sends the data of the original frame or compressed frame to be encoded or decoded to the client Socket, and optionally, it can also be the client RPC.

[0117] The hardware codec control unit 113 sends the data of the original frame or compressed frame to be encoded or decoded to the client Socket, and optionally, it can also be the client RPC.

[0118] Step 407: Start or stop the encoding or decoding service.

[0119] The software codec control unit sends a start or shutdown request, which is received by the client Socket (or RPC) and forwarded to the server Socket (or RPC) through the network to start or shut down the encoding or decoding service of the remote transcoding cluster.

[0120] The hardware decoding control unit sends a start or shutdown request, which is received by the client Socket (or RPC) and forwarded to the server Socket (or RPC) through the network to start or shut down the encoding or decoding service of the remote transcoding cluster.

[0121] In some embodiments, the request to enable or disable may be determined based on a request sent by an application (or user).

[0122] Step 408: Send the original frame or the compressed frame.

[0123] The client Socket (or RPC) sends the data of the encoded frame or decoded frame received from the network to the software encoding and decoding unit.

[0124] The client Socket (or RPC) sends the data of the encoded frame or decoded frame received from the network to the hardware encoding and decoding unit.

[0125] Step 409: Start or stop the response.

[0126] The client Socket (or RPC) receives the service response of the encoding or decoding cluster from the network and sends it to the software encoding and decoding unit to start or shut down the software encoding and decoding unit response operation.

[0127] The client Socket (or RPC) receives the service response of the encoding or decoding cluster from the network and sends it to the hardware decoding unit to start or shut down the response operation of the hardware decoding unit.

[0128] Step 410: Send a request instruction.

[0129] The server Socket (or RPC) sends the instruction received from the network to the transcoding cluster management 131. In some embodiments, the instruction may include starting encoding, starting decoding, closing encoding, closing decoding, etc.

[0130] Step 411: Send the original compressed frame or video frame.

[0131] The server Socket (or RPC) sends the original image frame or the encoded frame received from the network to the transcoding cluster management unit 122 .

[0132] Step 412: Return the processed decoded frame or compressed frame.

[0133] The transcoding cluster management 131 returns the processed decoded frame or compressed frame to the server Socket or RPC.

[0134] Step 413: Return a service response.

[0135] The transcoding cluster management unit returns a processing response, and the server Socket (or RPC) forwards the response.

[0136] Step 414: Send an instruction to start the encoder.

[0137] The transcoding cluster management 131 sends an instruction to start the encoder to the encoding cluster 132 , and the encoding cluster 132 starts encoding after receiving the start instruction.

[0138] Step 415: Send the original video frame.

[0139] The transcoding cluster management 131 sends the original video frames to the encoding cluster 132 .

[0140] Step 416: Send an instruction to close the encoder.

[0141] The transcoding cluster management 131 sends a command to shut down the encoder to the encoding cluster 132 , and the encoding cluster 132 stops transcoding after receiving the shut down command.

[0142] Step 417: Return the compressed frame.

[0143] The encoding cluster 132 returns the processed compressed frames to the transcoding cluster management 131 .

[0144] Step 418: Send a command to start the decoder.

[0145] The transcoding cluster management 131 sends a decoder startup instruction to the decoding cluster 133 , and the decoding cluster 133 starts decoding after receiving the startup instruction.

[0146] Step 419: Send an instruction to close the decoder.

[0147] The transcoding cluster management 131 sends a command to shut down the decoder to the decoding cluster 133 , and the decoding cluster 133 stops transcoding after receiving the start command.

[0148] Step 420: Send the original compressed frame.

[0149] The original compressed frames are sent to the decoding cluster 133 by the transcoding cluster management 131 .

[0150] Step 421, return the decoded frame.

[0151] The decoding cluster 133 returns the processed decoded frames to the transcoding cluster management 131 .

[0152] The relevant details of the above steps 401-421 can also be referred to the previous description.

[0153] Fig. 9 It is a schematic diagram of a method for cloud terminal data processing that may be involved in a cloud phone shopping live broadcast scenario provided in an embodiment of the present application. In some embodiments, first, according to the cloud phone usage scenario and scale, a certain number of cloud phones and a corresponding cloud server cluster must be purchased. The cloud server cluster can provide the encoding and decoding services described above. The cloud phone live broadcast requires a larger cloud server cluster than the mobile office scenario. Then, it is necessary to complete the configuration of the cloud phone to connect to the external transcoding cluster, such as the IP and port configuration of the Socket on the cloud phone. Finally, use the configured cloud phone and cloud server cluster for live broadcast and mobile office. Fig. 9 In the method diagram shown, serial numbers 501-514 represent the key steps involved in this embodiment, which describe how an embodiment of the method is to perform remote hardware video encoding services based on a cloud phone in a cloud phone shopping live broadcast, from user A starting the live broadcast, the shopping live broadcast application sending a software encoding request to user A closing the live broadcast.

[0154] Step 501: User A opens the shopping live broadcast application software on the cloud phone and starts live broadcasting. Figure 4 The visual interface of the terminal 005 shown remotely controls the application on the cloud terminal 001. For example, the terminal 005 can be a smart phone, including a camera or other shooting device, which is used to collect the user's video stream during the live broadcast in real time and transmit the video stream to the cloud phone synchronously.

[0155] Step 502: The shopping live broadcast application calls the MediaCodec interface to start the video encoding operation. The live broadcast application receives the video stream and sends a software encoding request to the MediaCodec interface to request the encoding service of the cloud server cluster;

[0156] Steps 503-404, the MediaCodec process calls the encoder control unit based on the software encoding plug-in to create an encoding thread.

[0157] Step 505: The encoder control unit generates a related instruction, which includes an encoding rule, and changes the encoding method in the encoding rule from software encoding to hardware encoding. Next, the encoder control unit establishes a socket connection with the external transcoding cluster through the client socket, and sends a request to start the external hardware encoder to the server socket.

[0158] Step 506: After receiving the hardware encoding instruction, the server socket of the cloud server cluster forwards it to the transcoding cluster management node. After receiving the instruction to start a specific external encoder, the transcoding cluster management node creates a thread to start the encoding cluster.

[0159] Step 507: The cloud phone encoder control unit sends a request to obtain encoding header information, and the client socket forwards the request to the client socket of the cloud server cluster.

[0160] Step 508: The transcoding cluster management node requests and obtains the encoding header from the encoder module of a specific server in the encoding cluster, and forwards it to the cloud phone through the network.

[0161] Step 509: The cloud phone encoder control unit receives a response to the encoding header information.

[0162] Step 510: The cloud phone encoder control module sends the original video frame data to the transcoding management node of the cloud server cluster, requesting the external encoding cluster to perform hardware encoding.

[0163] Step 511: The transcoding cluster management node is responsible for allocating an encoding server to the cloud phone, and the cloud phone sends the original video frame to the encoder of the specific server.

[0164] Step 512: The encoder returns the encoded compressed video frame to the transcoding cluster management node and forwards it to the cloud phone through the network.

[0165] Step 513: The encoder module of the cloud phone receives the compressed video frame data through the socket and returns it to the live broadcast application. The live broadcast application on the cloud phone forwards the compressed frame data stream to the public cloud. More network users can download the compressed frame data stream on their mobile phones or PCs and other terminals, and after decoding it locally, they can watch the shopping live broadcast in real time.

[0166] Step 514: User A closes the shopping live broadcast application on the cloud phone and ends the live broadcast.

[0167] Step 515: The shopping live broadcast application software ends the video encoding operation by calling the MediaCodec interface.

[0168] Steps 516-517, the MediaCodec process calls the encoder control unit based on the software encoding plug-in to request to end encoding;

[0169] Step 518: The encoder control unit releases the remote encoder, sends a request to shut down the server encoder, and closes the socket connection with the cloud server cluster.

[0170] Step 519: The transcoding cluster management node shuts down the encoding service of the specific server.

[0171] Fig.10 An embodiment of the present application provides a device 300 deployed on a server 002 , and the device 300 includes a receiving unit 301 , a processing unit 302 , and a sending unit 303 .

[0172] The receiving unit 301 is used to receive data from the cloud terminal 001. In some embodiments, the data may be data to be encoded or decoded. In other embodiments, the data may also include instructions for indicating encoding or decoding tasks. Please refer to the relevant description above for the contents of the instructions.

[0173] The processing unit 302 is used to encode or decode data. In an implementation scenario example, the processing unit encodes the data to be encoded, and the encoding method includes hardware encoding or software encoding. Please refer to the above description. In an implementation scenario example, the processing unit decodes the data to be decoded, and the decoding method includes hardware decoding or software decoding. Please refer to the above description.

[0174] The sending unit 303 is used to send data to the cloud terminal 001. The data sent is decoded or encoded data.

[0175] Optionally, the processing unit 302 is configured to perform hardware encoding on the data or software encoding on the data.

[0176] Optionally, the processing unit 302 is configured to perform hardware decoding on the data or software decoding on the data.

[0177] The present application embodiment also provides a computing device 600, such as Fig.11 The computing device 600 may be a device for deploying the cloud terminal 001 , or the computing device 600 may be a server 002 .

[0178] Fig.11 A schematic diagram of the structure of a computing device 600 is provided, and the computing device 600 includes a bus 601, a processor 602, and a memory 603. The processor 602 and the memory 603 communicate with each other via the bus 601.

[0179] The bus 601 may be a peripheral component interconnection standard bus or an extended industrial standard structure bus, etc. The processor 602 may be a central processing unit (CPU), in addition, there may be one or more of a graphics processing unit (GPU), a neural network processor (NPU), and an FPGA (Field Programmable Gate Array). The memory 603 may include one or more of a random access memory (RAM), a read-only memory (ROM), a flash memory, a mechanical hard disk (HDD), or a solid state drive (SSD). The memory 603 stores an executable program code, and the processor 602 executes the executable program code to implement the functions of the aforementioned cloud terminal 001 or the functions of the server 002, or executes the steps performed by the terminal 001 or the steps performed by the server 002 in the method described in the aforementioned embodiment.

[0180] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

Claims

1. A method for processing data, characterized in that: include: The cloud terminal obtains a request for processing data, the request is triggered by an application on the cloud terminal, the request includes a first processing rule issued by the application, the first processing rule is used to determine whether to perform hardware encoding or hardware decoding on the data, or the first processing rule is used to determine whether to perform software encoding or software decoding on the data; The cloud terminal sends the data; The server receives the data; The cloud terminal sends a second processing rule to the server, the second processing rule being the same as or different from the first processing rule, the second processing rule being used to determine whether to perform hardware encoding or hardware decoding on the data, or the second processing rule being used to determine whether to perform software encoding or software decoding on the data; The server processes the data, wherein the processing is encoding the data or decoding the data; The server sends the processed data to the cloud terminal.

2. The method according to claim 1, characterized in that The encoding of the data comprises: hardware encoding the data; or, The data is software encoded.

3. The method according to claim 1, characterized in that Decoding the data comprises: performing hardware decoding on the data; or, The data is software decoded.

4. The method according to any one of claims 1 to 3, characterized in that: The method comprises: Before the cloud terminal sends the data, the cloud terminal obtains a request from a terminal to process the data.

5. The method according to any one of claims 1 to 4, characterized in that: The method comprises: The cloud terminal sends a virtual desktop obtained according to the data to the terminal.

6. A system for processing data, characterized in that: The system comprises: A cloud terminal, used for obtaining a request for processing data, wherein the request is triggered by an application on the cloud terminal, and the request includes a first processing rule issued by the application, wherein the first processing rule is used to determine whether the data is to be hardware encoded or hardware decoded, or the first processing rule is used to determine whether the data is to be software encoded or software decoded; sending the data to a server; sending a second processing rule to the server, wherein the second processing rule is the same as or different from the first processing rule, and the second processing rule is used to determine whether the data is to be hardware encoded or hardware decoded, or the second processing rule is used to determine whether the data is to be software encoded or software decoded; The server is used to receive the data, process the data, and send the processed data to the cloud terminal, wherein the processing is encoding the data or decoding the data.

7. The system according to claim 6, characterized in that The server is used to perform hardware encoding on the data, or perform software encoding on the data.

8. The system according to claim 6, characterized in that The server is used to perform hardware decoding on the data, or perform software decoding on the data.

9. The system according to any one of claims 6 to 8, characterized in that: The cloud terminal is used to obtain a request from a terminal to process the data.

10. The system according to any one of claims 6 to 9, characterized in that: The cloud terminal is used to send a virtual desktop obtained according to the data to the terminal.

11. The system according to any one of claims 6 to 10, characterized in that: The cloud terminal is a cloud phone.

12. A computing device, characterized in that: The computing device includes a processor and a memory; the processor executes instructions in the memory, so that the computing device implements the steps performed by the cloud terminal in the system described in any one of claims 6 to 11.

13. A computer program product, characterized in that The computer program product comprises instructions for implementing the steps performed by the cloud terminal in the method according to any one of claims 1 to 5.