Method and device for reducing flow consumption of cloud mobile phone by using super-resolution and medium

Abstract

The invention discloses a method and device for reducing cloud mobile phone traffic consumption by using super-resolution and a medium. The method comprises the steps that a cloud mobile phone server pushes a data stream to a cloud mobile phone APP, stores a current frame, namely an original frame, then pushes the current frame to a picture detection module, compares whether a current frame picture is different from a previous frame picture or not, and directly skips if the current frame picture is a first frame picture; if the difference exists, the image is pushed to an image super-resolution module for image super-resolution; if no difference exists, continuing to use the frame after the last super-resolution as an output image; and finally, pushing the super-divided frame to a playing module. According to the invention, the traffic consumption of the cloud mobile phone is reduced, and 480P and 720P data streams can be super-distinguished to 720P and 1080P through a local real-time super-resolution technology, and can be effectively reduced by 2.25 times, so that the real-time data transmission quantity in any scene is reduced, the traffic use cost of a user is reduced, the problem of use interruption caused by insufficient traffic is avoided, and the user experience is improved. And meanwhile, the picture fluency when the network bandwidth is limited is improved.

Description

Technical Field

[0001] This invention relates to the field of cloud phone technology, and specifically to a method, device, and storage medium for reducing cloud phone data consumption using super-resolution. Background Technology

[0002] Currently, cloud phones support 720P and 1080P resolutions for their data streams, with a fixed refresh rate of 60fps. Despite having a dynamic bitrate adjustment mechanism, the following significant issues remain in practical applications: 1) Excessive data consumption In 1080P resolution scenarios, when users watch videos, the data transmission volume remains high due to the large amount of dynamic details and color changes in the video, averaging 1.5-3GB per hour. When playing games, the real-time rendering and rapid scene transitions lead to even more frequent data interaction, resulting in an average data consumption of 3-6GB per hour. This high data consumption not only significantly increases mobile data costs, especially for users with limited data plans, who are prone to running out of data prematurely and being forced to interrupt their use; furthermore, in environments with limited network bandwidth, large data transmissions can also cause network congestion, resulting in stuttering, latency, and other problems, severely impacting the user experience. 2) Higher network requirements Because cloud phones require continuous transmission of large amounts of data through their streaming mechanism, they place high demands on network quality and speed. In crowded areas or with poor network signal, the streaming mechanism can cause lag and stuttering. Therefore, reducing the bandwidth requirements of cloud phones is a pressing issue that needs to be addressed. Summary of the Invention

[0003] The present invention proposes a method, device, and storage medium for reducing cloud phone data consumption using super-resolution, which can at least solve one of the technical problems in the background art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A method for reducing cloud phone data consumption using super-resolution includes the following steps: S1. The cloud phone server pushes the data stream to the cloud phone APP, stores the current frame (original frame), and then pushes it to the screen detection module. It uses perceptual hashing technology to compare whether there is a difference between the current frame and the previous frame. If it is the first frame image, the comparison is skipped and only processed. S2. By distinguishing scenes, different image difference rates are set according to static and dynamic scenes. If there is a difference, the image is pushed to the image super-resolution module for super-resolution. If there is no difference, the previous super-resolution frame is used as the output image. S3. Finally, the super-resolution frames are pushed to the playback module.

[0005] Furthermore, the image detection module performs the following steps: Comparing the image pixels of the current frame and the previous frame, the perceptual hashing (pHash) algorithm is used instead of the mean hashing (aHash). By reducing the image to an 8×8 grayscale image, calculating the DCT transform and taking the low-frequency coefficients, a 64-bit hash value is generated. This can effectively ignore misjudgments caused by minor noise in the image (such as screen brightness fluctuations and pixel-level color shifts). The comparison efficiency is improved by more than 300% compared to pixel-by-pixel calculation (single frame detection time ≤1ms, meeting the 60fps refresh rate requirement). If they are completely consistent, the current frame is replaced with the previous super-resolution image. If they are inconsistent, the current frame is sent to the image super-resolution module. Judging image consistency by scene: 1. Static scenes (such as reading apps, settings interfaces): Set the pixel difference rate threshold to 1% to reduce high-frequency false detections; 2. Dynamic scenes (such as games and videos): Set the pixel difference rate threshold to 0.3% to avoid missing rapid and small changes; 3. Threshold adjustment logic: Combine mobile phone scene recognition to determine the current scene status (for example, read the package name of the APP in the cloud phone ("com.tencent.qqmusic" for music scene, "com.netease.game.onmyoji" for game scene), select different thresholds according to the current scene, and ensure that the detection accuracy is ≥98% in different scenes.

[0006] After detection is complete, the current frame image is stored as the original frame image.

[0007] Furthermore, the image super-resolution module performs the following steps: Run a local super-resolution model, using a lightweight convolutional neural network (CNN) model (such as FSRCNN-Lite, ESPCN-Mobile) instead of a heavyweight model (such as SRGAN, SwinIR), to ensure real-time processing speed (inference time ≤3ms for a single frame from 480P to 720P, and ≤5ms for 720P to 1080P, meeting the requirements for 60fps real-time playback). Super-resolution the transmitted image: if the transmitted image is 480P resolution, it is super-resolution to 720P; if the transmitted image is 720P resolution, it is super-resolution to 1080P. The super-resolution image is stored as the current super-resolution frame.

[0008] When the mobile device's computing power is insufficient (e.g., CPU utilization ≥ 85%), the "resolution downgrade strategy" is automatically triggered: For example, with an input of 480P, the super-resolution target temporarily drops to 640P (instead of 720P), and with an input of 720P, the super-resolution target drops to 840P (instead of 1080P), reducing inference time by 40%. At the same time, "image quality compensation" (such as edge sharpening and color enhancement) is retained to ensure that the subjective experience of image quality does not significantly decrease after downgrading. After over-resolution, the image is then adapted to the screen size through "image cropping / scaling" according to the display resolution of the cloud phone APP (such as a mobile phone screen of 2340×1080) to avoid stretching and distortion. When the device's computing power recovers, it will automatically restore the original super-resolution target.

[0009] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0010] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0011] As can be seen from the above technical solution, the method of reducing cloud phone data consumption by utilizing super-resolution in this invention aims to specifically solve the above-mentioned problems existing in existing cloud phone technology, and is specifically implemented as follows: 1) Reduce cloud phone data consumption: Through local real-time super-resolution technology, 480P and 720P data streams can be super-resolutioned to 720P and 1080P respectively, effectively reducing data usage by up to 55% (2.25 times). The Sobel operator extracts edge regions of the super-resolution image (such as text, icons, and game UI), sharpening edge pixels by 1.2 times (non-edge regions are not sharpened to avoid noise amplification), resolving edge blurring issues after super-resolution (e.g., improving the clarity of game character outlines and video subtitles by 15%). When the dynamic bitrate of the cloud phone server decreases due to insufficient network bandwidth (e.g., latency exceeding 150ms, or packet loss / retransmission rate greater than 10%), the super-resolution module automatically triggers "image quality compensation mode": dynamically increasing edge sharpening intensity (from 1.2 times to 1.5 times) to compensate for blurriness; thereby reducing real-time data transmission in any scenario, thus lowering user data usage costs, avoiding interruptions due to insufficient data, and improving image smoothness when network bandwidth is limited. 2) When the cloud phone is running statically (when the image change detection is within the threshold), reduce the CPU load when there are no image changes on the static page. By introducing an image change detection mechanism, unnecessary super-resolution processing operations are stopped when the page content has not changed, thereby reducing the amount of invalid CPU computation. Based on experience, this can reduce the CPU load by an average of 30-70% (compared to a fully real-time super-resolution scenario), thereby reducing device power consumption, extending the battery life of mobile devices, improving the system's response speed to user operations, and enhancing the stability of device operation. Attached Figure Description

[0012] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0014] like Figure 1 As shown in the figure, the method for reducing cloud phone data consumption using super-resolution described in this embodiment involves the following steps being executed by a computer device: S1. The cloud phone server pushes a data stream to the cloud phone APP, stores the current frame (original frame), and then pushes it to the image detection module (which uses perceptual hashing technology to calculate, reduces the image to an 8*8 grayscale image and combines it with the DCT algorithm for calculation). Based on the APP package name currently running on the cloud phone, it uses a dynamic threshold or a static threshold (default static threshold) to compare whether the difference between the current frame and the previous frame is within the threshold (the first frame image is processed directly without comparison). S2. If there is a difference, the image is pushed to the image super-resolution module for super-resolution. Specifically, the 480P or 720P image is super-resolution to 720P and 1080P respectively (if the CPU utilization is greater than or equal to 85%, it is super-resolution to 640P and 840P respectively, and then the image is stretched to the current cloud phone screen resolution size), and the super-resolution image (super-resolution frame) is stored. If there is no difference, the previous super-resolution frame is used as the output image.

[0015] S3. Finally, the super-resolution frames are pushed to the playback module; The specific execution process of the image detection module is as follows: It compares the image pixels of the current frame and the previous frame. By employing a perceptual hashing (pHash) algorithm instead of a mean hashing (aHash), and by reducing the image to an 8×8 grayscale image, calculating the DCT transform, taking the low-frequency coefficients, and generating a 64-bit hash value, it can effectively ignore misjudgments caused by minor image noise (such as screen brightness fluctuations and pixel-level color shifts). The comparison efficiency is improved by more than 300% compared to pixel-by-pixel calculation (single frame detection time ≤1ms, meeting the 60fps refresh rate requirement). If they are completely identical, the current frame is replaced with the previous super-resolution image; if they are inconsistent, the current frame is sent to the image super-resolution module. After detection, the current frame image is stored as the original frame image. Image super-resolution module: Runs a local super-resolution model, a lightweight convolutional neural network (CNN) model (such as FSRCNN-Lite, ESPCN-Mobile), rather than a heavyweight model (such as SRGAN, SwinIR), ensuring real-time processing speed (inference time ≤3ms for a single frame from 480P to 720P, and ≤5ms for 720P to 1080P, meeting 60fps real-time playback requirements). It super-resolutions the transmitted image; if the transmitted image is 480P, it super-resolutions it to 720P; if the transmitted image is 720P, it super-resolutions it to 1080P. The super-resolution image is stored as the current super-resolution frame. In summary, the present invention has the following characteristics: Real-time super-resolution technology runs on the actual mobile phone and is applied in the cloud phone APP, reducing the data requirements of cloud phone streaming, while achieving a screen display with almost the same resolution as the original streaming on the terminal. Solutions and methods for implementing super-resolution applications on cloud phones; A method to reduce the CPU load on a mobile phone without performing super-resolution while keeping the current frame and the previous frame unchanged. The specific real-time super-resolution method and image detection module are not protected. They can be any super-resolution model, such as CNN. The image detection method is relatively simple and does not require protection. In practical applications, if the screen of a mobile phone is displayed on a third-party device, such as a screen or projector, the third-party device may be able to achieve the same local super-resolution effect as the mobile phone through super-resolution.

[0016] Table 1

[0017] As shown in Table 1 above, the first column, the number of pixels per frame, represents the total number of pixels in different resolution scenarios. For example, 1080P has 1920*1080=2073600 pixels. The average bitrate (assuming the encoding format is H264, the bitrate will vary depending on the device, this is just an example) is 5-8Mbps. The average data consumption per hour is 2.25-3.6GB. The last column is the percentage of data usage reduction (compared to the previous row).

[0018] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0019] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0020] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0021] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0023] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0024] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reducing cloud phone data consumption using super-resolution, characterized in that, Includes the following steps, S1. The cloud phone server pushes the data stream to the cloud phone APP, stores the current frame (original frame), and then pushes it to the screen detection module to compare whether there is a difference between the current frame and the previous frame. If it is the first frame image, it is skipped directly. S2. If there is a difference, push it to the image super-resolution module for image super-resolution; if there is no difference, use the previous super-resolution frame as the output image. S3. Finally, the super-resolution frames are pushed to the playback module.

2. The method for reducing cloud phone data consumption using super-resolution according to claim 1, characterized in that: The steps performed by the image detection module include: Compare the image pixels of the current frame and the previous frame. Use a perceptual hashing algorithm instead of mean hashing. The algorithm reduces the image to an 8×8 grayscale image, calculates the DCT transform, takes the low-frequency coefficients, and generates a 64-bit hash value. If they are completely consistent, the current frame is replaced with the previous super-resolution image. If they are inconsistent, the current frame is sent to the image super-resolution module. After detection is complete, the current frame image is stored as the original frame image.

3. The method for reducing cloud phone data consumption using super-resolution according to claim 2, characterized in that: The image detection module also includes scene-specific consistency checks during its execution steps. Static scenarios include reading apps and settings interfaces: the pixel difference rate threshold is set to 1% to reduce high-frequency false detections; Dynamic scenes include games and videos: the pixel difference rate threshold is set to 0.3% to avoid missing rapid, small changes; Threshold adjustment logic: Combine mobile phone scene recognition to determine the current scene state, select different thresholds according to the current scene, and ensure that the detection accuracy is ≥98% under different scenes.

4. The method for reducing cloud phone data consumption using super-resolution according to claim 1, characterized in that: The image super-resolution module performs the following steps: The local super-resolution model uses a lightweight convolutional neural network model instead of a heavyweight model to ensure real-time processing speed: inference time from 480P to 720P ≤ 3ms, and from 720P to 1080P ≤ 5ms, meeting the requirements for 60fps real-time playback. It super-resolutions the transmitted image: if the transmitted image is 480P, it is super-resolutiond to 720P; if the transmitted image is 720P, it is super-resolutiond to 1080P. The super-resolution image is then stored as the current super-resolution frame. When the mobile device's computing power is insufficient, a resolution degradation strategy is automatically triggered; At the same time, image quality compensation is retained to ensure that the subjective experience of image quality does not decrease significantly after downgrading. After super-resolution, the image is cropped / scaled to adapt to the screen size according to the display resolution of the cloud phone APP to avoid stretching and distortion. When the device's computing power recovers, it will automatically restore the original super-resolution target.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 4.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 4.

Images