Picture processing method, apparatus, device, medium and program product

Abstract

The present disclosure relates to a picture processing method, device, equipment, medium and program product. According to an example of the present disclosure, the method comprises: determining at least two types of sub-regions from a screen region based on historical touch data, the historical touch data at least comprising touch point coordinates, the density of touch points in the at least two types of sub-regions being different; rendering the to-be-displayed content of the at least two types of sub-regions using different rendering modes, wherein the rendering accuracy corresponding to the sub-region with higher density of touch points is higher; and synthesizing and displaying a display picture according to the to-be-displayed content of each type of sub-region after rendering. The present disclosure can avoid full-screen high-precision rendering, can improve the flexibility of picture rendering, thereby saving the computing resources and operation time of the electronic device, reducing the performance requirements and system power consumption of the electronic device, and further improving the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of terminal display technology, and in particular to a screen processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the rapid development of display technology in terminal devices, high-resolution screens (such as 4K / 8K displays and high refresh rate OLED panels) have become standard hardware in consumer electronics products such as smartphones and tablets. At the same time, users' demands for image display quality continue to rise, especially in interactive scenarios (such as games, video editing, and AR / VR applications), where high-definition, low-latency visual experiences have become a core competitive advantage for device performance. To match hardware display capabilities with user needs, rendering technology, as a key element in improving image quality, urgently needs further optimization in areas such as resolution enhancement, dynamic detail restoration, and real-time response.

[0003] In related technologies, image quality enhancement solutions mainly include two types: hardware-driven and software rendering. Software rendering solutions primarily use image processing algorithms (such as super-resolution reconstruction, anti-aliasing, and dynamic frame interpolation) to post-process low-resolution or low-quality images, simulating high-resolution output in software. While this type of solution avoids the cost of hardware upgrades, its computational complexity and rendering load increase exponentially, especially when handling full-screen high-precision rendering, easily leading to overheating of terminal devices and reduced battery life. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, the present disclosure provides a screen processing method, apparatus, device, medium, and program product to solve the defects in the related technologies.

[0005] According to a first aspect of the present disclosure, a screen processing method is provided, the method comprising:

[0006] Based on historical touch data, at least two types of sub-regions are determined from the screen area. The historical touch data includes touch data generated by historical touch operations on the screen area. The historical touch data includes at least touch point coordinates. The density range of touch points in the at least two types of sub-regions is different.

[0007] The content to be displayed in the at least two types of sub-regions is rendered using different rendering methods, wherein the sub-region with a higher density of touch points corresponds to a higher rendering accuracy;

[0008] The display screen is synthesized and displayed based on the content to be displayed after rendering in various sub-regions.

[0009] In some embodiments, determining at least two types of sub-regions from the screen area based on historical touch data includes:

[0010] The touch point density distribution data of the screen area is determined based on the touch point coordinates, and the degree of aggregation of the touch point density distribution data is used to reflect the density of touch points in the screen area.

[0011] At least two types of sub-regions are determined in the screen area based on the density of touch points in the screen area, wherein the density of touch points in the same type of sub-region belongs to the same preset range.

[0012] In some embodiments, determining the touch density distribution data of the screen area based on the touch coordinates includes:

[0013] Map the coordinates of the touch point to the coordinate system of the screen area;

[0014] Calculate the contact density value at each point in the coordinate system.

[0015] In some embodiments, the method further includes:

[0016] In response to acquiring current touch data, a current touch event is determined based on the current touch data, wherein the current touch data includes touch data generated by the current touch operation on the screen area;

[0017] In response to the current touch event being a first type of event, the operation of rendering the content to be displayed in the at least two types of sub-regions using different rendering methods is performed. The first type of event includes pre-specified touch events that prioritize image quality.

[0018] In some embodiments, the method further includes:

[0019] In response to the current touch event being a second type of event, the content to be displayed in the at least two types of sub-regions is rendered using a preset resolution, which is lower than the smaller of the original image resolution and the screen physical resolution. The second type of event includes pre-specified touch events that prioritize response speed.

[0020] In some embodiments, the current touch data includes at least one of touch type, touch coordinates, touch time, and touch index;

[0021] Determining the current touch event based on the current touch data includes:

[0022] The current touch data is standardized to obtain standardized touch data. The content of the standardized touch data includes at least one of touch type, touch point coordinates, duration, acceleration direction, and touch index.

[0023] The standardized touch data is sorted in ascending order by touch time;

[0024] The sorted data is input into a preset touch event recognition model to obtain the current touch event.

[0025] In some embodiments, one of the different rendering methods is super-resolution rendering;

[0026] Super-resolution rendering is performed on the content to be displayed in the first type of sub-region within the at least two types of sub-regions, including:

[0027] Determine the target resolution based on the scaling factor of the current image;

[0028] Based on the target resolution, the content to be displayed in the first type of sub-region is reconstructed using a preset super-resolution algorithm, which includes a deep learning model and / or an interpolation algorithm.

[0029] In some embodiments, one of the different rendering methods is native resolution rendering;

[0030] Rendering the content to be displayed in the second type of sub-region within the at least two types of sub-regions at native resolution includes:

[0031] The content to be displayed in the second type of sub-region is rendered based on the original image resolution or the screen physical resolution;

[0032] The content to be displayed in the rendered second sub-region is scaled according to the scaling factor of the current screen.

[0033] In some embodiments, the method further includes performing the operation of determining at least two types of sub-regions from the screen area based on historical touch data in response to satisfying preset area determination conditions;

[0034] The conditions for satisfying the preset region determination include at least one of the following:

[0035] The current application is the target application;

[0036] The time interval since the first detected touch operation after the current application was launched has reached a preset duration;

[0037] The time interval since the last determination of at least two types of sub-regions from the screen area reaches a preset time interval.

[0038] According to a second aspect of the present disclosure, a screen processing apparatus is provided, the apparatus comprising:

[0039] A region determination module is used to determine at least two types of sub-regions from a screen area based on historical touch data. The historical touch data includes touch data generated by historical touch operations on the screen area, and the historical touch data includes at least touch point coordinates. The density range of touch points in the at least two types of sub-regions is different.

[0040] The content rendering module is used to render the content to be displayed in the at least two types of sub-regions using different rendering methods, wherein the sub-regions with higher touch density have higher rendering accuracy.

[0041] The composite display module is used to composite and display the display image based on the content to be displayed after rendering various sub-regions.

[0042] In some embodiments, the region determination module includes:

[0043] A distribution determination unit is used to determine the touch density distribution data of the screen area based on the touch coordinates, wherein the degree of aggregation of the touch density distribution data is used to reflect the density of touches in the screen area.

[0044] The region determination unit is used to determine at least two types of sub-regions in the screen region based on the density of touch points in the screen region, wherein the density of touch points in the same type of sub-region belongs to the same preset range.

[0045] In some embodiments, the distribution determination unit is further configured to:

[0046] Map the coordinates of the touch point to the coordinate system of the screen area;

[0047] Calculate the contact density value at each point in the coordinate system.

[0048] In some embodiments, the apparatus further includes:

[0049] A touch event determination module is used to determine a current touch event based on the current touch data obtained in response to acquiring the current touch data. The current touch data includes touch data generated by the current touch operation on the screen area.

[0050] The content rendering module is also used to respond to the current touch event being a first type event by performing the operation of rendering the content to be displayed in the at least two types of sub-regions using different rendering methods. The first type event includes a pre-specified touch event that focuses more on image quality.

[0051] In some embodiments, the content rendering module is further configured to render the content to be displayed in the at least two types of sub-regions using a preset resolution in response to the current touch event being a second type of event. The preset resolution is lower than the smaller of the original image resolution and the screen physical resolution. The second type of event includes a pre-specified touch event that prioritizes response speed.

[0052] In some embodiments, the current touch data includes at least one of touch type, touch coordinates, touch time, and touch index;

[0053] The touch event determination module includes:

[0054] The standardization unit is used to standardize the current touch data to obtain standardized touch data. The content of the standardized touch data includes at least one of touch type, touch point coordinates, duration, acceleration direction, and touch index.

[0055] A data sorting unit is used to sort the standardized touch data in ascending order of touch time;

[0056] The event determination unit is used to input the arranged data into a preset touch event recognition model to obtain the current touch event.

[0057] In some embodiments, one of the different rendering methods is super-resolution rendering;

[0058] The content rendering module includes a first rendering unit;

[0059] The first rendering unit is used for:

[0060] Determine the target resolution based on the scaling factor of the current image;

[0061] Based on the target resolution, the content to be displayed in the first type of sub-region is reconstructed using a preset super-resolution algorithm, which includes a deep learning model and / or an interpolation algorithm.

[0062] In some embodiments, one of the different rendering methods is native resolution rendering;

[0063] The content rendering module includes a second rendering unit;

[0064] The second rendering unit is used for:

[0065] The content to be displayed in the second type of sub-region is rendered based on the original image resolution or the screen physical resolution;

[0066] The content to be displayed in the rendered second sub-region is scaled according to the scaling factor of the current screen.

[0067] In some embodiments, the region determination module is further configured to perform the operation of determining at least two types of sub-regions from the screen region based on historical touch data in response to satisfying a preset region determination condition;

[0068] The conditions for satisfying the preset region determination include at least one of the following: the current application is the target application;

[0069] The time interval since the first detected touch operation after the current application was launched has reached a preset duration;

[0070] The time interval since the last determination of at least two types of sub-regions from the screen area reaches a preset time interval.

[0071] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising:

[0072] Processor and memory used to store computer programs;

[0073] The processor is configured to implement any of the above-described screen processing methods when executing the computer program.

[0074] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the screen processing method described in any of the preceding claims.

[0075] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the screen processing method described in any of the preceding claims.

[0076] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0077] This disclosure determines at least two types of sub-regions from the screen area based on historical touch data, and renders the content to be displayed in the at least two types of sub-regions using different rendering methods. Among them, the sub-regions with higher touch density correspond to higher rendering precision. Then, the display screen is synthesized and displayed based on the content to be displayed after rendering in each type of sub-region. This can achieve differentiated rendering of areas of interest and areas of non-interest to the user, avoid full-screen high-precision rendering, improve the flexibility of screen rendering, and thus save the computing resources and processing time of electronic devices while ensuring screen quality. It also reduces the performance requirements of electronic devices and system power consumption, thereby improving the user experience.

[0078] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0079] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0080] Figure 1A This is a flowchart illustrating a screen processing method according to an exemplary embodiment of the present disclosure;

[0081] Figure 1B This is a schematic diagram illustrating an application scenario of a screen processing method according to an exemplary embodiment of the present disclosure;

[0082] Figure 2A This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to determine at least two types of sub-regions from a screen area based on historical touch data;

[0083] Figure 2B This is a schematic diagram illustrating the process of dividing ROI and non-ROI according to an exemplary embodiment of the present disclosure;

[0084] Figure 3 This is a flowchart illustrating a screen processing method according to yet another exemplary embodiment of the present disclosure;

[0085] Figure 4 This is a flowchart illustrating how to determine the current touch event based on the current touch data, according to an exemplary embodiment of this disclosure;

[0086] Figure 5 This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to perform super-resolution rendering on the content to be displayed in the first type of sub-region within the at least two types of sub-regions;

[0087] Figure 6 This is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to render the content to be displayed in the second type of sub-region within the at least two types of sub-regions at native resolution;

[0088] Figure 7 This is a block diagram illustrating a screen processing apparatus according to an exemplary embodiment of the present disclosure;

[0089] Figure 8 This is a block diagram illustrating yet another screen processing apparatus according to an exemplary embodiment of the present disclosure;

[0090] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0091] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0092] With the rapid development of display technology in terminal devices, high-resolution screens (such as 4K / 8K displays and high refresh rate OLED panels) have become standard hardware in consumer electronics products such as smartphones and tablets. At the same time, users' demands for image display quality continue to rise, especially in interactive scenarios (such as games, video editing, and AR / VR applications), where high-definition, low-latency visual experiences have become a core competitive advantage for device performance. To match hardware display capabilities with user needs, rendering technology, as a key element in improving image quality, urgently needs further optimization in areas such as resolution enhancement, dynamic detail restoration, and real-time response.

[0093] Among related technologies, image quality enhancement solutions mainly include two types: hardware-driven and software rendering. Software rendering solutions primarily use image processing techniques (such as super-resolution reconstruction, anti-aliasing, and dynamic frame interpolation) to post-process low-resolution or low-quality images, simulating high-resolution output in software. While this type of solution avoids the cost of hardware upgrades, its computational complexity and rendering load increase exponentially, especially when handling full-screen high-precision rendering, easily leading to overheating of terminal devices and reduced battery life.

[0094] As one of the mainstream image processing technologies, super-resolution (SR) reconstruction technology aims to reconstruct low-resolution images or videos into high-resolution content through algorithms. Its core value lies in overcoming the limitations of hardware physical resolution. This technology is widely used in fields such as enhancing the virtual resolution of electronic devices (such as mobile phones), digital image restoration (such as high-definition conversion of old videos), and industrial inspection (such as enhancing security monitoring images).

[0095] In related technologies, mainstream super-resolution schemes mostly adopt a design approach of fixed resolution or fixed scene. Fixed-resolution super-resolution schemes refer to performing super-resolution processing at a specific resolution, upscaling low-resolution images to a fixed high resolution (e.g., 1080p→4K). The advantages of this type of super-resolution scheme are fast processing speed and ease of implementation. It typically relies on two common algorithmic techniques: The first is interpolation algorithms, which achieve super-resolution by interpolating pixel values. Common interpolation algorithms include bilinear interpolation and bicubic interpolation. The advantage of interpolation algorithms is their fast computation speed. The second is deep learning technology, which involves training deep learning models on large amounts of data to learn image features. Common deep learning models include SRCNN, ESPCN, and FSRCNN. The advantage of deep learning technology is its high processing efficiency.

[0096] Fixed-scene super-resolution solutions refer to super-resolution processing performed in specific scenarios (such as video conferencing and e-classrooms) to improve image quality and video transmission efficiency. Fixed-scene super-resolution solutions can be optimized using targeted intelligent algorithms. Through machine learning and other algorithms, they can adaptively adjust based on user needs and scenario characteristics to improve image quality and video transmission efficiency.

[0097] However, the above-mentioned super-resolution schemes are usually based on super-resolution of the entire screen. This processing method requires a lot of computing resources and processing time, places high demands on the performance of electronic devices, and has high power consumption.

[0098] It is evident that achieving a balance between image quality enhancement and rendering load under limited hardware resources has become a critical technical bottleneck that urgently needs to be overcome in the field of terminal device display technology.

[0099] In view of this, the present disclosure provides the following image processing methods, apparatus, devices, media and program products to overcome the above-mentioned technical bottlenecks.

[0100] Figure 1A This is a flowchart illustrating a screen processing method according to an exemplary embodiment. The method of this embodiment can be executed by a screen processing device, which can be configured in an electronic device with a touch display screen, such as a personal computer, mobile terminal (e.g., mobile phone, tablet computer), touch screen TV, in-vehicle terminal, wearable device (e.g., glasses, watch), etc. Figure 1A As shown, the method includes the following steps S101-S103:

[0101] In step S101, at least two types of sub-regions are determined from the screen area based on historical touch data.

[0102] In this embodiment, the electronic device can determine at least two types of sub-regions from the screen area based on historical touch data in response to the satisfaction of preset area determination conditions.

[0103] The aforementioned historical touch data may include touch data generated by historical touch operations on the screen area. For example, the historical touch data may include touch signal acquisition results from a preset duration (e.g., 50 milliseconds) prior to the current moment. Exemplarily, the historical touch data includes at least the touch point coordinates.

[0104] The density range of touch points differs in at least two of the aforementioned sub-regions. The density of touch points can be understood as the touch point data per unit area, which can be used to reflect the spatial concentration of user touch operations.

[0105] For example, the above-mentioned at least two types of sub-regions may include two types of sub-regions: "Region of Interest (ROI)" and "Region of Non-Interest (i.e., non-ROI)"; or, the above-mentioned at least two types of sub-regions may also include three or more types of sub-regions such as "Very Interested Region", "Generally Interested Region", ... and "Uninterested Region". In this embodiment, the number of sub-regions is not limited.

[0106] The following explanation uses the example of at least two types of sub-regions, including ROI and non-ROI sub-regions. For example, Figure 1B This is a schematic diagram illustrating an application scenario of a screen processing method according to an exemplary embodiment of this disclosure. For example... Figure 1B As shown, electronic devices can detect touch objects (e.g., based on preset touch sensors) Figure 1B The touch signal of the user's finger (or a specific input device such as a stylus) shown in the figure is used to obtain the aforementioned historical touch data. This data can then be used to determine the regions of interest and non-interest areas (such as...) on the screen. Figure 1B (The "ROI" and "non-ROI" are shown).

[0107] It is worth noting that the aforementioned preset touch sensor may include a capacitive sensor, a resistive sensor, or an electromagnetic induction sensor located below the screen, and this embodiment does not limit this.

[0108] In some embodiments, the preset region determination condition may be determined to be satisfied when at least one of the following conditions (a) to (c) occurs:

[0109] (a) The current application is the target application;

[0110] (b) The time interval since the first detection of a touch operation after the current application is launched reaches a preset duration;

[0111] (c) The time interval since the last determination of at least two types of sub-regions from the screen area reaches a preset time interval.

[0112] For example, some target apps that need to implement this embodiment can be pre-defined, such as video (mainly including interactive short videos), games, maps, weather, and other applications. Then, when the electronic device detects that the currently launched application is a target application, it can determine that the preset area determination conditions are met. For example, when the target app is launched, ROI and non-ROI are divided based on previously stored historical touch data.

[0113] Alternatively, based on detecting that the currently launched application is the target application, if the time interval between the current moment and the first detected touch operation after the current application starts reaches a preset duration, it can be determined that the preset region determination condition is met. For example, after the target APP starts, the user's touch operation is detected. When the user's first touch operation is detected, a timer starts. When the preset duration (e.g., 50 milliseconds) is reached, it can be determined that the preset region determination condition is met, and ROI and non-ROI can be divided based on the historical touch data during this period.

[0114] Based on this, a timer can be started after each division of ROI and non-ROI. When the preset time interval (e.g., 1 minute) is reached, it can be determined again that the preset area determination condition is met, and the ROI and non-ROI can be divided again based on the historical touch data of a preset duration (e.g., 50 milliseconds) before the current moment.

[0115] It is worth noting that the aforementioned preset duration of 50 milliseconds is used to ensure that there is enough touch data to calculate ROI and non-ROI. This value can be adjusted according to business needs during application, and this embodiment does not limit it.

[0116] By setting preset area determination conditions and dividing ROI and non-ROI when these conditions are met, the accuracy of ROI division can be ensured, thereby improving the accuracy and effect of subsequent dynamic rendering.

[0117] In other embodiments, the method of determining at least two types of sub-regions from the screen area based on historical touch data can be found in the following description. Figure 2A The embodiments shown will not be described in detail here.

[0118] In step S102, the content to be displayed in the at least two types of sub-regions is rendered using different rendering methods.

[0119] In this embodiment, after determining at least two types of sub-regions from the screen area based on historical touch data, different rendering methods can be used to render the content to be displayed in the at least two types of sub-regions. Among them, the sub-regions with higher touch density correspond to higher rendering accuracy.

[0120] For example, the content to be displayed in the above at least two types of sub-regions may include at least one of the following: (1) pixel matrix data of a graphics buffer, which includes a color buffer or a depth buffer; (2) layer data, which may consist of multiple superimposed graphics layers, each graphics layer corresponding to independent rendering parameters.

[0121] In some embodiments, different rendering methods in related technologies can be used to render the content to be displayed in the at least two types of sub-regions. The sub-regions with higher touch density have higher rendering accuracy (for example, the rendering accuracy of ROI is higher than that of non-ROI). This embodiment does not limit the selection of different rendering methods.

[0122] Taking ROI and non-ROI sub-regions as examples, we can perform super-resolution rendering and native resolution rendering on ROI and non-ROI sub-regions respectively.

[0123] The aforementioned super-resolution rendering refers to reconstructing a high-resolution image from a low-resolution image using a super-resolution (SR) algorithm. For example, super-resolution rendering can employ deep learning-based SR algorithms (such as SRCNN, ESRGAN, etc.), which involve pixel-level reconstruction of a low-resolution region of interest (ROI) using a pre-trained model. During the rendering process, low-resolution image patches of the ROI can be input into the SR model, outputting a high-resolution image while preserving local detail features.

[0124] The aforementioned native resolution rendering can refer to rendering based on the original image resolution or the screen's physical resolution.

[0125] In other embodiments, the methods described above for performing super-resolution rendering and native resolution rendering on the ROI and non-ROI sub-regions respectively can be found in the following... Figure 5 and Figure 6 The embodiments shown will not be described in detail here.

[0126] In step S103, the display screen is synthesized and displayed based on the content to be displayed after rendering of various sub-regions.

[0127] In this embodiment, after rendering the content to be displayed in the at least two types of sub-regions using different rendering methods, the display screen can be synthesized and displayed based on the rendered content to be displayed in each type of sub-region.

[0128] For example, after rendering the content to be displayed in at least two types of sub-regions using different rendering methods, the rendered content can be passed to a DPU (Display Processing Unit) or GPU (Graphics Processing Unit) for layer compositing. Then, it can be sequentially output to the screen for display at a preset frame rate (e.g., 60, 30, or 24). It is understood that this display compositing algorithm can be selected from relevant technologies according to business needs, and this embodiment does not limit it in this regard.

[0129] It is understandable that by determining at least two types of sub-regions from the screen area based on historical touch data, and then performing differentiated rendering on the content to be displayed in these at least two types of sub-regions, full-screen high-precision rendering (such as super-resolution rendering) can be avoided. This can reduce the rendering load of the CPU (Center Processing Unit) or GPU while ensuring image quality, and improve the system's response rate.

[0130] Compared to the full-screen super-resolution solution, this embodiment uses super-resolution rendering for ROIs while maintaining native resolution rendering for non-ROIs. This reduces the amount of floating-point operations on the CPU or GPU, lowers frame generation latency, and ensures image quality.

[0131] As described above, the method of this embodiment determines at least two types of sub-regions from the screen area based on historical touch data, and renders the content to be displayed in the at least two types of sub-regions using different rendering methods. Then, it synthesizes and displays the display screen based on the content to be displayed after rendering in each type of sub-region. This can achieve differentiated rendering of the user's areas of interest and areas of non-interest, avoid full-screen high-precision rendering, improve the flexibility of screen rendering, and thus save the computing resources and processing time of electronic devices while ensuring screen quality. This reduces the performance requirements of electronic devices and system power consumption, thereby improving the user experience.

[0132] Figure 2A This is a flowchart illustrating how to determine at least two types of sub-regions from a screen area based on historical touch data, according to an exemplary embodiment of this disclosure. This embodiment is based on the above embodiment and uses the example of how to determine at least two types of sub-regions from a screen area based on historical touch data for illustrative purposes.

[0133] like Figure 2A As shown, the step S104 above, which involves determining at least two types of sub-regions from the screen area based on historical touch data, may include the following steps S201-S202:

[0134] In step S201, the touch density distribution data of the screen area is determined based on the touch coordinates.

[0135] In this embodiment, after obtaining historical touch data (including touch point coordinates), the touch point density distribution data of the screen area can be determined based on the touch point coordinates. The degree of clustering of the touch point density distribution data reflects the density of touch points in the screen area.

[0136] For example, Figure 2B This is a schematic diagram illustrating the process of dividing ROI and non-ROI according to an exemplary embodiment of this disclosure. Figure 2B As shown, after acquiring historical touch data (at least including touch point coordinates) based on the touch sensor, the touch point coordinates can be converted into an unordered set of coordinates. Then, for this unordered set of coordinates, the touch density value of each coordinate point in the aforementioned coordinate system is calculated using a kernel density estimation algorithm. Subsequently, a heatmap can be generated based on the estimated touch density values ​​(i.e., ...). Figure 2B The "heat map" is located in the middle.

[0137] For example, when generating the heatmap based on the contact density estimate, the contact density estimate can be normalized and the heatmap generated using a color gradient mapping. Different colors in the heatmap can be used to indicate different contact densities. For example, as shown in 2B, the color depth in the heatmap can be positively correlated with the contact density, meaning the color depth linearly increases with the increase of the contact density estimate.

[0138] The formula for the above contact density estimate D(x,y) can be shown in equation (2-1):

[0139]

[0140] In the above formula, h is the bandwidth parameter used to control density smoothness, n is the total number of contacts, exp(·) is the Gaussian kernel function, and x i y i Let x and y be the horizontal and vertical coordinates of the i-th touch point, and let x and y be the horizontal and vertical coordinates of the two-dimensional coordinate points in the screen area.

[0141] In step S202, at least two types of sub-regions in the screen area are determined based on the density of touch points in the screen area.

[0142] In this embodiment, after determining the touch density distribution data of the screen area based on the touch coordinates, at least two types of sub-regions in the screen area can be determined according to the touch density in the screen area. The touch density in the same type of sub-region belongs to the same preset range.

[0143] Taking ROI and non-ROI sub-regions as examples again, such as Figure 2B As shown, after the heatmap is generated, the ROI and non-ROI can be rasterized according to the color depth in the heatmap. For example, the heatmap can be mapped to the screen area by coordinates, and the area with a color depth higher than a preset threshold delta (i.e., the area that the user touches frequently) can be regarded as the ROI, and the remaining area (i.e., the area that the user touches less) can be regarded as the non-ROI.

[0144] In some embodiments, the density threshold described above may be a static threshold set based on experience.

[0145] In other embodiments, the density threshold can also be a dynamic threshold adjusted based on the application scenario or real-time performance. For example, in scenarios such as games, the threshold can be appropriately increased to reduce the ROI; while in scenarios such as short videos, the threshold can be decreased to increase the ROI. The application scenario and the adjustment range of the threshold can be adjusted according to business needs, and this embodiment does not limit this. Alternatively, the threshold can be dynamically adjusted based on GPU load or frame rate fluctuations (e.g., when the GPU utilization is greater than 80%, the threshold can be increased by 20% to reduce the area of ​​the ROI).

[0146] By dynamically adjusting the density threshold based on the application scenario, threshold parameters can be automatically matched for different scenarios such as games, videos, and maps, thereby improving the accuracy of ROI segmentation. Furthermore, dynamically adjusting the density threshold based on real-time performance allows for dynamic adjustment of the threshold according to the device's computing power (such as GPU load), thus adapting to device performance fluctuations.

[0147] It is understandable that when determining more than n (where n is a preset positive integer) sub-regions from the screen area, this can be achieved by setting n-1 density thresholds, which will not be elaborated in this embodiment.

[0148] As described above, this embodiment determines the touch density distribution data of the screen area based on the touch coordinates, and determines at least two types of sub-regions in the screen area based on the touch density. This allows for accurate determination of at least two types of sub-regions from the screen area based on historical touch data. Consequently, it enables differentiated rendering of the content to be displayed in the at least two types of sub-regions, thus avoiding full-screen high-precision rendering. This saves computing resources and processing time of electronic devices while ensuring image quality, reduces performance requirements and system power consumption, and ultimately improves user experience.

[0149] Figure 3This is a flowchart illustrating a screen processing method according to another exemplary embodiment of the present disclosure; the method of this embodiment can be executed by a screen processing device, which can be configured in an electronic device with a touch display screen, such as a personal computer, a mobile terminal (e.g., a mobile phone, tablet computer, etc.), a touch screen TV, an in-vehicle terminal, a wearable device (e.g., glasses, a watch), etc. Figure 3 As shown, the method includes the following steps S301-S306:

[0150] In step S301, at least two types of sub-regions are determined from the screen area based on historical touch data.

[0151] The historical touch data includes touch data generated by historical touch operations on the screen area, and the historical touch data includes at least touch point coordinates, wherein the density range of touch points in the at least two types of sub-regions is different.

[0152] In step S302, in response to obtaining the current touch data, the current touch event is determined based on the current touch data.

[0153] In this embodiment, the electronic device can respond to the acquisition of current touch data and determine the current touch event based on the current touch data.

[0154] For example, after an electronic device has determined at least two types of sub-regions from the screen area based on historical touch data, or at the same time, current touch data can be collected, and then the current touch event can be determined based on the touch data.

[0155] The aforementioned current touch data may include touch data generated by a current touch operation targeting the screen area. For example, the touch operation may include any one of clicking, swiping, focusing, or zooming.

[0156] For example, after determining at least two types of sub-regions from the screen area based on historical touch data, the electronic device can continue to detect touch signals from a touch object (such as a user's finger or stylus) based on a preset touch sensor to obtain the current touch data. The preset touch sensor may include a capacitive sensor, a resistive sensor, or an electromagnetic induction sensor located below the screen; this embodiment does not limit the specific type of sensor.

[0157] Based on this, electronic devices can employ touch event recognition algorithms from related technologies to identify the touch event corresponding to the current touch data. This touch event is matched with the touch operation that generated the touch data; that is, the touch event can include any of the following: click, swipe, focus, or zoom. For example, if the current touch operation performed by a finger on the screen area is a "swipe," the electronic device can acquire the current touch data generated by the current touch operation, and then identify the current touch event, i.e., the "swipe" event, based on the current touch data.

[0158] In some embodiments, once the current touch data is obtained, the current touch event can be identified based on the sliding speed and / or acceleration analysis results of the touch point coordinate sequence.

[0159] In other embodiments, the method of determining the current touch event based on the current touch data described above can also be found in the following: Figure 4 The embodiments shown will not be described in detail here.

[0160] In step S303, it is determined whether the current touch event is a first type event or a second type event: if it is a first type event, then step S304 is executed; if it is a second type event, then step S305 is executed.

[0161] In step S304, the content to be displayed in the at least two types of sub-regions is rendered using different rendering methods.

[0162] Among them, the sub-regions with higher contact density correspond to higher rendering accuracy.

[0163] In step S305, the content to be displayed in the at least two types of sub-regions is rendered using a preset resolution.

[0164] Wherein, the preset resolution is lower than the smaller of the original image resolution and the screen physical resolution, as shown in equation (3-1):

[0165] Preset resolution ≤ min(original image resolution, screen physical resolution); (3-1)

[0166] It is understandable that the original image resolution can refer to the actual resolution of the image to be rendered (such as the 720×1280 resolution of a video source). The screen physical resolution can refer to the actual number of pixels of an electronic device (such as 1080×2340 for a mobile phone screen or 3840×2160 for a 4K screen), which can represent the hardware display capability of the screen.

[0167] In this embodiment, after determining the current touch event based on the current touch data, it can be determined whether the current touch event is a first type event or a second type event:

[0168] If it is a first type of event, then step S304 can be executed to render the content to be displayed in the at least two types of sub-regions using different rendering methods.

[0169] Among them, the sub-regions with higher contact density correspond to higher rendering accuracy.

[0170] Conversely, if it is the second type of event, then step S305 can be executed to render the content to be displayed in the at least two types of sub-regions using a preset resolution.

[0171] It's worth noting that the first type of event mentioned above can include pre-specified touch events that prioritize image quality (such as clicks or focus). The second type of event mentioned above can include pre-specified touch events that prioritize responsiveness (such as swipes or zooms). In other words, the first type of event focuses more on triggering static content rendering (such as UI details after a button click), while the second type of event focuses more on being associated with dynamic content operations (such as scenarios where page scrolling requires low latency).

[0172] Taking a map application as an example, assuming the original image resolution is 1080P (map data source), the screen physical resolution is 2K (i.e., 2560x1440), and the preset resolution is 480P (e.g., the minimum screen resolution), when a user performs a swipe operation on the user interface of the map application, the electronic device can detect that the current touch event is "swipe". Then, the content to be displayed in the at least two types of sub-areas can be rendered at 480P, which can significantly reduce the rendering load of the CPU or GPU, thereby reducing the power consumption of the device.

[0173] In step S306, the display screen is synthesized and displayed based on the content to be displayed after rendering of various sub-regions.

[0174] The explanations and descriptions of steps S301, S304, and S306 can be found above. Figure 1A Steps S101-S103 in the illustrated embodiment will not be described in detail here.

[0175] As described above, this embodiment, in response to obtaining current touch data, determines the current touch event based on the current touch data, and in response to the current touch event being a first type of event, performs the operation of rendering the content to be displayed in the at least two types of sub-regions using different rendering methods. In response to the current touch event being a second type of event, the content to be displayed in the at least two types of sub-regions is rendered using a preset resolution. The preset resolution is lower than the smaller of the original image resolution and the screen physical resolution. This allows for determining whether to perform differentiated rendering based on the different types of the current touch event, based on the determination of at least two types of sub-regions. This further improves the flexibility and rationality of image rendering, reduces the rendering load of the CPU or GPU, and balances the smoothness and image quality of the displayed image, improving the user's visual experience while reducing the overall power consumption of the device.

[0176] Figure 4 This is a flowchart illustrating how to determine the current touch event based on the current touch data according to an exemplary embodiment of the present disclosure; this embodiment is an exemplary description based on the above embodiment, taking how to determine the current touch event based on the current touch data as an example.

[0177] In this embodiment, the current touch data may include at least one of touch type, touch coordinates, touch time, and touch index.

[0178] Furthermore, such as Figure 4 As shown, the step S302 above, which involves determining the current touch event based on the current touch data, may include the following steps S401-S403:

[0179] In step S401, the current touch data is standardized to obtain standardized touch data.

[0180] In this embodiment, after obtaining the current touch data, the current touch data can be standardized. For example, when standardizing the current touch data, the following process can be performed:

[0181] (1) Touch type encoding, such as encoding touch type (e.g., press, lift or move) as a numerical value or one-hot vector for model processing.

[0182] (2) Touch point coordinate normalization, such as normalizing physical coordinates (e.g., pixel values) to the range of [0,1] to eliminate device resolution differences. For example, if the screen resolution is 1080×2400, the touch point coordinates (540,1200) can be normalized to (0.5,0.5).

[0183] (3) Duration calculation, such as calculating the duration based on the time difference between the touch point being "pressed" and "released". For example, when a user performs a click operation, if the press timestamp is 100ms and the release timestamp is 150ms, the duration can be calculated as 50ms. It is understood that the timestamp content in this embodiment is for illustrative purposes only and is not intended to limit the scope of this embodiment.

[0184] (4) Calculation of acceleration direction, such as calculating the direction vector (i.e., x-axis / y-axis components) based on the coordinate changes of the moving touch point. For example, if the movement trajectory is from coordinates (0.2, 0.3) to (0.5, 0.6), the acceleration direction vector can be calculated as (0.3, 0.3).

[0185] (5) Touch index matching, such as assigning a unique index to multi-touch operations. For example, when a user performs a two-finger zoom operation, the first touch index can be 0 and the second touch index can be 1, thereby preventing data confusion between the two fingers.

[0186] After completing the above process, standardized touch data can be obtained. The standardized touch data includes at least one of the following: touch type, touch point coordinates, duration, acceleration direction, and touch index. For example, the standardized touch data can be in the form of <touch type encoding, touch point coordinate normalization, duration, acceleration direction, touch index>.

[0187] In step S402, the standardized touch data is arranged in ascending order of touch time.

[0188] In this embodiment, after the current touch data is standardized to obtain standardized touch data, it can be sorted in ascending order by touch time (e.g., touch point timestamp) to ensure that the timing logic is correct.

[0189] For example, in the case where the original data sequence is in random order of timestamps, such as:

[0190] [Time: 200ms, Type: Move, Index: 0];

[0191] [Time: 150ms, Type: Pressed, Index: 0];

[0192] [Time: 250ms, Type: Lift, Index: 0].

[0193] After sorting them in ascending order, we get the following results:

[0194] [Time: 150ms, Type: Pressed, Index: 0];

[0195] [Time: 200ms, Type: Move, Index: 0];

[0196] [Time: 250ms, Type: Lift, Index: 0].

[0197] It is understandable that when there is multi-touch data (e.g., pressing two fingers simultaneously) within the same time window, it can be sorted in ascending order by index, which will not be elaborated here.

[0198] In step S403, the arranged data is input into a preset touch event recognition model to obtain the current touch event.

[0199] In this embodiment, after the standardized touch data is sorted in ascending order by touch time, the sorted data can be input into a preset touch event recognition model to obtain the current touch event.

[0200] In some embodiments, the touch event recognition model described above can be a state machine. For example, the state definition of the state machine can include: an initial state (IDLE), a pressed state (PRESSED), a moving state (MOVING), a released state (RELEASED), and a focused state (FOCUSING). The focused state (FOCUSING) can be an intermediate state between the pressed state (PRESSED) and the released state (RELEASED).

[0201] Based on this, state transition determination rules for different touch events can be defined:

[0202] (1) Click event: If the current touch data corresponds to the state transition of "IDLE→PRESSED→RELEASED (duration < 200ms and no movement)", then the current touch event can be determined as "click".

[0203] (2) Swipe event: If the state transition corresponding to the current touch data is "IDLE→PRESSED→MOVING (continuous movement beyond the threshold distance)→RELEASED", then the current touch event can be determined as "swipe".

[0204] (3) Zoom event: If the current touch data corresponds to the state transition of "two fingers down → two fingers move (distance increases / decreases) → two fingers lift up", then the current touch event can be determined as "zoom".

[0205] (4) Focusing event: If the current state transition corresponding to the current touch data is "IDLE→PRESSED→FOCUSING (duration ≥ 500ms and coordinate offset ≤ threshold)→RELEASED", then the current touch event can be determined as "focusing".

[0206] For example, when the sequence input into the model is: "press (index 0) → move (index 0, coordinate change) → release (index 0)", the model can output the current touch event as "swipe event".

[0207] It is worth noting that the above state transition judgment strategy is only for illustrative purposes and can be adjusted according to requirements in application. This embodiment does not limit it in this regard.

[0208] As described above, this embodiment standardizes the current touch data to obtain standardized touch data, arranges the standardized touch data in ascending order of touch time, and then inputs the arranged data into a preset touch event recognition model to obtain the current touch event. This allows for accurate determination of the current touch event based on the current touch data, and enables subsequent decisions on whether to perform differentiated rendering based on the different types of the current touch event. This further improves the flexibility and rationality of screen rendering, reduces the rendering load of the CPU or GPU, and balances the smoothness and image quality of the displayed screen, thereby improving the user's visual experience while reducing the overall power consumption of the device.

[0209] In some embodiments, one of the different rendering methods may be super-resolution rendering. Furthermore, Figure 5 This is a flowchart illustrating how to perform super-resolution rendering of the content to be displayed in the first type of sub-region in the at least two types of sub-regions, according to an exemplary embodiment of this disclosure. This embodiment is based on the above embodiment and takes how to perform super-resolution rendering of the content to be displayed in the first type of sub-region in the at least two types of sub-regions as an example for illustrative explanation.

[0210] like Figure 5 As shown, step S102 above may include super-resolution rendering of the content to be displayed in the first type of sub-region in the at least two types of sub-regions based on the following steps S501-S502:

[0211] In step S501, the target resolution is determined based on the scaling factor of the current image.

[0212] In this embodiment, when performing super-resolution rendering on the content to be displayed in the first type of sub-region in the at least two types of sub-regions, the target resolution can be determined first based on the scaling factor of the current screen.

[0213] In some embodiments, a mapping table between scaling factors (such as 1.5x, 2.0x, 3.0x, etc.) and target resolution can be established to ensure that the pixel density of the first type of sub-region matches the user's intent. Based on this, when the scaling factor of the current image is detected, the above mapping table can be queried to obtain the corresponding target resolution.

[0214] For example, when a user triggers a click (or focus) operation, the electronic device can recognize the current touch event as a "click," and thus obtain the current zoom level (e.g., 2x). Based on this, it can determine that the target resolution of the first type of sub-region is twice that of the original image. If the original image resolution is 800x600, then the target resolution is 1600*1200.

[0215] In other embodiments, when determining the target resolution, an upper limit can also be set based on the device's hardware capabilities (such as the screen's physical resolution or the GPU's maximum texture size). For example, if the device's screen resolution is 4K (3840x2160), then the maximum resolution is 4K. Therefore, when the original resolution of the first type of sub-region is greater than or equal to 1080P (1920*1080), and the current scaling factor is 2.0x, the target resolution can be set to 4K.

[0216] In step S502, based on the target resolution, the content to be displayed in the first type of sub-region is reconstructed using a preset super-resolution algorithm.

[0217] In this embodiment, after determining the target resolution based on the scaling factor of the current screen, the content to be displayed in the first type of sub-region can be reconstructed based on the target resolution using a preset super-resolution algorithm.

[0218] The super-resolution algorithm mentioned above can be configured based on business requirements, such as being set as a deep learning model and / or an interpolation algorithm.

[0219] In some embodiments, a mapping relationship can be established between a preset range of scaling factors and corresponding super-resolution algorithms. Then, when the current scaling factor is obtained, the mapping relationship can be queried based on its preset range to determine the corresponding super-resolution algorithm. For example, this mapping relationship can be shown in Table 1 below:

[0220] Table 1

[0221]

[0222] For example, when a user uses a map app, if the electronic device detects the user's action on a point of interest on the map, the system can determine that the touch event is "focusing". Then, based on the current zoom level (e.g., 2.0x), it can select a 4K target resolution and use deep learning models such as ESRGAN to perform super-resolution rendering on the first type of sub-region (e.g., the ROI containing the aforementioned point of interest) to achieve clear display of relevant information in the map (e.g., road signs, buildings, etc.), thereby improving the user experience.

[0223] It is understood that the content in Table 1 above is for illustrative purposes only and can be adjusted according to actual needs when applied. This embodiment does not limit this.

[0224] Furthermore, the process of determining the target resolution and performing super-resolution rendering on the first type of sub-region can also be executed based on a super-resolution module deployed in the electronic device. For example, the scaling factor of the current image can be passed to the super-resolution module, which can then select the target resolution based on the scaling factor to perform super-resolution on the first type of sub-region to obtain higher quality image details. It is understandable that by deploying a trained super-resolution module, end-to-end calculation of the target resolution of the first type of sub-region and super-resolution rendering can be achieved, improving the accuracy and rationality of the target resolution, thereby increasing the accuracy of the displayed content details and colors in the first type of sub-region. Moreover, this is beneficial for improving the accuracy of image content recognition in subsequent applications such as computer vision and artificial intelligence technologies.

[0225] As described above, this embodiment determines the target resolution based on the scaling factor of the current screen, and reconstructs the content to be displayed in the first type of sub-region based on the target resolution using a preset super-resolution algorithm. This enables super-resolution rendering of the content to be displayed in the first type of sub-region within the at least two types of sub-regions, avoiding the need to divide the entire screen. This saves computing resources and processing time for electronic devices while ensuring image quality, reducing performance requirements and system power consumption. Furthermore, by dynamically mapping the target resolution and super-resolution algorithm based on the scaling factor of the current screen, a balance between image quality and performance can be achieved, improving adaptability to different application scenarios and ultimately enhancing the user experience.

[0226] In some embodiments, one of the different rendering methods may be native resolution rendering. Furthermore, Figure 6 This is a flowchart illustrating how to perform native resolution rendering of the content to be displayed in the second type of sub-region in the at least two types of sub-regions, according to an exemplary embodiment of this disclosure. This embodiment is based on the above embodiment and takes as an example how to perform native resolution rendering of the content to be displayed in the second type of sub-region in the at least two types of sub-regions.

[0227] like Figure 6 As shown, step S102 above may include rendering the content to be displayed in the second type of sub-region within the at least two types of sub-regions at native resolution based on the following steps S601-S602:

[0228] In step S601, the content to be displayed in the second type of sub-region is rendered based on the original image resolution or the screen physical resolution.

[0229] In this embodiment, when rendering the content to be displayed in the second type of sub-region in the at least two types of sub-regions at the native resolution, the content to be displayed in the second type of sub-region can be rendered based on the original image resolution or the screen physical resolution first.

[0230] The original image resolution can refer to the actual resolution of the image to be rendered (such as the 720×1280 resolution of a video source). The screen physical resolution can refer to the actual number of pixels of an electronic device (such as 1080×2340 for a mobile phone screen or 3840×2160 for a 4K screen), which can represent the hardware display capability of the screen.

[0231] In some embodiments, the original image resolution of 1080p or the screen physical resolution can be selected to render the content to be displayed in the second type of sub-region according to the needs of the scene. For example, the resolution corresponding to the second type of sub-region can be selected based on the type of the current touch event.

[0232] For example, when the current touch event is a swipe or zoom event that prioritizes responsiveness, the resolution can be selected based on the difference between the screen's physical resolution and the original image resolution.

[0233] (1) If the screen physical resolution is less than or equal to the original image resolution, the content to be displayed in the second type of sub-region can be rendered using the screen physical resolution (e.g., the lowest screen resolution is 480p) to avoid additional scaling calculations.

[0234] (2) If the screen physical resolution is greater than the original image resolution, the content to be displayed in the second type of sub-area can be rendered at the original image resolution to reduce the load on the CPU or GPU.

[0235] However, when the current touch events are clicks or focus events that prioritize image quality:

[0236] The content to be displayed in the second type of sub-region can be rendered using the original image resolution (such as 1080p). Since full-screen super-resolution rendering is not performed, the rendering time of the CPU or GPU can be reduced.

[0237] Understandably, if the screen supports HDR display, the content to be displayed in the second type of sub-region can also be rendered using the screen's physical resolution to adapt to the screen's high dynamic range.

[0238] It is worth noting that the above-mentioned method of rendering the content to be displayed in the second type of sub-region based on different resolutions can be found in the relevant technical records. For example, the content to be displayed in the second type of sub-region can be divided into multiple sub-blocks (e.g., 16x16 pixels) and then each sub-block can be rendered. This embodiment does not limit this.

[0239] In step S602, the content to be displayed in the rendered second type of sub-region is scaled according to the scaling factor of the current screen.

[0240] In this embodiment, after rendering the content to be displayed in the second type of sub-region based on the original image resolution or the screen physical resolution, the rendered content to be displayed in the second type of sub-region can be scaled according to the scaling factor of the current screen.

[0241] In some embodiments, a bilinear interpolation algorithm or a nearest neighbor interpolation algorithm may be used to scale the content to be displayed in the rendered second type of sub-region according to the scaling factor of the current screen.

[0242] In other embodiments, the scaling algorithm corresponding to the second type of sub-region can also be selected based on the type of the current touch event. For example:

[0243] When the current touch event is a click or focus event that prioritizes image quality, a bilinear interpolation algorithm can be used for scaling to maintain image smoothness.

[0244] When the current touch events are touch events that prioritize response speed, such as swiping or zooming, the nearest neighbor interpolation algorithm can be used for scaling to reduce computational latency.

[0245] Taking ROI and non-ROI sub-regions as examples, when a user takes a photo using a camera application, if the user uses two fingers to zoom in on the currently captured image to 3x, the electronic device can recognize the current touch event as "zooming". It can then perform low-resolution rendering of 480P on the non-ROI (background in the image) and high-resolution super-resolution of 4K on the ROI (subject in the image). On this basis, the background area can be scaled by 3x and blended with the super-resolution subject to avoid image tearing or blurring, thereby improving image quality.

[0246] As described above, this embodiment renders the content to be displayed in the second type of sub-region based on the original image resolution or the screen's physical resolution, and scales the rendered content in the second type of sub-region according to the scaling factor of the current screen. This allows for the subsequent synthesis and display of the display screen based on the ROI content rendered by super-resolution and the non-ROI content rendered at native resolution. This avoids full-screen high-precision rendering, improves the flexibility of screen rendering, and saves the computing resources and processing time of electronic devices while ensuring screen quality. It also reduces the performance requirements of electronic devices and system power consumption. Furthermore, by combining native resolution rendering with touch event recognition, a balance between performance and smoothness can be achieved, reducing the use of system computing resources and thus improving the user experience.

[0247] Figure 7 This is a block diagram illustrating a screen processing apparatus according to an exemplary embodiment of the present disclosure; the apparatus of this embodiment can be configured in an electronic device, such as a personal computer, a mobile terminal (e.g., a mobile phone, a tablet computer, etc.), a touchscreen TV, an in-vehicle terminal, a wearable device (e.g., glasses, a watch), etc. Figure 7 As shown, the device may include: a region determination module 110, a content rendering module 120, and a composite display module 130, wherein:

[0248] The region determination module 110 is used to determine at least two types of sub-regions from the screen region based on historical touch data. The historical touch data includes touch data generated by historical touch operations on the screen region. The historical touch data includes at least touch point coordinates. The density range of touch points in the at least two types of sub-regions is different.

[0249] The content rendering module 120 is used to render the content to be displayed in the at least two types of sub-regions using different rendering methods.

[0250] Among them, the sub-regions with higher contact density correspond to higher rendering accuracy;

[0251] The composite display module 130 is used to composite and display the display screen based on the content to be displayed after rendering various sub-regions.

[0252] As described above, the device in this embodiment determines at least two types of sub-regions from the screen area based on historical touch data, and renders the content to be displayed in the at least two types of sub-regions using different rendering methods. Then, it synthesizes and displays the display screen based on the content to be displayed after rendering in each type of sub-region. This allows for differentiated rendering of areas of interest and areas of non-interest to the user, avoiding full-screen high-precision rendering and improving the flexibility of screen rendering. As a result, it can save the computing resources and processing time of electronic devices while ensuring screen quality, reduce the performance requirements of electronic devices and system power consumption, and thus improve the user experience.

[0253] Figure 8 This is a block diagram illustrating another image processing apparatus according to an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be configured in an electronic device, such as a personal computer, a mobile terminal (e.g., a mobile phone, tablet computer), a touchscreen TV, an in-vehicle terminal, or a wearable device (e.g., glasses, a watch). The region determination module 210, the content rendering module 220, and the composite display module 230 are as described above. Figure 7 The functions of the region determination module 110, content rendering module 120, and composite display module 130 in the illustrated embodiment are the same, and will not be described in detail here.

[0254] like Figure 8 As shown, the region determination module 210 may include:

[0255] The distribution determination unit 211 is used to determine the touch density distribution data of the screen area based on the touch coordinates, wherein the degree of aggregation of the touch density distribution data is used to reflect the density of touches in the screen area.

[0256] The region determination unit 212 is used to determine at least two types of sub-regions in the screen region based on the density of touch points in the screen region, wherein the density of touch points in the same type of sub-region belongs to the same preset range.

[0257] In some embodiments, the distribution determination unit 211 can also be used for:

[0258] Map the coordinates of the touch point to the coordinate system of the screen area;

[0259] Calculate the contact density value at each point in the coordinate system.

[0260] In some embodiments, the above-described apparatus may further include:

[0261] The touch event determination module 240 is configured to determine a current touch event based on the current touch data in response to acquiring the current touch data, wherein the current touch data includes touch data generated by the current touch operation on the screen area;

[0262] Furthermore, the content rendering module 220 can also be used to respond to the current touch event being a first type of event and perform the operation of rendering the content to be displayed in the at least two types of sub-regions using different rendering methods, wherein the first type of event includes a pre-specified touch event that focuses more on image quality.

[0263] In some embodiments, the content rendering module 220 may also be used to render the content to be displayed in the at least two types of sub-regions using a preset resolution in response to the current touch event being a second type of event. The preset resolution is lower than the smaller of the original image resolution and the screen physical resolution. The second type of event includes a pre-specified touch event that emphasizes response speed.

[0264] In some embodiments, the aforementioned current touch data may include at least one of touch type, touch coordinates, touch time, and touch index;

[0265] Furthermore, the touch event determination module 240 may include:

[0266] The standardization unit 241 is used to standardize the current touch data to obtain standardized touch data. The content of the standardized touch data includes at least one of touch type, touch point coordinates, duration, acceleration direction, and touch index.

[0267] Data sorting unit 242 is used to sort the standardized touch data in ascending order of touch time;

[0268] The event determination unit 243 is used to input the arranged data into a preset touch event recognition model to obtain the current touch event.

[0269] In some embodiments, one of the different rendering methods is super-resolution rendering;

[0270] The above-mentioned content rendering module 220 may include a first rendering unit 221;

[0271] The first rendering unit 221 can be used for:

[0272] Determine the target resolution based on the scaling factor of the current image;

[0273] Based on the target resolution, the content to be displayed in the first type of sub-region is reconstructed using a preset super-resolution algorithm, which includes a deep learning model and / or an interpolation algorithm.

[0274] In some embodiments, one of the different rendering methods is native resolution rendering;

[0275] The above-mentioned content rendering module 220 may include a second rendering unit 222;

[0276] The second rendering unit 222 can be used for:

[0277] The content to be displayed in the second type of sub-region is rendered based on the original image resolution or the screen physical resolution;

[0278] The content to be displayed in the rendered second sub-region is scaled according to the scaling factor of the current screen.

[0279] In some embodiments, the region determination module 210 can also be used to perform the operation of determining at least two types of sub-regions from the screen region based on historical touch data in response to meeting preset region determination conditions;

[0280] The conditions for satisfying the preset region determination include at least one of the following:

[0281] The current application is the target application;

[0282] The time interval since the first detected touch operation after the current application was launched has reached a preset duration;

[0283] The time interval since the last determination of at least two types of sub-regions from the screen area reaches a preset time interval.

[0284] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0285] In an exemplary embodiment, this disclosure also provides an electronic device including a processor and a memory for storing a computer program; wherein the processor is configured to implement the above-described screen processing method when executing the computer program. For example, Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0286] Reference Figure 9 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0287] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the aforementioned image processing method. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0288] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc., where the application implements the aforementioned screen processing methods when executed by the processor. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0289] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0290] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0291] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0292] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0293] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 can detect the on / off state of device 900, the relative positioning of components such as the display panel and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0294] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0295] In an exemplary embodiment, device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described image processing method.

[0296] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of the device 900 to complete the above-described screen processing method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0297] In an exemplary embodiment, a computer program product including instructions is also provided, which can be executed by the processor 920 of the device 900 to complete the above-described screen processing method.

[0298] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the foregoing claims.

[0299] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for image processing, characterized in that, The method includes: Based on historical touch data, at least two types of sub-regions are determined from the screen area. The historical touch data includes touch data generated by historical touch operations on the screen area. The historical touch data includes at least touch point coordinates. The density range of touch points in the at least two types of sub-regions is different. The content to be displayed in the at least two types of sub-regions is rendered using different rendering methods, wherein the sub-region with a higher density of touch points corresponds to a higher rendering accuracy; The display screen is synthesized and displayed based on the content to be displayed after rendering in various sub-regions.

2. The method according to claim 1, characterized in that, Based on historical touch data, at least two types of sub-regions are determined from the screen area, including: The touch point density distribution data of the screen area is determined based on the touch point coordinates, and the degree of aggregation of the touch point density distribution data is used to reflect the density of touch points in the screen area. At least two types of sub-regions are determined in the screen area based on the density of touch points in the screen area, wherein the density of touch points in the same type of sub-region belongs to the same preset range.

3. The method according to claim 2, characterized in that, Determining the touch density distribution data of the screen area based on the touch coordinates includes: Map the coordinates of the touch point to the coordinate system of the screen area; Calculate the contact density value at each point in the coordinate system.

4. The method according to claim 1, characterized in that, The method further includes: In response to acquiring current touch data, a current touch event is determined based on the current touch data, wherein the current touch data includes touch data generated by the current touch operation on the screen area; In response to the current touch event being a first type of event, the operation of rendering the content to be displayed in the at least two types of sub-regions using different rendering methods is performed. The first type of event includes pre-specified touch events that prioritize image quality.

5. The method according to claim 4, characterized in that, The method further includes: In response to the current touch event being a second type of event, the content to be displayed in the at least two types of sub-regions is rendered using a preset resolution, which is lower than the smaller of the original image resolution and the screen physical resolution. The second type of event includes pre-specified touch events that prioritize response speed.

6. The method according to claim 4, characterized in that, The current touch data includes at least one of touch type, touch coordinates, touch time, and touch index; Determining the current touch event based on the current touch data includes: The current touch data is standardized to obtain standardized touch data. The content of the standardized touch data includes at least one of touch type, touch point coordinates, duration, acceleration direction, and touch index. The standardized touch data is sorted in ascending order by touch time; The sorted data is input into a preset touch event recognition model to obtain the current touch event.

7. The method according to claim 1, characterized in that, One of the different rendering methods is super-resolution rendering; Super-resolution rendering is performed on the content to be displayed in the first type of sub-region within the at least two types of sub-regions, including: Determine the target resolution based on the scaling factor of the current image; Based on the target resolution, the content to be displayed in the first type of sub-region is reconstructed using a preset super-resolution algorithm, which includes a deep learning model and / or an interpolation algorithm.

8. The method according to claim 1, characterized in that, One of the different rendering methods is native resolution rendering; Rendering the content to be displayed in the second type of sub-region within the at least two types of sub-regions at native resolution includes: The content to be displayed in the second type of sub-region is rendered based on the original image resolution or the screen physical resolution; The content to be displayed in the rendered second sub-region is scaled according to the scaling factor of the current screen.

9. The method according to claim 1, characterized in that, The method further includes, in response to satisfying a preset area determination condition, performing the operation of determining at least two types of sub-regions from the screen area based on historical touch data; The conditions for satisfying the preset region determination include at least one of the following: The current application is the target application; The time interval since the first detected touch operation after the current application was launched has reached a preset duration; The time interval since the last determination of at least two types of sub-regions from the screen area reaches a preset time interval.

10. A screen processing device, characterized in that, The device includes: A region determination module is used to determine at least two types of sub-regions from a screen area based on historical touch data. The historical touch data includes touch data generated by historical touch operations on the screen area, and the historical touch data includes at least touch point coordinates. The density range of touch points in the at least two types of sub-regions is different. The content rendering module is used to render the content to be displayed in the at least two types of sub-regions using different rendering methods, wherein the sub-regions with higher touch density have higher rendering accuracy. The composite display module is used to composite and display the display image based on the content to be displayed after rendering various sub-regions.

11. An electronic device, characterized in that, The device includes: Processor and memory used to store computer programs; The processor is configured to implement the screen processing method according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the screen processing method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the screen processing method according to any one of claims 1 to 9.

Images