Display processing method and apparatus, storage medium, and electronic device

Abstract

This application discloses a display processing method, apparatus, storage medium, and electronic device, relating to the field of display technology. The method includes: in split-screen mode, determining the device processing load based on device-related data; determining the user attention level of the main screen based on user-related data; determining a picture quality adjustment strategy for the secondary screen based on the device processing load and the user attention level; and adjusting the picture quality of the secondary screen according to the picture quality adjustment strategy. This application can improve resource utilization in split-screen mode and reduce issues such as lag, overheating, and high power consumption caused by excessive resource consumption in split-screen mode.

Description

Technical Field

[0001] This application relates to the field of display technology, specifically to a display processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Smart TVs, monitors, projectors, and other electronic devices typically have split-screen functionality. In split-screen mode, multiple screens are displayed on the device, greatly enhancing the user experience. However, currently, electronic devices in split-screen mode usually use a fixed image quality output mode, meaning that multiple screens are displayed with the same fixed image quality. This can easily lead to excessive resource consumption, causing the device to lag, overheat, and experience other abnormal conditions such as high power consumption. Summary of the Invention

[0003] This application provides a solution that can effectively improve resource utilization in split-screen mode and reduce issues such as lag, overheating, and high power consumption caused by excessive resource consumption in split-screen mode.

[0004] The embodiments of this application provide the following technical solutions: According to one embodiment of this application, a display processing method includes: in split-screen mode, determining the device processing pressure based on device-related data; determining the user attention level of the main screen based on user-related data; determining the image quality adjustment strategy of the secondary screen based on the device processing pressure and the user attention level; and adjusting the image quality of the secondary screen according to the image quality adjustment strategy.

[0005] In some embodiments of this application, determining the user attention level of the main screen based on user-related data includes: when the user-related data includes user-specified data, determining the user-specified sub-screen as the main screen based on the user-specified data, and determining the preset highest attention level as the user attention level of the main screen; when the user-related data does not include the user-specified data, estimating the user attention level of the sub-screen based on at least one of user behavior data and user visual data, and determining the sub-screen whose user attention level meets the predetermined attention conditions as the main screen.

[0006] In some embodiments of this application, determining the user attention level of the main screen based on user-related data includes: performing predictive analysis based on the user-related data and screen display data within a preset time period to obtain the main screen and the user attention level.

[0007] In some embodiments of this application, determining the image quality adjustment strategy for the secondary screen based on the device processing pressure and the user attention includes: when the device processing pressure is greater than or equal to a first preset pressure and the user attention is greater than an intermediate attention level, the image quality adjustment strategy is to reduce the resolution or color depth of non-critical areas in the secondary screen; when the device processing pressure is greater than or equal to the first preset pressure and the user attention is less than the intermediate attention level, the image quality adjustment strategy is to reduce the overall resolution or overall color depth of the secondary screen; when the device processing pressure is greater than or equal to a second preset pressure and the user attention is less than the intermediate attention level, the image quality adjustment strategy is to reduce the frame rate of the secondary screen; when the device processing pressure is greater than or equal to the second preset pressure and the user attention is greater than an upper limit attention level, or when the pressure difference between the device processing pressure and the preset upper limit pressure is less than a preset difference, the image quality adjustment strategy is to statically display the secondary screen.

[0008] In some embodiments of this application, the method further includes: determining the key area and the non-key area in the secondary screen based on user-related data and screen display data; or, determining the non-key area as the area outside the predetermined key content area in the secondary screen.

[0009] In some embodiments of this application, adjusting the image quality of the secondary screen according to the image quality adjustment strategy includes: adjusting the image quality of the secondary screen according to the image quality adjustment strategy using a region-by-region block update method; or, adjusting the image quality of the secondary screen according to the image quality adjustment strategy using a frame-by-frame gradual update method.

[0010] In some embodiments of this application, determining the device processing pressure based on device-related data includes: weighting and summing multi-dimensional weight coefficients and multi-dimensional load data to obtain the device processing pressure; or, determining the device processing pressure based on rendering latency data; or, determining the device processing pressure based on the number of buffered frames.

[0011] According to one embodiment of this application, a display processing apparatus includes: a pressure determination module, configured to: determine device processing pressure based on device-related data in split-screen mode; an attention determination module, configured to: determine user attention on the main screen based on user-related data; a strategy determination module, configured to: determine a picture quality adjustment strategy for the secondary screen based on the device processing pressure and the user attention; and a picture quality adjustment module, configured to: adjust the picture quality of the secondary screen according to the picture quality adjustment strategy.

[0012] According to another embodiment of this application, a storage medium stores a computer program thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the methods described in the embodiments of this application.

[0013] According to another embodiment of this application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the methods described in the embodiments of this application.

[0014] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0015] In this embodiment of the application, in split-screen mode, the device processing pressure is determined based on device-related data; the user attention level of the main screen is determined based on user-related data; the image quality adjustment strategy of the secondary screen is determined based on the device processing pressure and the user attention level; and the image quality of the secondary screen is adjusted according to the image quality adjustment strategy.

[0016] In this embodiment of the application, the image quality adjustment strategy of the secondary screen is dynamically determined by combining the overall device processing pressure of the electronic device and the user's attention to the main screen. While maintaining the key image quality of the main screen, the image quality of the secondary screen is dynamically and flexibly adjusted according to the image quality adjustment strategy. This dynamically and flexibly adjusts resource consumption, improves resource utilization in split-screen mode, and reduces the occurrence of lag, overheating, and high power consumption caused by excessive resource consumption in split-screen mode. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a display processing method according to an embodiment of this application is shown.

[0019] Figure 2 A flowchart illustrating the strategy determination process according to one embodiment of this application is shown.

[0020] Figure 3 A flowchart illustrating image quality adjustment according to an embodiment of this application is shown.

[0021] Figure 4 A block diagram of a display processing apparatus according to an embodiment of this application is shown.

[0022] Figure 5 A block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0023] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present disclosure and are not intended to limit the present disclosure. Furthermore, the embodiments provided below are some embodiments for implementing the present disclosure, and not all embodiments for implementing the present disclosure. Unless otherwise specified, the technical solutions described in the embodiments of the present disclosure can be implemented in any combination.

[0024] It should be noted that, in the embodiments of this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus; for example, a unit may be a portion of circuitry, a portion of a processor, a portion of a program or software, etc.) in the method or apparatus that includes that element.

[0025] For example, the display processing method provided in the embodiments of this disclosure includes a series of steps, but the display processing method provided in the embodiments of this disclosure is not limited to the steps described. Similarly, the display processing apparatus provided in the embodiments of this disclosure includes a series of units, but the apparatus provided in the embodiments of this disclosure is not limited to the units explicitly described, but may also include units that need to be set up for obtaining relevant information or processing based on information.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure.

[0027] It is understood that in the specific implementation of this application, data related to devices and users are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0028] Smart TVs, monitors, projectors, and other electronic devices typically have split-screen functionality. In split-screen mode, multiple screens are displayed on the device, greatly enhancing the user experience. However, currently, electronic devices in split-screen mode usually use a fixed image quality output mode, meaning that multiple screens are displayed with the same fixed image quality. This can easily lead to excessive resource consumption, causing the device to lag, overheat, and experience other abnormal conditions such as high power consumption.

[0029] To address these issues, this application provides a display processing solution that can effectively improve resource utilization in split-screen mode and reduce issues such as lag, overheating, and high power consumption caused by excessive resource consumption in split-screen mode.

[0030] The following is a detailed description of the relevant embodiments of the display processing solution provided in this application.

[0031] Figure 1 A flowchart illustrating an embodiment of an air conditioning control method according to this application is shown. The entity executing this air conditioning control method may be an electronic device with display processing capabilities, such as a television, mobile phone, computer, monitor, in-vehicle equipment, virtual display device, etc.

[0032] like Figure 1 As shown, the display processing method may include steps S110 to S140.

[0033] Step S110: In split-screen mode, determine the device processing pressure based on relevant device data; Step S120: Determine the user attention level of the main screen based on relevant user data; Step S130: Determine the image quality adjustment strategy for the secondary screen based on the device's processing pressure and user attention. Step S140: Adjust the image quality of the secondary screen according to the image quality adjustment strategy.

[0034] In split-screen mode, multiple split screens can be displayed on the screen of an electronic device. These multiple split screens can include at least one main screen (i.e., the main split screen) and at least one secondary screen (i.e., the secondary split screen). For example, in one example, the electronic device can display two split screens, one of which can serve as the main screen and the other as the secondary screen.

[0035] The main screen can be a user-specified sub-screen (i.e., one or more sub-screens that the user is most interested in by default), or a sub-screen whose user attention meets the predetermined attention criteria (such as one or more sub-screens with the highest user attention, or one or more sub-screens with the highest user attention and a duration longer than the predetermined duration). In other words, the user attention of the secondary screen is lower than that of the main screen.

[0036] In split-screen mode, the processing load of the electronic device is determined based on device-related data, and the user engagement level of the main screen is determined based on user-related data. Device processing load refers to the overall processing load of the electronic device; the higher the device processing load, the higher the overall processing load of the electronic device. User engagement level of the main screen refers to the degree of user attention paid to the main screen; the higher the user engagement level of the main screen, the greater the user's attention to the main screen.

[0037] While maintaining the key image quality of the main screen, the image quality adjustment strategy for the secondary screen is determined by combining the overall processing pressure of the electronic device and the user's attention to the main screen. The image quality of the secondary screen is dynamically and flexibly adjusted according to the image quality adjustment strategy (which may include adjustments to resolution, color depth, frame rate and / or display mode, etc.), thereby dynamically and flexibly adjusting resource consumption (i.e. adjusting system load) and improving resource utilization in split-screen mode.

[0038] In summary, the method described in this application embodiment dynamically determines the image quality adjustment strategy for the secondary screen by combining the overall processing pressure of the electronic device and the user's attention to the main screen. While maintaining the key image quality of the main screen, the image quality of the secondary screen is dynamically and flexibly adjusted according to the image quality adjustment strategy. This dynamically and flexibly adjusts resource usage, improves resource utilization in split-screen mode, and reduces issues such as lag, overheating, and high power consumption caused by excessive resource usage in split-screen mode.

[0039] The following description Figure 1 Further optional specific embodiments are provided for each step performed during the display processing in the example.

[0040] In one embodiment, step S110, determining the equipment processing pressure based on equipment-related data, may include the following optional methods: The first method involves weighting and summing the multi-dimensional weighting coefficients and multi-dimensional load data to obtain the equipment processing pressure. Alternatively, the second approach is to determine the device's processing load based on rendering latency data; Alternatively, a third approach is to determine the device's processing load based on the number of buffered frames.

[0041] In the first method, the multi-dimensional weighting coefficients and multi-dimensional load data are weighted and summed to obtain the device processing pressure. The multi-dimensional load data may include, but is not limited to, data related to the Central Processing Unit (CPU), Video Processing Unit (VPU), Graphics Processing Unit (GPU), chip specifications, and bandwidth. The multi-dimensional weighting coefficients can be preset; they can also be dynamically adjusted based on measured multi-dimensional load data and its changing trends.

[0042] In one example, the device processing pressure is obtained by weighted summation of multi-dimensional weighting coefficients and multi-dimensional load data. Specifically, it can be calculated using the formula LV = α * VPU_usage + β * CPU_usage + γ * GPU_usage + δ * Temp_factor + η * (ΔBw / AvgBw), where α, β, γ, δ, and η are the multi-dimensional weighting coefficients, VPU_usage is the VPU utilization rate, CPU_usage is the CPU utilization rate, GPU_usage is the GPU utilization rate, Temp_factor is the chip temperature, ΔBw is the change in bandwidth Bw between two sampling times, and AvgBw is the average bandwidth. The device processing pressure LV can be a scalar value between 0 and 100, with a higher LV indicating greater processing pressure. It can be understood that other formulas can be chosen to calculate the device processing pressure based on device operating status, scenario, user settings, and historical data.

[0043] In the second approach, the device processing load is determined based on rendering latency data. Specifically, the system's native performance APIs (such as Android's SurfaceFlinger / HwComposer) can be used to directly obtain the system's rendering latency data instead of using underlying sensors to collect multi-dimensional load data. The rendering latency data can then be used to determine the device processing load. Alternatively, the device processing load matching the rendering latency data can be queried from a preset load table.

[0044] In the third method, the device processing load is determined based on the number of buffered frames. This can be achieved by using the main decoder software to count the number of buffered frames and using that count as the device processing load, or by querying a preset load table to find a device processing load with a matching number of buffered frames.

[0045] Among the three methods mentioned above, one of them can be randomly selected, selected by the user, or selected according to a preset method to determine the equipment processing pressure; or, after determining the equipment processing pressure using at least two of the three methods, the average of the equipment processing pressures determined by the at least two methods can be taken as the final equipment processing pressure.

[0046] In one embodiment, step S120, determining the user attention level of the main screen based on user-related data, may include: when the user-related data includes user-specified data, determining the user-specified sub-screen as the main screen based on the user-specified data, and determining the preset highest attention level as the user attention level of the main screen; when the user-related data does not include user-specified data, estimating the user attention level of the sub-screen based on at least one of user behavior data and user visual data, and determining the sub-screen whose user attention level meets the predetermined attention conditions as the main screen.

[0047] User-specified data is used to record the sub-screens designated by the user as the main screen. When the user specifies one or more sub-screens as the main screen via remote control, UI buttons, voice commands, etc., the electronic device can record the user-specified data. When user-specified data exists (i.e., when the user specifies the main screen), the user-specified sub-screen is preferentially determined as the main screen based on the user-specified data, and the preset highest attention level (such as 0.9 or 0.95) is determined as the user attention level of the main screen. That is, at this time, there is no need to dynamically estimate the user attention level of the main screen, but the user attention level of the main screen is directly defaulted to the preset highest attention level (such as 0.9 or 0.95).

[0048] Furthermore, when user-specified data is not included (i.e., when the user does not specify a main screen), the user attention of the split screen is estimated based on at least one of user behavior data and user visual data, and the split screen whose user attention meets the predetermined attention conditions is determined as the main screen (e.g., according to the order of user attention from high to low, one or more split screens with the highest user attention are determined as the main screen, or one or more split screens with the highest user attention and a duration longer than the predetermined duration are determined as the main screen).

[0049] When setting a split-screen that meets the predetermined user attention criteria as the main screen, one method is to automatically set it as the main screen; another method is for the electronic device to prompt the user whether to set the split-screen that meets the predetermined attention criteria as the main screen, and, with the user's consent, set it as the main screen.

[0050] User behavior data refers to data related to user behavior on electronic devices. This data can include remote control operation data (such as data related to switching, pausing, fast-forwarding, etc.), screen operation data (screen swiping data, screen switching data, etc.), and / or voice commands (such as voice control of split-screen display content). User visual data refers to data related to user vision. This data can include user eye movement data (such as gaze direction and angle), face orientation data, and split-screen position data.

[0051] Based on at least one of user behavior data and user visual data, the user attention of each split screen can be estimated. Specifically, at least one of user behavior data and user visual data can be input into a preset estimation model. The preset estimation model performs estimation processing based on at least one of the user behavior data and user visual data to obtain the user attention of each split screen.

[0052] In one embodiment, step S120, determining the user attention level of the main screen based on user-related data, includes: performing predictive analysis based on user-related data and screen display data within a preset time period to obtain the main screen and user attention level.

[0053] In this embodiment, based on user-related data and screen display data from a preset time period prior to the current time, the main screen's performance and user attention in the future are predicted in advance. Then, based on the device's processing load and the future performance of the main screen and user attention, a secondary screen image quality adjustment strategy can be determined. The secondary screen's image quality is adjusted in advance according to this strategy, further avoiding lag in resource usage adjustments. The time difference between the current time and the future time can be set according to actual conditions; for example, the time difference can be 1 second, 2 seconds, or 5 seconds.

[0054] User-related data within a given time period may include, but is not limited to, user behavior data and user visual data within the preset time period; screen display data within the preset time period may include the content displayed in each sub-screen within the preset time period and the position of each sub-screen, etc.

[0055] Predictive analysis is performed based on user-related data and screen display data within a preset time period to obtain the main screen and user attention. Specifically, this can be done by inputting user-related data and screen display data within a preset time period into a preset prediction model. The preset main screen prediction model performs prediction processing based on at least one of the user-related data and screen display data within the preset time period to obtain the main screen and user attention at future times.

[0056] See Figure 2In one embodiment, step S130, determining the image quality adjustment strategy for the secondary screen based on device processing pressure and user attention, may specifically include: step S210, when the device processing pressure is greater than or equal to a first preset pressure and user attention is greater than a median attention level, the image quality adjustment strategy is to reduce the resolution or color depth of non-critical areas in the secondary screen; step S220, when the device processing pressure is greater than or equal to the first preset pressure and user attention is less than a median attention level, the image quality adjustment strategy is to reduce the overall resolution or overall color depth of the secondary screen; step S230, when the device processing pressure is greater than or equal to a second preset pressure and user attention is less than a median attention level, the image quality adjustment strategy is to reduce the frame rate of the secondary screen; step S240, when the device processing pressure is greater than or equal to the second preset pressure and user attention is greater than an upper limit attention level, or when the pressure difference between the device processing pressure and the preset upper limit pressure is less than a preset difference, the image quality adjustment strategy is to display the secondary screen statically.

[0057] The pressure range is defined as a preset lower pressure limit (e.g., 0) to a preset upper pressure limit (e.g., 100). The first and second preset pressures are located within this range. The first and second preset pressures can be set according to actual conditions, and this application does not impose any special limitations on them. The median level of attention is the midpoint of the attention range; when the attention range is 0 to 1, the median level of attention equals 0.5. The upper limit of attention can be set according to actual conditions, and this application does not impose any special limitations on it. For example, the upper limit of attention could be equal to 0.9 or 0.95, etc.

[0058] Taking a scenario where "the pressure range is 0 to 100, the first preset pressure is 60, the second preset pressure is 80, the attention range is 0 to 1, the middle attention is 0.5, the upper limit attention can be 0.9, and the preset difference is 0.02" as an example, the specific strategy for adjusting the secondary screen's image quality based on the device's processing pressure (LV) and the user's attention (P) can include: (1) Level 0 (LV<60, P=0.5): The image quality of the secondary screen is not adjusted (the main screen and the secondary screen can be in full HD). (2) Level 1 (LV≥60, P>0.5): The image quality adjustment strategy of the secondary screen is to reduce the resolution or color depth of non-critical areas (i.e., areas outside the critical areas of the secondary screen) (e.g., the resolution or color depth of non-critical areas of the secondary screen is reduced by a set ratio). This can significantly reduce the system load without affecting the user's acquisition of key information of the secondary screen. (3) Level 3 (LV≥60, P<0.5): The image quality adjustment strategy is to reduce the overall resolution or overall color depth of the secondary screen (e.g., the resolution or color depth of all areas of the secondary screen is reduced by a set ratio). (4) Level 4 (LV≥80, P<0.5): The image quality adjustment strategy is to reduce the frame rate of the secondary screen (e.g., from 60fps to 30fps). (5) Level 5 (LV≥80, P>0.9, or, when the pressure difference between LV and 100 is less than the preset difference of 0.02): The image quality adjustment strategy is to display statically on the secondary screen (such as displaying the preset static logo / cover on the secondary screen or pausing the secondary screen display, etc.).

[0059] It is understood that, optionally, in other embodiments, in step S130, the image quality adjustment strategy of the secondary screen is determined according to the device processing pressure and user attention. Specifically, it may include: dividing the screen into more ranges according to the device processing pressure LV and user attention P, and setting a corresponding image quality adjustment strategy for the secondary screen for each range.

[0060] Furthermore, in one embodiment, the user can actively select the image quality adjustment strategy for the secondary screen, and the image quality adjustment of the secondary screen will be performed preferentially according to the image quality adjustment strategy selected by the user. In another embodiment, if the user does not actively select the image quality adjustment strategy for the secondary screen, the image quality adjustment strategy for the secondary screen will be determined based on the device processing load and the user's attention level, and the image quality of the secondary screen will be adjusted accordingly.

[0061] Furthermore, in one embodiment, it may also include: determining key areas and non-key areas in the secondary screen based on user-related data and screen display data; or, determining areas outside the predetermined key content areas in the secondary screen as non-key areas.

[0062] In one approach, key and non-key areas in the secondary screen can be determined based on user-related data and screen display data. This involves combining user behavior and screen display data to accurately identify the key areas (ROI areas) and non-key areas that users need to focus on. Specifically, a preset area prediction model can perform predictive processing based on user-related data and screen display data within a preset time period to obtain the key areas in the secondary screen at future times. Areas outside the key areas in the secondary screen are considered non-key areas.

[0063] In another approach, a predetermined key content area (such as the area containing predetermined key content like faces or text) is directly detected in the secondary screen, and the area outside the predetermined key content area in the secondary screen is identified as a non-key area.

[0064] See Figure 3 In one embodiment, step S140, adjusting the image quality of the secondary screen according to the image quality adjustment strategy, may include: step S310, adjusting the image quality of the secondary screen according to the image quality adjustment strategy using a region-by-region update method; or step S320, adjusting the image quality of the secondary screen according to the image quality adjustment strategy using a frame-by-frame gradual update method. Adjusting the image quality of the secondary screen using either a region-by-region update method or a frame-by-frame gradual update method can achieve smooth switching of image quality on the secondary screen, avoiding screen flickering or tearing.

[0065] When adjusting the image quality of the secondary screen using a region-by-region update method according to the image quality adjustment strategy, for example, if the image quality adjustment strategy is to reduce the overall resolution of the secondary screen, the secondary screen is divided into region 1 and region 2. The resolution of region 1 in the secondary screen is reduced in the first frame, and the resolution of region 1 and region 2 in the secondary screen is reduced again in the second frame.

[0066] When adjusting the image quality of the secondary screen using a frame-by-frame gradual update method according to the image quality adjustment strategy, for example, if the image quality adjustment strategy is to reduce the overall resolution of the secondary screen by 20%, the overall resolution of the secondary screen is reduced by 10% in the first frame and then reduced by 20% in the second frame.

[0067] Optionally, in other embodiments, step S140, adjusting the image quality of the secondary screen according to the image quality adjustment strategy, may include: a one-step approach, adjusting the image quality of the secondary screen according to the image quality adjustment strategy. For example, if the image quality adjustment strategy is to reduce the overall resolution of the secondary screen by 20%, the overall resolution of the secondary screen is directly reduced by 20% in the first frame.

[0068] This involves splitting content transmission between the main screen and the secondary screen. For example, the content of the main screen is transmitted to the main screen for display with a preset image quality, while the content of the secondary screen is transmitted to the secondary screen for display with an adjusted image quality, ensuring consistent screen layout and low latency.

[0069] In the foregoing embodiments of this application, the preset estimation model performs estimation processing based on at least one of user behavior data and user visual data to obtain the user attention level for each split screen. The preset estimation model is a pre-trained first deep learning model, such as a CNN (Convolutional Neural Network) model, an LSTM (Long Short-Term Memory Network) model, or a Large Language Model (LLM), etc.

[0070] Taking the first deep learning model as a CNN model as an example, the CNN model can include an input layer, a convolutional layer, an activation function layer, a pooling layer, a fully connected layer, and an output layer. At least one of the user behavior data and user visual data passes through the input layer, convolutional layer, activation function layer, pooling layer, fully connected layer, and output layer in turn. Finally, the output layer outputs the user attention of each split screen.

[0071] The training process of the preset estimation model may include: acquiring first training sample data, which includes different first data samples (i.e., at least one of user behavior data and user visual data as samples) and preset screen-by-screen attention corresponding to different first data samples; using a first deep learning model to analyze the first data samples in the training sample data and output predicted screen-by-screen attention; using a loss function to calculate the loss based on the predicted screen-by-screen attention and the preset screen-by-screen attention; and calculating the gradient of the loss and adjusting the weights of each layer in the first deep learning model according to the gradient; repeating the above steps until the training stopping condition is met (such as the analysis accuracy of the first deep learning model reaching a preset accuracy or the number of training iterations reaching a preset number of iterations, etc.), then the trained preset estimation model can be obtained. The loss function can be selected according to the actual situation, and this application does not impose any special limitations on it. For example, a loss function may be the cross-entropy loss function.

[0072] In the foregoing embodiments of this application, the preset main screen prediction model performs prediction processing based on at least one of user-related data and screen display data within a preset time period to obtain the main screen and user attention at future times. The preset main screen prediction model is a pre-trained second deep learning model, such as a CNN (Convolutional Neural Network) model, an LSTM (Long Short-Term Memory Network) model, or a Large Language Model (LLM), etc.

[0073] Taking the second deep learning model as a CNN model as an example, the CNN model can include an input layer, a convolutional layer, an activation function layer, a pooling layer, a fully connected layer, and an output layer. At least one of the user-related data and screen display data within a preset time period passes through the input layer, convolutional layer, activation function layer, pooling layer, fully connected layer, and output layer in sequence. Finally, the output layer outputs the main screen and user attention at a future time.

[0074] The training process of the preset home screen prediction model may include: acquiring second training sample data, which includes different second data samples (i.e., at least one of user-related data and screen display data within a preset time period as samples) and preset home screens and preset home screen attention corresponding to different second data samples; using a second deep learning model to analyze the second data samples in the training sample data and output the predicted home screen and predicted home screen attention; using a loss function to calculate the loss based on the preset home screen and preset home screen attention and the predicted home screen and predicted home screen attention; and calculating the gradient of the loss and adjusting the weights of each layer in the second deep learning model according to the gradient; repeating the above steps until the training stopping condition is met (such as the analysis accuracy of the second deep learning model reaching a preset accuracy or the number of training iterations reaching a preset number of iterations, etc.), then the trained preset home screen prediction model can be obtained. The loss function can be selected according to the actual situation, and this application does not impose any special limitations on it. For example, a loss function may be the cross-entropy loss function.

[0075] To facilitate better implementation of the display processing method provided in the embodiments of this application, the embodiments of this application also provide a display processing apparatus based on the above-described display processing method. The meanings of the terms used are the same as in the above-described display processing method, and specific implementation details can be found in the descriptions in the method embodiments. Figure 4 A block diagram of a display processing apparatus according to an embodiment of this application is shown.

[0076] like Figure 4 As shown, the display processing device 400 may include: a pressure determination module 410, which can be used to: determine the device processing pressure based on device-related data in split-screen mode; an attention determination module 420, which can be used to: determine the user attention of the main screen based on user-related data; a strategy determination module 430, which can be used to: determine the image quality adjustment strategy of the secondary screen based on the device processing pressure and the user attention; and an image quality adjustment module 440, which can be used to: adjust the image quality of the secondary screen according to the image quality adjustment strategy.

[0077] In some embodiments of this application, when determining the user attention level of the main screen based on user-related data, the attention level determination module 420 can be used to: when the user-related data includes user-specified data, determine the user-specified sub-screen as the main screen based on the user-specified data, and determine the preset highest attention level as the user attention level of the main screen; when the user-related data does not include the user-specified data, estimate the user attention level of the sub-screen based on at least one of the user behavior data and the user visual data, and determine the sub-screen whose user attention level meets the predetermined attention conditions as the main screen.

[0078] In some embodiments of this application, when determining the user attention level of the main screen based on user-related data, the attention level determination module 420 can be used to: perform predictive analysis based on the user-related data and screen display data within a preset time period to obtain the main screen and the user attention level.

[0079] In some embodiments of this application, when determining the image quality adjustment strategy for the secondary screen based on the device processing pressure and the user attention, the strategy determination module 430 can be used to: reduce the resolution or color depth of non-critical areas in the secondary screen when the device processing pressure is greater than or equal to a first preset pressure and the user attention is greater than an intermediate attention level; reduce the overall resolution or overall color depth of the secondary screen when the device processing pressure is greater than or equal to the first preset pressure and the user attention is less than the intermediate attention level; reduce the frame rate of the secondary screen when the device processing pressure is greater than or equal to the second preset pressure and the user attention is greater than an upper limit attention level, or when the pressure difference between the device processing pressure and the preset upper limit pressure is less than a preset difference value, the image quality adjustment strategy is to statically display the secondary screen.

[0080] In some embodiments of this application, the device further includes a region detection module that can be used to: determine the key regions and non-key regions in the secondary screen based on user-related data and screen display data; or, determine the non-key regions as regions outside the predetermined key content regions in the secondary screen.

[0081] In some embodiments of this application, when adjusting the image quality of the secondary screen according to the image quality adjustment strategy, the image quality adjustment module 440 can be used to: adjust the image quality of the secondary screen according to the image quality adjustment strategy in a region-by-region block-by-block update manner; or, adjust the image quality of the secondary screen according to the image quality adjustment strategy in a frame-by-frame gradual update manner.

[0082] In some embodiments of this application, when determining the device processing pressure based on device-related data, the pressure determination module 410 can be used to: perform a weighted summation of multi-dimensional weight coefficients and multi-dimensional load data to obtain the device processing pressure; or, determine the device processing pressure based on rendering latency data; or, determine the device processing pressure based on the number of buffered frames.

[0083] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0084] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, Figure 5 A block diagram of an electronic device according to an embodiment of this application is shown, specifically: The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 501 is the control center of the electronic device, connecting various parts of the computer device via various interfaces and lines. It executes software programs and / or modules stored in the memory 502, and calls data stored in the memory 502, to perform various functions of the computer device and process data. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and application programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0085] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0086] The electronic device also includes a power supply 503 that supplies power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0087] The electronic device may also include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0088] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device can load the executable files corresponding to the processes of one or more computer programs into the memory 502 according to the following instructions, and the processor 501 runs the computer programs stored in the memory 502, thereby realizing the various functions in the foregoing embodiments of this application.

[0089] For example, processor 501 can perform the following: in split-screen mode, determine the device processing pressure based on device-related data; determine the user attention level of the main screen based on user-related data; determine the image quality adjustment strategy for the secondary screen based on the device processing pressure and the user attention level; and adjust the image quality of the secondary screen according to the image quality adjustment strategy.

[0090] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0091] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the methods provided in embodiments of this application.

[0092] The storage medium can be a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0093] Since the computer program stored in the storage medium can execute the steps of any of the methods provided in the embodiments of this application, the beneficial effects that the methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0094] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0095] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0096] It should be understood that this application is not limited to the embodiments described above and shown in the accompanying drawings, but various modifications and changes can be made without departing from its scope.

Claims

1. A display processing method, characterized in that, include: In split-screen mode, the device's processing load is determined based on relevant device data; Determine user engagement on the home screen based on relevant user data; Based on the device's processing load and the user's attention level, determine the image quality adjustment strategy for the secondary screen; The secondary screen is configured to adjust its image quality according to the aforementioned image quality adjustment strategy.

2. The method according to claim 1, characterized in that, The process of determining user attention on the main screen based on relevant user data includes: When the user-related data includes user-specified data, the user-specified split screen is determined as the main screen based on the user-specified data, and the preset highest attention level is determined as the user attention level of the main screen; When the user-related data does not include the user-specified data, the user attention level of the split screen is estimated based on at least one of user behavior data and user visual data, and the split screen whose user attention level meets the predetermined attention conditions is determined as the main screen.

3. The method according to claim 1, characterized in that, The process of determining user attention on the main screen based on relevant user data includes: Based on the user-related data and screen display data within a preset time period, predictive analysis is performed to obtain the main screen and the user attention level.

4. The method according to claim 1, characterized in that, The step of determining the image quality adjustment strategy for the secondary screen based on the device processing load and the user's attention level includes: When the device processing pressure is greater than or equal to the first preset pressure and the user attention is greater than the intermediate attention, the image quality adjustment strategy is to reduce the resolution or color depth of non-critical areas in the secondary screen. When the device processing pressure is greater than or equal to the first preset pressure and the user attention is less than the intermediate attention, the image quality adjustment strategy is to reduce the overall resolution or overall color depth of the secondary screen. When the device processing pressure is greater than or equal to the second preset pressure and the user attention is less than the intermediate attention, the image quality adjustment strategy is to reduce the frame rate of the secondary screen. When the device processing pressure is greater than or equal to the second preset pressure and the user attention is greater than the upper limit attention, or when the pressure difference between the device processing pressure and the preset upper limit pressure is less than the preset difference, the image quality adjustment strategy is to display the secondary screen in a static manner.

5. The method according to claim 4, characterized in that, The method further includes: The key areas and non-key areas in the secondary screen are determined based on user-related data and screen display data. Alternatively, the area outside the predetermined key content area in the secondary screen can be defined as the non-key area.

6. The method according to claim 1, characterized in that, The step of adjusting the image quality of the secondary screen according to the image quality adjustment strategy includes: The secondary screen is updated in a block-by-block manner according to the image quality adjustment strategy; Alternatively, the secondary screen can be adjusted in a frame-by-frame gradual update manner according to the image quality adjustment strategy.

7. The method according to claim 1, characterized in that, The step of determining the equipment processing pressure based on relevant equipment data includes: The processing pressure of the device is obtained by weighting and summing the multi-dimensional weight coefficients and multi-dimensional load data. Alternatively, the processing load of the device can be determined based on the rendering latency data; Alternatively, the processing pressure of the device can be determined based on the number of buffered frames.

8. A display processing device, characterized in that, include: The pressure determination module is used to: determine the device processing pressure based on relevant device data in split-screen mode; The attention determination module is used to: determine the user attention level of the main screen based on relevant user data; The strategy determination module is used to: determine the image quality adjustment strategy of the secondary screen based on the device processing pressure and the user attention. The image quality adjustment module is used to adjust the image quality of the secondary screen according to the image quality adjustment strategy.

9. A storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores computer programs; A processor reads a computer program stored in memory to perform the method described in any one of claims 1 to 7.

Images