Information display optimization method and equipment for intelligent equipment and medium

Abstract

The embodiment of the invention discloses an information display optimization method and device for an intelligent device and a medium, and relates to the technical field of intelligent device.The method comprises the steps that real-time eye movement data and current device display information of a user are collected, and a pupil movement prediction model and the real-time eye movement data which are constructed in advance are used for predicting the pupil movement of the user; predicting the visual position of the user, and determining a predicted attention area of the user; analyzing current equipment display information, and determining at least one corresponding current information dense area and an information element display weight corresponding to each current information dense area; information arrangement is carried out through the current information dense area and the predicted attention area, initial delivery parameters corresponding to the target delivery object are determined, and the initial delivery parameters comprise position parameters and display parameters; and based on the information element display weight corresponding to each current information dense area, adjusting the display parameter of the initial putting parameter, determining a target putting parameter, and performing display putting on the target putting object.

Description

Technical Field

[0001] The present invention relates to the field of smart device display technology, and in particular to an information display optimization method, device and medium for smart devices. Background Art

[0002] In today's era of widespread popularity of smart devices, from daily mobile phones and tablets to emerging augmented reality (AR) and virtual reality (VR) devices, efficient display of information and improvement of user experience are in an unprecedented critical position. As emerging smart products, augmented reality (AR) and virtual reality (VR) devices are gradually changing people's entertainment, education, and work methods. In the AR shopping experience, users expect to see the virtual display of the product and related detailed information, such as size, material, and usage effect, clearly and intuitively through the device; if this information cannot be displayed efficiently, users may give up buying the product because it is difficult to fully understand the product. In the VR education scene, relying on the device to immersively learn history, geography and other knowledge, if the information is chaotic, such as unclear introductions of key figures and events in historical scenes, and unintuitive display of topographic information in geography courses, it will seriously affect the learning effect and reduce users' recognition of this emerging education method. However, there are many problems in the information display of current smart devices, which hinder the efficient display of information and the improvement of user experience.

[0003] On the one hand, traditional display methods often ignore the actual visual focus of users. Whether it is an application based on a fixed interface layout or a system that relies on simple information push, it is difficult to accurately grasp the area that users are about to focus on, making information delivery blind and important information difficult for users to obtain in the first place. Users have to spend extra time and energy searching for the content they need on the screen, which not only wastes users' time, but may also lead to user loss due to poor user experience. On the other hand, when the information on the screen is complex, elements are prone to accumulation in information-intensive areas. In the limited screen space, if these elements lack reasonable display allocation, they will interfere with each other.

[0004] Therefore, the information delivery in the traditional display method ignores the changes in the user's actual visual focus, and there is a risk of information accumulation in the display of information, which leads to the problem of blind information delivery and poor display effect, affecting the user's experience of using smart devices. Summary of the invention

[0005] One or more embodiments of the present specification provide an information display optimization method, device and medium for smart devices, which are used to solve the following technical problems: the information delivery in the traditional display method ignores the changes in the user's actual visual focus, and there is a risk of information accumulation in the display of information, resulting in blind information delivery and poor display effect, affecting the user's experience of using the smart device.

[0006] One or more embodiments of this specification adopt the following technical solutions:

[0007] One or more embodiments of the present specification provide an information display optimization method for smart devices, the method comprising: collecting real-time eye movement data of a user and current device display information, using a pre-built pupil movement prediction model and the real-time eye movement data to predict the user's visual position and determine the user's predicted attention area; analyzing the current device display information to determine at least one corresponding current information-intensive area and the information element display weight corresponding to each current information-intensive area; arranging information through the current information-intensive area and the predicted attention area to determine the initial delivery parameters corresponding to the target delivery object, wherein the initial delivery parameters include position parameters and display parameters; based on the information element display weight corresponding to each current information-intensive area, adjusting the display parameters of the initial delivery parameters to determine the target delivery parameters, so as to display and deliver the target delivery object through the target delivery parameters.

[0008] One or more embodiments of this specification provide an information display optimization device for a smart device, including:

[0009] at least one processor; and,

[0010] a memory communicatively connected to the at least one processor; wherein,

[0011] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0012] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0013] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: through the technical solution of the embodiments of this specification, the traditional display method cannot perceive the dynamic changes of the user's visual focus. The embodiments of this specification collect the user's real-time eye movement data and use the pre-built pupil movement prediction model to track and accurately predict the user's visual position in real time and determine the predicted focus area; the location that the user may focus on can be known in advance, and based on this, the target delivery object can be accurately delivered to the vicinity of the area or a reasonable location, so that important information is highly matched with the user's focus, reducing the user's time and energy in finding information and improving the efficiency of information acquisition; the traditional display method is prone to information accumulation, resulting in poor display effect, and it is difficult for users to distinguish key information; the embodiments of this specification conduct an in-depth analysis of the information displayed by the current device, and determine at least one current information-intensive area and the display weight of the information elements in each area. In this way, complex information can be clearly sorted out and the importance of each information element can be clarified; Based on the information element display weight, the display parameters in the initial delivery parameters of the target delivery object are adjusted, which not only ensures the prominence of key information, but also reasonably utilizes the page space, effectively solves the problem of information accumulation, improves the information display effect, enables users to easily obtain key information, and greatly improves the user experience of smart devices; by determining information-intensive areas and accurate information arrangement, smart devices can focus display resources on areas and important information elements that users actually pay attention to; due to blind information delivery and information accumulation in traditional display methods, users need to spend a lot of energy to screen information, which can easily cause visual fatigue and high cognitive burden. The embodiments of this specification optimize the precise information delivery and reasonable information display, so that users no longer need to search in a large amount of useless information, reducing eye fatigue and brain processing pressure; traditional display methods cannot meet personalized needs. The embodiments of this specification optimize information according to each user's unique real-time eye movement data, thereby achieving highly personalized information delivery. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0015] Figure 1 A flowchart of an information display optimization method for a smart device provided in an embodiment of this specification;

[0016] Figure 2A schematic diagram of the structure of an information display optimization device for smart devices provided in an embodiment of this specification. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0018] The embodiments of this specification provide an information display optimization method for smart devices. It should be noted that the execution subject in the embodiments of this specification can be a server or any device with data processing capabilities. Figure 1 A flowchart of an information display optimization method for a smart device provided in an embodiment of this specification is shown in FIG. Figure 1 As shown, it mainly includes the following steps:

[0019] Step S101, collect the user's real-time eye movement data and current device display information, use the pre-built pupil movement prediction model and real-time eye movement data to predict the user's visual position, and determine the user's predicted attention area.

[0020] In one embodiment of the present specification, the pupil position and movement trajectory of the user are captured in real time by a low-resolution camera and sensor of a smart device. The low-resolution camera continuously captures image frames containing the user's eyes, focuses light on the image sensor through an optical system, and the image sensor converts the light signal into an electrical signal to generate digital image data. The image data obtained by the camera is subjected to an image processing algorithm, such as a method based on color features, edge detection or template matching, to identify the position of the pupil in the image. The movement trajectory of the pupil is calculated by the change in the pupil position in multiple consecutive frames of images. For example, in a method based on color features, the difference in color between the pupil and the surrounding area is used to segment the pupil from the image by setting a suitable color threshold, thereby determining its position. The sensor can assist in providing other relevant information, such as head posture data, to help more accurately locate the actual position of the pupil in space, thereby collecting the user's real-time eye movement data. In addition, the current device display information is collected from the display screen of the smart device. The user's visual position is predicted by a pre-built pupil movement prediction model and real-time eye movement data to determine the user's predicted focus area.

[0021] The method uses a pre-built pupil movement prediction model and the real-time eye movement data to predict the user's visual position and determine the user's predicted area of ​​interest, specifically including: collecting the user's historical pupil movement data, wherein the historical pupil movement data includes multiple pupil positions within a preset historical time period; determining the pupil movement prediction model through a preset recurrent neural network model and historical pupil movement data, so as to use the real-time eye movement data to determine the user's corresponding predicted pupil position; obtaining the pixel occupancy parameters of the pixel area occupied by the target delivery object, and determining the predicted area of ​​interest with the predicted pupil position as the center and the pixel occupancy parameters.

[0022] In one embodiment of the present specification, the user's historical pupil movement data is collected, which records the user's pupil position and movement trajectory information over the past period of time. At the same time, combined with the real-time eye movement data currently obtained through the camera and sensor, it is used as the input of the prediction model. In addition, the current device display information can also be used as one of the input information of the model. For example, the content currently displayed by the device may affect the user's visual focus, and the model can learn the association between different display contents and the user's pupil movement. The model here can be a micro LSTM (long short-term memory network) or a convolutional neural network (CNN) as the basic model architecture. Micro LSTM is suitable for processing time series data because it can capture long-term dependencies in long sequence data. It is very suitable for data with time series characteristics such as pupil movement. For example, there is a certain regularity in the user's pupil movement trajectory over a period of time. Micro LSTM can learn this regularity and use it to predict future pupil positions. If the pupil movement data is converted into a form suitable for CNN processing, such as time series imaging (arranging pupil movement data in time sequence into an image-like form) or constructing a multi-dimensional feature image (using features such as pupil position and size change as different channels to construct a multi-channel image), the convolutional neural network can automatically extract features from the data through the convolution layer to predict the future position of the pupil. The model selection is not specifically limited here.

[0023] After the model is selected, in order to be suitable for low-power devices, a lightweight machine learning model that has undergone pruning, quantization, and model distillation is required to meet the low power requirements of the end-side device. Analyze the importance of each connection or neuron in the model to the prediction result, and remove the connections or neurons that have little impact on the prediction accuracy. For example, by calculating the size of the weight, the connection with a smaller weight value is pruned, thereby reducing the number of model parameters and reducing the computational complexity. In addition, the weights and activation values ​​in the model are converted from high-precision data types (such as 32-bit floating point numbers) to low-precision data types (such as 8-bit integers) to significantly reduce the storage requirements and computational complexity of the model. For example, after the weight data type is converted, the model storage size can be reduced to 1 / 4 of the original. Finally, a simple small model (student model) is trained with a large model (teacher model) with good performance but complex calculations as a guide. Let the student model learn the output distribution or feature representation of the intermediate layer of the teacher model, so as to achieve lightweight model while maintaining certain performance. For example, the student model learns the soft label (output with probability information) of the teacher model, so that the student model can better capture the uncertainty and distribution information in the data and improve the prediction performance.

[0024] The prepared data is input into the pre-built and optimized pupil movement prediction model. The model uses the computing logic of micro LSTM or CNN to predict the future position of the user's pupil based on the input historical movement data and real-time eye movement data, combined with its learned patterns and features. For example, micro LSTM processes time series data through its internal memory unit and gating mechanism and outputs the predicted pupil position; CNN extracts features and predicts the future position of the pupil by performing convolution and pooling operations on the converted image data.

[0025] Get the pixel occupancy parameters of the pixel area occupied by the target delivery object. The pixel occupancy parameters here include the pixel size corresponding to the target delivery object. Generally, the shape of the delivery object corresponding to the pixel size of the target delivery object is a regular shape, such as a rectangular area. With the predicted pupil position as the center, the predicted area of ​​interest is determined with the pixel size of the pixel occupancy parameter. The shape of the determined predicted area of ​​interest is the same as the shape of the delivery object. If it is an irregular shape, determine the minimum regular shape that can include the pixel size corresponding to the target delivery object, and determine the predicted area of ​​interest with the size corresponding to this minimum regular shape. The predicted area of ​​interest is the predicted screen area that the user may pay attention to next. Information can be optimized and displayed and delivered based on this area in the future.

[0026] Step S102: Analyze the current device display information to determine at least one corresponding current information-intensive area and the information element display weight corresponding to each current information-intensive area.

[0027] Analyze the current device display information to determine at least one corresponding current information intensive area and the information element display weight corresponding to each current information intensive area, specifically including: perform visual density detection on the current device display information to determine the information display density of each preset unit area, and determine at least one current information intensive area in the preset unit area based on the information display density, wherein the information display density of the current information intensive area is higher than a preset density threshold; obtain element evaluation data of the information element in each of the current information intensive areas, wherein the element evaluation data includes an element importance evaluation index, an element timeliness evaluation index and a task relevance evaluation index; and determine the information element display weight corresponding to each of the current information intensive areas according to the element evaluation data.

[0028] In one embodiment of the present specification, image data for display information is obtained from the current device. For example, data is read from a screen buffer, a screenshot is taken, or an image stream is obtained through a specific device interface. The obtained color image is converted into a grayscale image, and the grayscale image is subjected to noise reduction processing to smooth the image and remove noise interference, while retaining the main structural information of the image.

[0029] Perform visual density detection on the current device display information to determine the information display density of each preset unit area, specifically including: obtaining pixel occupancy parameters of the pixel area occupied by the target delivery object; dividing the current device display information into regions based on the pixel occupancy parameters of the occupied pixel area to determine multiple preset unit areas; identifying element edges of information elements in each of the preset unit areas to determine the number of element edge pixels in each of the preset unit areas; counting the number of statistical pixels in each of the preset unit areas, and determining the information display density corresponding to each of the preset unit areas based on the ratio of the number of element edge pixels in each of the preset unit areas to the corresponding number of statistical pixels.

[0030] In one embodiment of the present specification, the pixel area occupied by the target delivery object on the screen is determined, and this is used as a standard to divide the display information of the entire screen. For a target delivery object of a regular shape, such as a rectangle, the pixel occupancy parameters can be determined by its upper left corner coordinates and width and height. In practical applications, this may involve obtaining these coordinates and size information from a graphics rendering engine, interface layout data, or directly parsing the image content. If the target delivery object is an irregular shape, the pixel occupancy parameters are determined by the pixel area of ​​the smallest regular shape that can contain the target delivery object. According to the pixel occupancy parameters of the target delivery object, the entire area of ​​the current device display information is divided into a plurality of preset unit areas of the same size. Assuming that the target delivery object is a rectangle with a width of 50 pixels and a height of 30 pixels, the entire screen can be gridded with a size of 50×30 pixels, ensuring that each unit area is consistent in size and shape, which is convenient for subsequent unified analysis and processing. Dividing the preset unit area makes it possible to analyze complex screen information. After the entire screen information is split into multiple small areas, independent information density analysis can be performed for each small area, thereby accurately locating the information-intensive area. The edge of the information element in the image is identified by edge detection algorithms, such as the Canny edge detection algorithm. The edge of the information element represents the boundary or important feature between different information. The number of edge pixels reflects the richness and complexity of the information in the area to a certain extent. By identifying the edge, the activity of the information in each preset unit area can be quantified. For each preset unit area, the total number of pixels contained in it is calculated, that is, the width of the area multiplied by the height. For example, a preset unit area with a width of 50 pixels and a height of 30 pixels has a total number of pixels of 50×30=1500. The ratio of the number of element edge pixels in each preset unit area divided by the total number of pixels in the area is the information display density of the preset unit area, which reflects the proportion of the edge of the information element in the area to the entire area. The higher the ratio, the greater the information display density of the area, that is, the more dense and complex the information is.

[0031] In one embodiment of the present specification, at least one current information-intensive area is determined in a plurality of preset unit areas according to the information display density, and the information display density of the current information-intensive area is higher than the preset density threshold, where the preset density threshold can be set to 60% of the total number of pixels in each preset unit area, and the current information-intensive area refers to a preset unit area whose information display density is higher than 60% of the total number of pixels in the preset unit area. Through a pre-constructed information element evaluation index system, element evaluation data of each information element in the current information-intensive area is obtained, wherein the element evaluation data includes an element importance evaluation index, an element timeliness evaluation index, and a task relevance evaluation index. Different types of information have large differences in importance. Taking news applications as an example, news headlines are often a high-level summary of the entire news event, and their importance is higher than that of ordinary paragraphs in the text. In financial applications, real-time data of stock prices is more critical than secondary annotation information in historical price trend charts. By pre-defining the importance levels of different types of information, the importance of elements can be preliminarily determined. For information with clear time stamps, such as news release time and weather forecast update time, the timeliness is measured by calculating the difference between the current time and the information release time. The smaller the difference, the higher the timeliness. For example, a breaking news released a few minutes ago is more timely than ordinary news released a day ago. For real-time and dynamic information such as stock prices and sports scores, the update frequency reflects the importance of timeliness. Information with a high update frequency means that it has strong timeliness. In addition, the user's current operation behavior is collected, and the task relevance of the information element and the task corresponding to the current operation behavior is determined through the user's current operation behavior. If the user is viewing the "travel guide", the task relevance of the information elements related to the introduction of tourist destination attractions and hotel recommendations on the page is very high; while the ad space information at the bottom of the page, if it is not related to travel, has a low task relevance. By analyzing the user's search keywords, browsing history, click behavior, etc. in the application, it is possible to determine the degree of fit between the information element and the user's current task. It should be noted that the evaluation index can be represented by the indicator values ​​corresponding to the three levels of low, medium and high. The indicator values ​​corresponding to the three levels are pre-set. For example, the low level value is -1, the medium level value is 0, and the high level value is 1. According to the above-mentioned information element evaluation method, the multiple evaluation levels corresponding to the information element are first determined, and the element importance evaluation index, element timeliness evaluation index and task relevance evaluation index of the information element are determined according to the corresponding indicator value method. And the information element display weight of each information element in each current information-intensive area is determined by summing. If there are multiple information elements, they can be stored in the form of a data table. Set the position of each information element and the corresponding information element display weight in the data table.It should be noted that, in addition to determining the information element display weight for the information-intensive area, it is also necessary to determine the corresponding information element display weight for each preset unit area in the above manner.

[0032] Through the above technical solution, the information display density is determined by visual density detection, which can accurately find the information-dense areas in the information displayed by the current device, and help identify information areas that may make users feel visually overloaded or difficult to quickly obtain key content; the information elements in each information-dense area are evaluated and the display weight is determined, and the information elements in the information-dense area can be quantified to facilitate the information arrangement of subsequent target delivery objects.

[0033] Step S103, arranging information through the current information-intensive area and the predicted focus area, and determining the initial delivery parameters corresponding to the target delivery object.

[0034] The initial delivery parameters include position parameters and display parameters;

[0035] Arranging information through the current information-intensive area and the predicted focus area to determine the initial delivery parameters corresponding to the target delivery object, specifically including: obtaining the focus area position of the predicted focus area and the dense area position of the current information-intensive area; determining the relative position relationship between the predicted focus area and the current information-intensive area according to the focus area position and multiple dense area positions, wherein the relative position relationship includes an overlapping relationship and a non-overlapping relationship; adjusting the predicted focus area according to the relative position relationship between the predicted focus area and the current information-intensive area to determine the initial delivery parameters corresponding to the target delivery object.

[0036] In one embodiment of the present specification, first, the position of the predicted focus area (i.e., the focus area position) and the position of the current information-intensive area (i.e., the intensive area position) are obtained, which can be represented by pixel coordinates on the screen. Based on the acquired position information, the relative position relationship between the predicted focus area and the current information-intensive area is determined. The relative position relationship is divided into an overlapping relationship and a non-overlapping relationship. The overlapping relationship indicates that the two areas partially or completely overlap in space; the non-overlapping relationship indicates that the two areas do not have any spatial intersection. Through the relative position relationship between the predicted focus area and the current information-intensive area, the predicted focus area is adjusted to determine the initial delivery parameters corresponding to the target delivery object.

[0037] The predicted focus area is adjusted based on the relative position relationship between the predicted focus area and the current information-intensive area, and the initial delivery parameters corresponding to the target delivery object are determined, specifically including: when the relative position relationship is a non-overlapping relationship, the predicted focus area is used as the initial delivery position of the target delivery object; when the relative position relationship is an overlapping relationship, a first information-intensive area that overlaps with the predicted focus area is determined; based on the real-time eye movement data and the predicted focus area, a second adjacent preset unit area adjacent to the first information-intensive area is determined; first element edge pixel point distribution information corresponding to the first information-intensive area and second element edge pixel point distribution information of the second adjacent preset unit area are obtained; the delivery position corresponding to the target delivery object is determined based on the first element edge pixel point distribution information and the second element edge pixel point distribution information, and the initial delivery display parameters corresponding to the target delivery object are determined using the preset delivery display parameters.

[0038] In one embodiment of the present specification, when the two are in a non-overlapping relationship, it means that the predicted attention area does not interfere with the information-intensive area, and the predicted attention area is directly used as the initial placement position of the target placement object. When the two are in an overlapping relationship, the information-intensive area that overlaps with the predicted attention area is first found and defined as the first information-intensive area. Based on the real-time eye movement data and the predicted attention area, a second adjacent preset unit area adjacent to the first information-intensive area is determined.

[0039] Based on the real-time eye movement data and the predicted attention area, a second adjacent preset unit area adjacent to the first information intensive area is determined, specifically including: based on the real-time eye movement data and the predicted attention area, a directional predicted trajectory of the user is determined; and a plurality of adjacent preset unit areas of the first information intensive area are obtained to determine a second adjacent preset unit area located in the directional predicted trajectory among the plurality of adjacent preset unit areas.

[0040] In one embodiment of the present specification, when the relative position relationship is an overlapping relationship, it means that the predicted attention area partially overlaps or completely overlaps with the information-intensive area. At this time, the user's visual attention area should be approached while avoiding the information-intensive area. Therefore, the judgment can be assisted by the user's directional predicted trajectory. The user's directional predicted trajectory is determined based on real-time eye movement data and the predicted attention area. The real-time eye movement data records the real-time gaze position of the user's pupil. Combined with the predicted attention area, the direction of the predicted attention area pointed by the real-time gaze position can be inferred from the direction of the predicted attention area, thereby determining the directional predicted trajectory. A plurality of adjacent preset unit areas of the first information-intensive area are obtained. The adjacent areas are preset unit areas divided when the visual density detection is performed before. The plurality of adjacent preset unit areas of the first information-intensive area refer to the adjacent areas of the area overlapping with the predicted attention area. Then, from these adjacent preset unit areas, the area located in the directional predicted trajectory is found and defined as the second adjacent preset unit area. An area that is adjacent to the first information-intensive area and conforms to the direction of movement of the user's sight is found, so that the target delivery object can be placed later to make it more in line with the user's visual attention trend.

[0041] In one embodiment of the present specification, the placement position and initial placement display parameters are determined. The first element edge pixel distribution information corresponding to the first information-intensive area and the second element edge pixel distribution information of the second adjacent preset unit area are obtained respectively. The element edge pixel distribution information reflects the distribution of information elements in the area. By analyzing this information, the information density and layout of each area can be understood. For example, an area with concentrated element edge pixels may indicate that the information is relatively dense. By comparing the first element edge pixel distribution information and the second element edge pixel distribution information, the placement position of the target placement object is determined, and a position where the information is relatively sparse and in line with the user's visual focus direction is found to avoid the target placement object being too crowded with the original information, while facilitating user attention.

[0042] Finally, the initial delivery display parameters corresponding to the target delivery object are determined according to the preset delivery display parameters, including display attributes such as font size, color, and transparency, so as to complete the initial delivery parameter determination of the target delivery object under the overlapping relationship. It should be noted that the display attributes here can be the display attributes of the existing information elements in the current display information.

[0043] Through the above technical scheme, by analyzing the relative position relationship between the predicted attention area and the current information-intensive area, the initial delivery position of the target delivery object can be accurately determined, whether the target is directly delivered to the predicted attention area under a non-overlapping relationship, or the target is placed by considering the user's line of sight trajectory and the information distribution of the adjacent area under an overlapping relationship, which makes the information delivery more targeted; under an overlapping relationship, by considering the edge pixel distribution information of the information-intensive area, the delivery position of the target delivery object can be reasonably selected to avoid mutual interference with the existing dense information; the user's directional predicted trajectory is determined according to real-time eye movement data, so that the information delivery position conforms to the user's visual habits, and the target delivery object is placed on the path where the user's line of sight naturally flows, which allows the user to obtain information more smoothly; the information layout is more reasonable, and the user can find important information faster; the information delivery position and display parameters can be dynamically adjusted according to real-time eye movement data to achieve personalized information services. Different users have different visual habits and focus. Through this dynamic adaptation method, each user can be provided with an information display method that suits their own needs; by optimizing the information layout, the device's screen space can be used more effectively, avoiding disordered stacking of information, so that more valuable information can be displayed in the limited screen space; precise information delivery reduces unnecessary information loading and rendering, and only requires displaying the target delivery object at key locations and at the right time, rather than frequently updating and rendering all information within the entire screen range, thereby reducing the consumption of the device's computing and storage resources and improving overall performance.

[0044] Step S104, based on the information element display weight corresponding to each current information-intensive area, the display parameters of the initial delivery parameters are adjusted to determine the target delivery parameters, so as to display and deliver the target delivery object according to the target delivery parameters.

[0045] Based on the information element display weight corresponding to each current information-intensive area, the display parameter of the initial delivery parameter is adjusted to determine the target delivery parameter, specifically including: obtaining the element edge pixel point distribution information corresponding to the position parameter in the initial delivery parameter, and judging whether there is an initial display information element at the delivery position through the element edge pixel point distribution information; if there is an initial display information element, determining the information element display weight corresponding to the initial display information element; according to the information element display weight, adjusting the display parameter in the initial delivery parameter to determine the target delivery parameter, wherein the target delivery parameter includes the display parameter corresponding to the initial display information element and the display parameter corresponding to the target delivery object.

[0046] In one embodiment of the present specification, it is determined that the initial delivery parameters of the target delivery object include position parameters and display parameters, and based on the information element display weight corresponding to each current information-intensive area, the display parameters are further optimized to achieve a better information display effect. First, according to the position parameters in the initial delivery parameters, the preset unit area corresponding to the position is determined, and the element edge pixel distribution information of the preset unit area is obtained, and the distribution of the information elements currently existing at the position is understood by analyzing the distribution of the pixels. For example, in an image, the distribution of element edge pixels can reflect the outline and position of information elements such as text and graphics. By analyzing the distribution information of the element edge pixels, it is determined whether other information elements already exist at the target delivery position. If so, it means that the position already has an initial display information element; if not, there is no need to make complex display parameter adjustments.

[0047] After determining that there is an initial display information element in the delivery position, it is necessary to determine the information element display weight corresponding to the element. The information element display weight is obtained by comprehensively considering the element importance evaluation index, the element timeliness evaluation index and the task relevance evaluation index. According to the information element display weight of the determined initial display information element, the display parameters in the initial delivery parameters are adjusted. The goal of the adjustment is to highlight the target delivery object and ensure the coordination and rationality of the entire information display. For example, if the weight of the initial display information element is high, it means that the initial display information element cannot be omitted or hidden. Then, when adjusting the display parameters of the target delivery object, its brightness, font size or transparency may be increased, and the brightness, font size or transparency of the initial display information element may be reduced to ensure the simultaneous display of the initial display information element and the target delivery object; on the contrary, if the weight of the initial display information element is low, it means that the importance of the initial display information element is not high. At this time, in order to ensure the display effect of the target delivery object, the low-priority information can be automatically hidden or simplified to make the target delivery object more attractive to users. After adjustment, the final target delivery parameters are obtained. The target delivery parameters include two parts: one is the display parameters corresponding to the initial display information elements, and the other is the display parameters corresponding to the target delivery objects. These target delivery parameters can be used to display the target delivery objects, ensuring that the new target delivery objects are displayed in the best way based on the existing information, thereby improving the overall effect of information display.

[0048] Through the above technical scheme, by analyzing the distribution of existing information elements at the target delivery position, the display parameters of the target delivery object are adjusted according to their display weights, effectively avoiding the visual conflict between the new delivery information and the original information; the weights of various information elements are comprehensively considered to adjust the display parameters, making the entire information display layout more reasonable, and information with different weights can be presented in an appropriate manner, with important information highlighted and secondary information not overshadowing the main content, thereby improving the hierarchy and logic of information display; according to the weights of the initial display information elements, the display parameters of the target delivery object are flexibly adjusted, which helps to highlight the target delivery object; reasonable information display methods and prominent target delivery objects enable users to It is possible to obtain the required information more quickly and accurately without having to search through complex information, saving users' time and energy and improving information acquisition efficiency; by optimizing display parameters, it provides a more comfortable and convenient information browsing experience, meeting users' needs for clear and orderly information display, thereby enhancing user satisfaction with products or applications; by adjusting display parameters according to the weight of information elements, it achieves a reasonable allocation of display resources, and important information can obtain more display resources for prominent display. For the initial display information elements with lower weights, automatic hiding or simplification and other operations are adopted, which reduces the system's rendering and processing of these information and reduces the consumption of system resources.

[0049] In one embodiment of the present specification, a method for dynamic management of energy consumption is also provided. The eye movement frequency is calculated by collecting real-time eye movement data of the user. For example, the eye movement frequency is determined by counting the number of changes in pupil position in a short period of time (such as 1 second). The collected eye movement data is preprocessed to remove noise and outliers. An energy consumption statistics module is embedded in the system to record the energy consumption of the machine learning model (such as the pupil movement prediction model) in real time during the reasoning process. Using the performance counters provided by the processor, these counters can track information such as the number of instruction executions, cache hit rate, memory bandwidth usage, etc., and convert this information into energy consumption data through specific algorithms.

[0050] Through a large number of experiments and user tests, the relationship between eye movement frequency and user-perceived refresh rate is established. For example, when the eye movement frequency is low, the user may not be very sensitive to a lower refresh rate; when the eye movement frequency is high, a higher refresh rate is required to ensure the visual experience. Based on the above experimental results, the appropriate refresh rate threshold corresponding to different eye movement frequency intervals is determined. For example, when the eye movement frequency is lower than 10 times / second, the refresh rate can be reduced to 30Hz; when the eye movement frequency is between 10-20 times / second, the refresh rate is maintained at 60Hz; when the eye movement frequency is higher than 20 times / second, the refresh rate is increased to 90Hz.

[0051] Consider the changes in processor power consumption at different refresh rates, including the energy consumption of the display module, graphics processing unit, etc. Combined with the power consumption characteristics of the processor, a mathematical model between refresh rate and energy consumption is established. For example, for every 10Hz increase in refresh rate, energy consumption may increase by a certain percentage (such as 10%). According to the energy consumption-refresh rate model, a trade-off strategy is formulated. Try to reduce energy consumption while ensuring user experience. For example, if the energy saved by reducing the refresh rate is greater than the loss caused by the decrease in user experience due to the reduction in refresh rate, then choose to reduce the refresh rate. During the operation of the system, the eye movement frequency and model reasoning energy consumption are monitored in real time, and according to the pre-established decision rules, it is determined whether the display refresh rate needs to be adjusted. The refresh rate adjustment can be implemented by sending a refresh rate adjustment instruction to the display module through the hardware control interface, through the interface provided by the operating system or hardware driver. In order to avoid visual discomfort to users caused by sudden changes in refresh rate, a smooth transition mechanism is adopted. For example, when the refresh rate needs to be reduced, the refresh rate can be slowly reduced to the target value within a few seconds by gradually reducing the pixel clock frequency. By dynamically adjusting the display refresh rate of the processor, the power consumption of the display module can be reasonably reduced according to the actual eye movement frequency and model inference energy consumption. When the eye movement frequency is low or the model inference energy consumption is high, reducing the refresh rate can effectively reduce unnecessary energy consumption; in addition to the display module, this dynamic management strategy also takes into account the energy consumption of model inference, balancing the power consumption of each component at the system level, and avoiding a high-energy consumption link (such as machine learning model inference) from having too much impact on the overall energy consumption of the device; while reducing power consumption, this strategy can maintain the best user experience. The dynamic management strategy can adjust the refresh rate according to the actual eye movement of each user, so as to better meet the needs of different users.

[0052] Through the technical solution of the embodiments of this specification, the traditional display method cannot perceive the dynamic changes of the user's visual focus. The embodiments of this specification collect the user's real-time eye movement data and use the pre-built pupil movement prediction model to track and accurately predict the user's visual position in real time and determine the predicted focus area; the location that the user may focus on can be known in advance, and based on this, the target delivery object can be accurately delivered to the vicinity of the area or a reasonable location, so that important information is highly matched with the user's focus, reducing the user's time and energy in finding information and improving information acquisition efficiency; traditional display methods are prone to information accumulation, resulting in poor display effects and users finding it difficult to distinguish key information; the embodiments of this specification conduct an in-depth analysis of the information displayed on the current device, determine at least one current information-intensive area and the display weights of the information elements in each area, and in this way, it is possible to clearly sort out complex information and clarify the importance of each information element; based on the information element display weights, the target delivery object is The display parameters in the initial delivery parameters of the object are adjusted, which not only ensures the prominence of key information, but also makes rational use of page space, effectively solves the problem of information accumulation, improves the information display effect, enables users to easily obtain key information, and greatly improves the user experience of smart devices; by determining information-intensive areas and accurate information arrangement, smart devices can focus display resources on areas and important information elements that users are actually concerned about; due to blind information delivery and information accumulation in traditional display methods, users need to spend a lot of energy to screen information, which can easily cause visual fatigue and high cognitive burden. The embodiments of this manual use accurate information delivery and reasonable information display optimization, so that users no longer need to search in a large amount of useless information, reducing eye fatigue and brain processing pressure; traditional display methods cannot meet personalized needs. The embodiments of this manual optimize information according to each user's unique real-time eye movement data, thereby achieving highly personalized information delivery.

[0053] The embodiment of this specification also provides an information display optimization device for a smart device, such as Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0054] The embodiments of the present specification also provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0055] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0056] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0057] The devices and media provided in the embodiments of this specification correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0058] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0059] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0060] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0063] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0064] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0066] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A method for optimizing information display for smart devices, characterized in that: The method comprises: Collecting the user's real-time eye movement data and current device display information, using a pre-built pupil movement prediction model and the real-time eye movement data to predict the user's visual position, and determine the user's predicted focus area; Analyze the current device display information to determine at least one corresponding current information-intensive area and an information element display weight corresponding to each current information-intensive area; Arranging information by the current information-intensive area and the predicted focus area, and determining initial delivery parameters corresponding to the target delivery object, wherein the initial delivery parameters include position parameters and display parameters; Based on the information element display weight corresponding to each current information-intensive area, the display parameters of the initial delivery parameters are adjusted to determine the target delivery parameters, so as to display and deliver the target delivery object through the target delivery parameters.

2. The information display optimization method for smart devices according to claim 1, characterized in that: Analyzing the current device display information to determine at least one corresponding current information-intensive area and an information element display weight corresponding to each current information-intensive area specifically includes: Performing visual density detection on the current device display information to determine the information display density of each preset unit area, so as to determine at least one current information-intensive area in the preset unit area based on the information display density, wherein the information display density of the current information-intensive area is higher than a preset density threshold; Acquire element evaluation data of each information element in the current information-intensive area, wherein the element evaluation data includes an element importance evaluation index, an element timeliness evaluation index, and a task relevance evaluation index; The information element display weight corresponding to each of the current information-intensive areas is determined according to the element evaluation data.

3. The information display optimization method for smart devices according to claim 2, characterized in that: Performing visual density detection on the current device display information to determine the information display density of each preset unit area specifically includes: Obtaining pixel occupancy parameters of the pixel area occupied by the target delivery object; Divide the current device display information into regions according to the pixel occupation parameters of the occupied pixel region to determine a plurality of preset unit regions; Identify the element edge of the information element in each of the preset unit areas, and determine the number of element edge pixel points in each of the preset unit areas; The number of statistical pixels in each of the preset unit areas is counted, and the information display density corresponding to each of the preset unit areas is determined by the ratio of the number of element edge pixels in each of the preset unit areas to the corresponding number of statistical pixels.

4. The information display optimization method for smart devices according to claim 1, characterized in that: Arranging information in the current information-intensive area and the predicted focus area to determine initial delivery parameters corresponding to the target delivery object specifically includes: Acquire the focus area position of the predicted focus area and the dense area position of the current information dense area; Determine the relative position relationship between the predicted region of interest and the current information dense area according to the position of the region of interest and the positions of the plurality of dense areas, wherein the relative position relationship includes an overlapping relationship and a non-overlapping relationship; According to the relative position relationship between the predicted focus area and the current information intensive area, the predicted focus area is adjusted to determine the initial delivery parameters corresponding to the target delivery object.

5. The information display optimization method for smart devices according to claim 4, characterized in that: According to the relative position relationship between the predicted focus area and the current information-intensive area, the predicted focus area is adjusted to determine the initial delivery parameters corresponding to the target delivery object, specifically including: When the relative position relationship is a non-overlapping relationship, the predicted focus area is used as the initial placement position of the target placement object; When the relative position relationship is an overlapping relationship, determining a first information-intensive area that overlaps with the predicted focus area; Determine, according to the real-time eye movement data and the predicted focus area, a second adjacent preset unit area adjacent to the first information-intensive area; Obtaining first element edge pixel point distribution information corresponding to the first information-intensive area and second element edge pixel point distribution information of the second adjacent preset unit area; The delivery position corresponding to the target delivery object is determined by using the first element edge pixel point distribution information and the second element edge pixel point distribution information, and the initial delivery display parameters corresponding to the target delivery object are determined by using preset delivery display parameters.

6. The information display optimization method for smart devices according to claim 5, characterized in that: Determining a second adjacent preset unit area adjacent to the first information-intensive area according to the real-time eye movement data and the predicted focus area specifically includes: Determining a user's predicted directional trajectory based on the real-time eye movement data and the predicted focus area; A plurality of adjacent preset unit regions of the first information-intensive region are acquired to determine a second adjacent preset unit region located in the directional prediction trajectory from among the plurality of adjacent preset unit regions.

7. The information display optimization method for smart devices according to claim 1, characterized in that: Based on the information element display weight corresponding to each current information-intensive area, the display parameters of the initial delivery parameters are adjusted to determine the target delivery parameters, specifically including: Obtaining element edge pixel point distribution information corresponding to the position parameter in the initial delivery parameter, and judging whether there is an initial display information element at the delivery position according to the element edge pixel point distribution information; If there is an initial display information element, determining an information element display weight corresponding to the initial display information element; According to the information element display weight, the display parameters in the initial delivery parameters are adjusted to determine target delivery parameters, wherein the target delivery parameters include display parameters corresponding to the initial display information element and display parameters corresponding to the target delivery object.

8. The information display optimization method for smart devices according to claim 1, characterized in that: Using the pre-built pupil movement prediction model and the real-time eye movement data, the user's visual position is predicted to determine the user's predicted focus area, specifically including: Collecting historical pupil movement data of the user, wherein the historical pupil movement data includes a plurality of pupil positions within a preset historical time period; Determine a pupil movement prediction model through a preset recurrent neural network model and historical pupil movement data to determine the predicted pupil position corresponding to the user using real-time eye movement data; Obtain pixel occupancy parameters of the pixel area occupied by the target delivery object, and determine the predicted focus area based on the pixel occupancy parameters with the predicted pupil position as the center.

9. An information display optimization device for a smart device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images