Screen refreshing method, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a screen refreshing method, electronic equipment, a storage medium and a computer program product. The screen refreshing method comprises the steps that the position of an action point formed on a screen when a target object pays attention to interested content on the screen is acquired; wherein when the display positions of the interested contents are different, the positions of the action points are different; determining a local refreshing area of the screen based on the position of the action point; and updating the interested content in the local refresh area. Through the screen refreshing method provided by the invention, the local refreshing area can be determined based on the attention position of the target object to the screen, and the content corresponding to the local refreshing area is directionally refreshed, so that the power consumption of full-screen refreshing can be effectively reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of screen display and image processing, and in particular to a screen refresh method, electronic equipment, storage medium, and computer program product. Background Art

[0002] With the rapid development of science and technology, electronic devices are constantly innovating and gradually becoming an indispensable tool in people's lives. In order to meet users' various needs such as entertainment, office, watching videos or browsing the web, the screen area of electronic devices is also constantly expanding.

[0003] Currently, the screen refresh method of electronic devices requires rendering the display image of the entire screen. For the display area that does not need to be updated for each frame, full-screen refresh increases the power consumption of the electronic device. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a screen refresh method, an electronic device, a storage medium, and a computer program product. Through the screen refresh method proposed in the present disclosure, a local refresh area can be determined based on the focus position of the target object on the screen, and the content corresponding to the local refresh area can be refreshed in a targeted manner, which can effectively reduce the power consumption of full-screen refresh.

[0005] According to a first aspect of an embodiment of the present disclosure, a screen refreshing method is provided, the method comprising:

[0006] Acquire the position of an action point formed on the screen when the target object pays attention to the content of interest on the screen; wherein the position of the action point is different when the display position of the content of interest is different;

[0007] Based on the position of the action point, determining a local refresh area of the screen;

[0008] The content of interest is updated in the partial refresh area.

[0009] In some embodiments, determining the local refresh area of the screen based on the position of the action point includes:

[0010] Based on the positions of the plurality of action points collected within a preset time period, determining an area of interest where the target object is paying attention to the content of interest;

[0011] The local refresh area is defined on the screen according to the region of interest.

[0012] In some embodiments, the position of the action point includes the coordinates of the screen touch point formed on the screen when the target object touches the screen and / or the coordinates of the screen gaze point formed on the screen when the target object gazes at the screen; determining the region of interest where the target object focuses on the content of interest based on the positions of multiple action points collected within a preset time period includes:

[0013] Based on the coordinates of multiple screen touch points collected within the preset time period, determining the region of interest corresponding to touching the screen; and / or, based on the coordinates of multiple screen gaze points collected within the preset time period, determining the region of interest corresponding to gazing at the screen.

[0014] In some embodiments, delineating the partial refresh region on the screen according to the region of interest includes:

[0015] Taking the smallest rectangular region including the region of interest corresponding to touching the screen as the partial refresh region; or,

[0016] Taking the smallest rectangular region including the region of interest corresponding to gazing at the screen as the partial refresh region; or,

[0017] Determining the partial refresh region based on the region of interest corresponding to touching the screen and the region of interest corresponding to gazing at the screen.

[0018] In some embodiments, determining the partial refresh region based on the region of interest corresponding to touching the screen and the region of interest corresponding to gazing at the screen includes:

[0019] Performing weight adjustment on the region of interest corresponding to touching the screen to obtain a first adjusted region;

[0020] Performing weight adjustment on the region of interest corresponding to gazing at the screen to obtain a second adjusted region;

[0021] Taking the smallest rectangular region including the first adjusted region and the second adjusted region as the partial refresh region.

[0022] In some embodiments, determining the region of interest where the target object focuses on the content of interest based on the positions of multiple action points collected within a preset time period includes:

[0023] Determining multiple target points among the multiple action points collected within the preset time period;

[0024] Taking the position of the target point as the center of a circle and the length corresponding to the continuous occurrence duration of collecting the target point at the same position as the diameter, calibrate a plurality of initial circular regions on the screen;

[0025] Based on the plurality of initial circular regions, determine the region of interest.

[0026] In some embodiments, among the plurality of action points collected within the preset time period, determining a plurality of target points includes:

[0027] Construct an initial matrix based on the plurality of action points collected within the preset time period, where the number of rows of the initial matrix is the number of the plurality of action points, and the number of columns of the initial matrix is the number of coordinate groups of the action points;

[0028] Perform dimensionality reduction processing on the initial matrix to obtain a target matrix, where the number of rows of the target matrix is less than the number of rows of the initial matrix;

[0029] Take the plurality of action points forming the target matrix as the plurality of target points.

[0030] In some embodiments, the determining the region of interest based on the plurality of initial circular regions includes:

[0031] Take the smallest circular region including the plurality of initial circular regions as the region of interest.

[0032] In some embodiments, the taking the smallest circular region including the plurality of initial circular regions as the region of interest includes:

[0033] When the data N of the initial circular region is equal to 2, obtain the (N - 1)th minimum circumscribed circle region based on the (N - 1)th initial circular region and the Nth initial circular region;

[0034] When N is greater than 2, obtain the (N - 1)th minimum circumscribed circle region based on the Nth initial circular region and the (N - 2)th minimum circumscribed circle region;

[0035] Take the (N - 1)th minimum circumscribed circle region as the region of interest;

[0036] Wherein, N is a positive integer greater than 1.

[0037] In some embodiments, the position of the action point includes the coordinates of the screen fixation point formed on the screen when the target object gazes at the screen, and the obtaining the position of the action point formed on the screen when the target object focuses on the content of interest on the screen includes:

[0038] Based on the imaging component of the electronic device, collect the eyes of the target object, and obtain eye movement data of the pupil of the eye relative to the cornea of the eye changing in position;

[0039] Based on the eye movement data and the preset mapping relationship between the eye movement data and the coordinates of the screen fixation point, obtain the coordinates of the screen fixation point.

[0040] In some embodiments, updating the content of interest in the local refresh area includes:

[0041] According to the diagonal coordinates of the local refresh area on the screen, determine the pixel rows and pixel columns for image refresh;

[0042] Based on the pixel rows and the pixel columns, update the content of interest.

[0043] According to a second aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0044] An acquisition module, configured to acquire the position of the action point formed on the screen when the target object focuses on the content of interest on the screen; wherein, when the display positions of the content of interest are different, the positions of the action points are different;

[0045] A determination module, configured to determine the local refresh area of the screen based on the position of the action point;

[0046] An update module, configured to update the content of interest in the local refresh area.

[0047] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, at least including:

[0048] A processor;

[0049] A memory for storing computer programs or instructions;

[0050] Wherein, the processor is configured to execute the computer program or instruction to implement the screen refresh method as described in the first aspect above.

[0051] According to a fourth aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, and the storage medium stores computer programs or instructions, and when the computer programs or instructions in the storage medium are executed by a processor, the screen refresh method as described in the first aspect above is implemented.

[0052] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, including computer programs or instructions, and when the computer programs or instructions are executed by a processor, the screen refresh method as described in the first aspect above is implemented.

[0053] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

[0054] In the screen refreshing method proposed by the embodiments of the present disclosure, a local refreshing area is determined based on the attention position of the target object on the screen, and the content of interest corresponding to the local refreshing area is directionally refreshed; in this way, on the one hand, it is possible to selectively and quickly update the content of interest with a high degree of attention of the target object on the screen, improving the user experience; on the other hand, by locally refreshing the screen, the power consumption of full-screen refreshing and the system I / O read-write frequency can be effectively reduced; and, compared with the system continuously caching and reading the previous frame image to calculate the image difference for local screen refreshing, the present disclosure reduces the damage caused to the electronic device by I / O busy and the long-term high-load operation of the core processor; further improving the smoothness of the mobile phone operation and greatly reducing the screen power consumption.

[0055] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure.

[0057] Figure 1 is a flowchart showing a screen refreshing method according to an exemplary embodiment;

[0058] Figure 2 is a schematic diagram showing the refreshing of a local refreshing area according to an exemplary embodiment;

[0059] Figure 3 is a flowchart showing a method for local screen refreshing by combining gaze at the screen and touch on the screen according to an exemplary embodiment;

[0060] Figure 4 is a flowchart showing the dimensionality reduction of features according to an exemplary embodiment;

[0061] Figure 5 is a schematic diagram showing a gaze point trajectory area map according to an exemplary embodiment;

[0062] Figure 6 is a schematic diagram showing a touch trajectory area map according to an exemplary embodiment;

[0063] Figure 7a is a schematic diagram showing the principle of determining a local refreshing area according to an exemplary embodiment Figure 1 ;

[0064] Figure 7b Schematic diagram of the principle for determining a local refresh area shown according to an exemplary embodiment Figure 2 ;

[0065] Figure 7c Schematic diagram of the principle for determining a local refresh area shown according to an exemplary embodiment Figure 3 ;

[0066] Figure 8 Schematic diagram of the principle for jointly determining a local refresh area by combining a gaze screen and a touch screen shown according to an exemplary embodiment;

[0067] Figure 9 Schematic diagram of the structure of an electronic device shown according to an exemplary embodiment;

[0068] Figure 10 Block diagram of the structure of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0069] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0070] Refer to Figure 1 , Figure 1 Flow schematic diagram of a screen refresh method shown according to an exemplary embodiment; As Figure 1 shown, a screen refresh method proposed by the present disclosure can be implemented through the following steps:

[0071] Step S101, obtain the position of the action point formed on the screen when the target object focuses on the interesting content on the screen; wherein, when the display position of the interesting content is different, the position of the action point is different;

[0072] Step S102, determine the local refresh area of the screen based on the position of the action point;

[0073] Step S103, update the interesting content in the local refresh area.

[0074] Here, the screen refreshing method proposed by the present disclosure is applied to the scenario where a target object uses an electronic device. The way the target object uses the electronic device can be holding the electronic device, wearing the electronic device, or gazing at the electronic device within a certain distance, etc. When the target object focuses on the screen of the electronic device, the content of interest may be only a small area of the screen. The screen refreshing method proposed by the present disclosure can determine a local refreshing area based on the focus position of the target object on the screen, and perform directional refreshing on the content corresponding to the local refreshing area, thereby effectively reducing the power consumption of full-screen refreshing and the I / O read-write frequency, and further increasing the smoothness of the operation of the electronic device.

[0075] The electronic device to which the screen refreshing method proposed by the present disclosure is applied can be a fixed terminal, a mobile terminal, or a portable electronic device. The mobile terminal includes, but is not limited to, mobile phones and tablet computers. The fixed terminal includes, but is not limited to, in-vehicle terminals and touch-screen TVs. The portable electronic device includes, but is not limited to, smart watches and wearable glasses. The embodiments of the present disclosure do not make further limitations in this regard.

[0076] In step S101, the electronic device obtains the attention result generated by the target object's attention to the content of interest on the screen. Here, collecting the attention result of the target object's attention to the content of interest on the screen can be obtained by tracking the touch action point of the target object's hand touching the screen of the electronic device; or by tracking the eye movement action point of the target object's eyes gazing at the screen of the electronic device; or by collecting the gesture / pose action point generated by the target object's hand / body near the area of interest on the screen.

[0077] It should be noted that the content of interest can be content scattered on the screen or content concentrated in a specified area of the screen. The present disclosure does not limit this. When the position of the content of interest displayed on the screen is different, the focus position of the target object on the screen is different. Correspondingly, the position of the action point is also different.

[0078] It should also be noted that when the user simultaneously collects the positions of the action points formed by two or more attention methods on the screen, the content of interest concerned by different attention methods may be the same, and the corresponding positions of the action points may also coincide.

[0079] In some examples, taking the case where the position of the action point for collecting the target object's attention to the content of interest on the screen is obtained by collecting gestures / body postures as an example, the front camera of the electronic device can collect the position of the screen pointed by the target object's finger and the gesture change state to determine the position of the gesture action point formed on the screen; or the front camera of the electronic device can collect the moving distance and moving direction of the target object's body towards the screen to determine the position of the posture action point formed on the screen.

[0080] In some other examples, taking the case where the position of the point of action for the target object to focus on the content of interest on the screen is obtained by collecting finger touches on the screen as an example, when the under-screen capacitive sensor of the electronic device senses and reports a touch event, the touch position of the target object's finger on the screen is determined.

[0081] In this way, by using acquisition components such as the camera or under-screen capacitance of the electronic device, the position of the point of action for the target object to focus on the content of interest on the screen can be located, which is convenient for determining the focus result of the target object on the screen.

[0082] In step S102, the position of the point of action collected within a preset time period can be analyzed and processed to obtain a locally refreshed area delimited on the screen. Here, the locally refreshed area at least partially includes the content of interest that the target object is concerned about. In some examples, the present disclosure can draw the action trajectory of the point of action based on the position of the point of action, and delimit the locally refreshed area based on the action trajectory.

[0083] In step S103, after delimiting the locally refreshed area, the content of interest within the locally refreshed area is refreshed.

[0084] In some embodiments, the updating of the content of interest in the locally refreshed area in step S103 can be implemented in the following manner:

[0085] Determine the pixel rows and pixel columns for image refreshing according to the diagonal coordinates of the locally refreshed area on the screen;

[0086] Update the content of interest based on the pixel rows and pixel columns.

[0087] Here, the screen display of the electronic device mainly includes the screen end and the processor end (AP end). After determining the locally refreshed area according to the data collected by acquisition components such as the camera or under-screen capacitance at the AP end, the diagonal coordinates of the locally refreshed area are determined on the screen. In the present disclosure, the locally refreshed area can be a regular graphic area or an irregular graphic area, and the diagonal coordinates proposed in the present disclosure are the coordinates of two points with the farthest relative distance within the locally refreshed area.

[0088] It should be noted that when determining the diagonal coordinates, it can be determined by using the world coordinate system, or by creating a plane rectangular coordinate system on the screen with a specified position on the screen as the origin. The present disclosure does not limit this. Here, the specified position on the screen can be the lower left corner of the screen, or the center position of the screen.

[0089] In some examples, taking the establishment of a plane rectangular coordinate system with the lower left corner of the screen as the origin as an example, based on the locally refreshed area delimited on the screen, the corresponding diagonal coordinates determined are: the coordinates of the upper left vertex of the locally refreshed area O1 = {a1, b1} and the coordinates of the lower right vertex O2 = {a2, b2}.

[0090] Here, according to the coordinates O1 = {a1, b1} of the upper left vertex and the coordinates O2 = {a2, b2} of the lower right vertex, the pixel rows and pixel columns for local rendering at the AP end are determined. The AP end generates a picture sending instruction of 2A (defining the column of the picture sent) / 2B (defining the termination row of the picture sent) based on the pixel rows and pixel columns, and sends it to the screen end.

[0091] In the present disclosure, the display driving chip at the screen end receives the picture sending instruction sent by the AP end, determines which area of the screen the image data of the AP is updated in, and parses the instruction to obtain pixel data, thereby refreshing the specified local refresh area to update the content of interest.

[0092] During the actual implementation process, refer to Figure 2 , Figure 2 is a schematic diagram showing the refreshing of the local refresh area according to an exemplary embodiment; as Figure 2 shown, after determining the local refresh area according to the target object's attention to the screen, the AP end will send pictures to the screen end at the first refresh frequency to update the content of interest in the local refresh area A, and the AP end will also send pictures to the screen end at the second refresh frequency to update the picture content in the non-updated area B except the local refresh area A. Here, the first refresh frequency is greater than the second refresh frequency. Exemplarily, the first refresh frequency is 120 Hz and the second refresh frequency is 30 Hz.

[0093] Here, the present disclosure can break free from the shackles of the screen medium on screen refreshing, directly control the AP end instead of the screen end, and has good universality for local screen refreshing of current LCD displays or OLED displays.

[0094] In the screen refreshing method proposed by the embodiments of the present disclosure, the local refresh area is determined based on the attention position of the target object to the screen, and the content of interest corresponding to the local refresh area is directionally refreshed; thus, on the one hand, it is possible to selectively and quickly update the content of interest with a high degree of attention from the target object on the screen, improving the user experience; on the other hand, by performing local screen refreshing, the power consumption of full-screen refreshing and the system I / O read-write frequency can be effectively reduced; and compared with the system constantly caching and reading the previous frame of image to calculate the image difference for local screen refreshing, the present disclosure reduces the damage to the electronic device caused by I / O busyness and the long-term high-load operation of the core processor; further improving the smoothness of the mobile phone operation and greatly reducing the screen power consumption.

[0095] In some embodiments, based on the position of the action point, determining the local refresh area of the screen in step S102 can be implemented through the following steps:

[0096] Based on the positions of multiple action points collected within a preset time period, determining an area of interest where the target object is paying attention to the content of interest;

[0097] Based on the area of interest, a local refresh area is delineated on the screen.

[0098] Here, collecting multiple action points in a preset time period can be to continuously track the target object's attention to the area of interest on the screen at a specified sampling frequency within the preset time period. In the present disclosure, the preset time period and the sampling frequency can be pre-set by the electronic device, or dynamically adjusted according to the actual scene in which the electronic device is located. For example, if the actual scene in which the electronic device is located is a video playback scene, the preset time period can be appropriately reduced or the sampling frequency can be appropriately increased; if the actual scene in which the electronic device is located is a reading scene, the preset time period can be appropriately increased or the sampling frequency can be appropriately reduced; illustratively, the preset time can be 1 second, and the sampling frequency can be 240Hz, that is, 240 samples per second.

[0099] In some examples, based on the positions of multiple action points collected within a preset time period, the action trajectory can be determined according to the positions of the action points, and the region of interest where the content of interest is located can be determined using the action trajectory as the edge contour. In some examples, the action trajectory can be determined according to the positions of the action points by forming a displacement vector based on the displacement and movement direction of the action points collected at adjacent time points, and arranging the displacement vectors formed by all action points within the preset time period in the order of acquisition time to obtain the action trajectory; in other examples, the action trajectory can also be determined according to the positions of the action points by analyzing all action points using a linear regression data model, eliminating some action points, obtaining action points with high linear correlation, and connecting the remaining action points with arcs to form an action trajectory.

[0100] In other examples, after collecting the positions of multiple action points within a preset time period, an area containing all the action points (or the remaining action points after some action points are eliminated by a model such as a linear regression data model) can be determined based on the positions of all the action points, and this area can be used as the area of interest.

[0101] After determining the region of interest that the target object is concerned about according to the method proposed in the present disclosure, a partial refresh region of the screen is determined based on the region of interest. Here, determining the partial refresh region based on the region of interest may be determining the region of interest as the screen refresh region, or may be expanding or reducing the range of the region of interest according to a certain ratio, or may be adjusting the screen refresh region according to the content of interest in the region of interest.

[0102] Exemplarily, after determining the region of interest, the content of interest can be analyzed. If the content of interest includes half of a person's body, the region corresponding to the other half of the person's body can be combined with the region of interest, and the combined region can be determined as the local refresh region.

[0103] In the embodiments of the present disclosure, the positions of multiple action points are collected within a preset time period, and the region of interest of the target object is determined, so as to delimit the local refresh region, which can fully focus on the attention direction of the target object to the screen within the preset time period, thereby being able to provide different screen refresh results for different users and improving the effectiveness of screen refresh and the user experience.

[0104] In some embodiments, the positions of the above-mentioned action points include the coordinates of the screen touch points formed on the screen when the target object touches the screen and / or the coordinates of the screen gaze points formed on the screen when the target object gazes at the screen; based on the positions of multiple action points collected within a preset time period, determining the region of interest where the target object is interested in the content can be implemented through the following steps:

[0105] Based on the coordinates of multiple screen touch points collected within a preset time period, determine the region of interest corresponding to touching the screen; and / or, based on the coordinates of multiple screen gaze points collected within a preset time period, determine the region of interest corresponding to gazing at the screen.

[0106] Here, the sampling frequency of collecting the screen touch points formed on the screen when the target object touches the screen and the sampling frequency of collecting the screen gaze points formed on the screen when the target object gazes at the screen within a preset time period can be the same or different, and the present disclosure does not further limit this.

[0107] In the present disclosure, obtaining the positions of the action points formed on the screen when the target object focuses on the content of interest on the screen in step S101 can be implemented in the following manner: when a touch event is reported by an under-screen sensor of the screen, such as a capacitive sensor, according to the touch points of the target object's finger touching the screen collected at a preset sampling frequency, such as 240Hz, within a preset time period, determine the coordinates of the screen touch points formed on the screen.

[0108] In actual implementation, the under-screen capacitive sensor also calculates the sliding acceleration of the finger according to the touch event. When the sliding acceleration meets the preset conditions, it is determined that this touch is effective; if it is determined that the touch is ineffective within a preset number of times, such as 5 times, it is determined that the target object does not focus on the content of interest by touching the screen. At this time, the region of interest can be determined according to other actions of the target object, such as eye movement or body posture change. Here, the preset condition can be whether the value of the sliding acceleration exceeds 3m / s 2 .

[0109] In some embodiments, the position of the above action point includes the coordinates of the screen fixation point formed on the screen when the target object gazes at the screen. The step of obtaining the position of the action point formed on the screen when the target object focuses on the interesting content on the screen in step S101 can be implemented through the following steps:

[0110] Based on the imaging component of the electronic device, collect the eyes of the target object, and obtain the eye movement data of the position change of the pupil of the eye relative to the cornea of the eye;

[0111] Based on the eye movement data and the preset mapping relationship between the eye movement data and the coordinates of the screen fixation point, obtain the coordinates of the screen fixation point.

[0112] Here, the imaging component includes a front camera; the present disclosure, based on the cooperation of the front camera and the control module of the electronic device, combines the pupil-corneal reflection tracking method to determine the coordinates of the screen fixation point. Specifically, the front camera captures an eye image, and calibrates the central position of the pupil of the eye through an image processing method; the corneal reflection point (yellow spot) of the eye is used as the base point of the relative position between the camera and the eye. At this time, according to the calibrated pupil center, the eye movement data such as the line-of-sight vector of the position change of the pupil of the eye relative to the cornea of the eye can be obtained; at this time, according to the mapping relationship between the eye movement data and the coordinates of the screen fixation point (such as the mapping function between the vector formed between the pupil and the corneal reflection point and the screen fixation point), the coordinates of the screen fixation point are obtained.

[0113] The embodiment of the present disclosure can effectively determine the focusing position of the target object viewing the screen by collecting the eye movement data of the target object gazing at the screen, which helps to reasonably delimit the local refresh range. Moreover, through the method proposed by the present disclosure, the use scenario of the imaging component can also be effectively broadened, and the function of the camera can be maximally exerted, so that the camera is not only limited to a photographing tool, but can also be used to capture eye movement data.

[0114] It should be noted that the determination of the coordinates of the screen touch point and the determination of the coordinates of the screen fixation point in the present disclosure are discussed in the same coordinate system. Here, the same coordinate system can be a world coordinate system, or a plane rectangular coordinate system created on the screen with a specified position on the screen as the origin; exemplarily, the specified position on the screen can be the lower left corner of the screen.

[0115] After the present disclosure determines the coordinates of the screen touch point and / or determines the coordinates of the screen fixation point, the interesting region can be determined separately or jointly.

[0116] In this way, the present disclosure can determine the interesting content of the target object on the screen according to the action changes of the target object touching the screen and / or gazing at the screen, which helps to locate the local refresh region and makes the content update effect of the local refresh region meet the user's usage requirements.

[0117] In some embodiments, the local refresh area can be delimited on the screen according to the region of interest as described above, which can be implemented through the following steps:

[0118] Take the smallest rectangular area including the region of interest corresponding to the touch screen as the local refresh area; or,

[0119] Take the smallest rectangular area including the region of interest corresponding to the gaze screen as the local refresh area; or,

[0120] Determine the local refresh area based on the region of interest corresponding to the touch screen and the region of interest corresponding to the gaze screen.

[0121] Here, in the embodiments of the present disclosure, the region of interest can be a regular graphic area or an irregular graphic area. For example, the region of interest can be a circular area, and the method proposed by the present disclosure for determining the local refresh area can be delimited according to the smallest rectangular area including the region of interest.

[0122] In the case where the screen refresh method proposed by the present disclosure determines the local refresh range based on the target object's gaze at the screen, take the smallest rectangular area including the region of interest corresponding to the gaze screen as the local refresh area; in the case where the screen refresh method proposed by the present disclosure determines the local refresh range based on the target object's touch on the screen, take the smallest rectangular area including the region of interest corresponding to the touch screen as the local refresh area; in the case where the screen refresh method proposed by the present disclosure determines the local refresh area by simultaneously referring to the target object's gaze and touch on the screen, delimit the local refresh range by combining the two regions of interest. Exemplarily, the way to determine the local refresh area by combining the region of interest corresponding to the touch screen and the region of interest corresponding to the gaze screen can be to enclose the local refresh area with the smallest rectangular area that simultaneously includes the region of interest corresponding to the touch screen and the region of interest corresponding to the gaze screen.

[0123] In some other embodiments, regions of other shapes that enclose the region of interest, such as the smallest circular area, can also be determined as the local refresh area.

[0124] The embodiments of the present disclosure can determine the local refresh range of the screen by combining the region of interest corresponding to the target object's touch on the screen and / or the region of interest corresponding to the gaze screen in different situations, so that the content update effect of the local refresh area meets the usage requirements of the target object and improves the user experience; moreover, taking the smallest rectangle including the region of interest as the local refresh area can minimize the refresh range as much as possible, thereby reducing the power consumption of the screen refresh.

[0125] In some embodiments, the determination of the local refresh area based on the area of interest corresponding to the touch screen and the area of interest corresponding to the gaze screen can be implemented by the following steps:

[0126] Performing weight adjustment on the region of interest corresponding to the touch screen to obtain a first adjustment region;

[0127] Adjusting the weight of the region of interest corresponding to the gazed screen to obtain a second adjusted region;

[0128] The smallest rectangular area including the first adjustment area and the second adjustment area is used as the local refresh area.

[0129] In the embodiment of the present disclosure, the method for determining the weight of the interest area corresponding to the touch screen and the weight of the interest area corresponding to the gaze screen can be determined by the attention mechanism model. Here, the parameter S of the interest area corresponding to the touch screen is touch The parameter S of the region of interest corresponding to the gaze screen eye Input into the attention mechanism model; here, corresponding to S eye and S touch It can be coordinate data / size data representing the range of the area of interest; the attention mechanism model can output the weight ω of the area of interest corresponding to the gaze screen eye And the weight ω of the area of interest corresponding to the touch screen touch It should be noted that the process of determining the weight also needs to be processed using the softmax activation function so that the final weight ω eye and ω touch The sum is 1; here, determine the weight ω eye and ω touch It can be calculated by the following formula (1):

[0130] (ω eye ,ω touch )=soft max(f atte (S eye ,S touch )+ε) (1)

[0131] Here, f atte The function is used to perform weight analysis on the touch screen and the gaze screen to obtain ω eye and ω touch ; ε is a model correction parameter, with a value between 0 and 1. In order to ensure that the model does not have non-zero output or numerical overflow, ε is usually very small; for example, ε is 1e-8 or 1e-6. The value of the model correction parameter can be preset or dynamically adjusted according to actual conditions.

[0132] Here, in determining ω eye and ω touch After that, the regions of interest corresponding to the touch screen are adjusted based on the weights to obtain the second adjustment region ω eye S eye and the first adjustment region ω touch S touch ; The present disclosure is based on the second adjustment area ω eye S eye and the first adjustment region ω touch S touch The method for determining the local refresh area can be implemented according to the following formula:

[0133] S optimal =f square (ω eye S eye ∪ω touch S touch ) (2)

[0134] Here, f square is a method function for determining the minimum rectangular area containing the first adjustment area and the second adjustment area, S optimal Characterize the local refresh area after fusion.

[0135] In the present disclosure, taking the region of interest as a circular region as an example, if only the region of interest corresponding to the gaze screen or the region of interest corresponding to the touch screen is determined within the preset time period, according to the Attention mechanism model, the weight of the corresponding region of interest within the preset time period is set to the highest (for example, 1), the weight of the other region of interest is set to 0, and a minimum rectangular area that can cover the circular area is determined as a local refresh area; if the region of interest corresponding to the gaze screen or the region of interest corresponding to the touch screen both exist and have no intersection, then the weights of the two regions of interest are adjusted according to the weight output by the Attention mechanism. In the present disclosure, if the weight is less than the preset weight threshold, the weight of the region of interest is set to 0 and the weight of the other region of interest is set to 1, and then a minimum rectangle that can cover the two regions of interest is determined as a local refresh area; if the region of interest corresponding to the gaze screen or the region of interest corresponding to the touch screen both exist and have an intersection, then a minimum rectangle that can accommodate the union of the two regions of interest is determined as a local refresh area, where the preset weight threshold is pre-set, and exemplarily, the preset weight threshold is 0.025.

[0136] The disclosed embodiment can determine the weight by analyzing the degree of attention paid by the target object to the screen, and adjust the range of the region of interest according to the weight, thereby improving the rationality of the local refresh area finally determined and meeting the actual needs of the target object in using the screen.

[0137] In some embodiments, to determine the region of interest where the target object focuses on the content of interest based on the positions of multiple action points collected within a preset time period as described above, the following steps may be implemented:

[0138] Determine multiple target points among the multiple action points collected within the preset time period;

[0139] Taking the position of the target point as the center of a circle and the length corresponding to the continuous appearance duration of the target points collected at the same position as the diameter, calibrate multiple initial circular regions on the screen;

[0140] Determine the region of interest based on the multiple initial circular regions.

[0141] Here, after collecting multiple action points within the preset time period, the action trajectory can be determined according to the coordinates of the action points; in combination with the above, it can be the touch trajectory determined according to the coordinates of the screen touch points ζ touch ={ζ t1 , ζ t2 ,... ζ tn}, and the fixation point trajectory φ is formed according to the coordinates of the screen fixation points eye ={φ t1 , φ t2 ,... φ tn}.

[0142] Among them, in φ t1 =(x1, y1), x1 and y1 are the abscissa and ordinate of the screen fixation point at time t1 within the preset time period in the mobile phone coordinate system; in ζ t1 =(m1, n1), m1 and n1 are the abscissa and ordinate of the screen touch point at time t1 in the mobile phone coordinate system. Here, φ t1 , φ t2 ,... φ tn and ζ t1 , ζ t2 ,... ζ tn are also represented by coordinates. The mobile phone coordinate system is the same reference coordinate system. The preset time period sets multiple time points t1 to tn according to the preset sampling frequency.

[0143] The manner in which the embodiments of the present disclosure determine multiple target points among the multiple action points collected within the preset time period may be: for the coordinate data in the fixation point trajectory and the touch trajectory, after removing some action points by using a linear regression data model, the action points with high linear correlation are obtained as the target points.

[0144] In some embodiments, to determine multiple target points among the multiple action points collected within the preset time period as described above, the following steps may also be implemented:

[0145] Construct an initial matrix based on multiple action points collected within a preset time period. The number of rows of the initial matrix is the number of multiple action points, and the number of columns of the initial matrix is the number of groups of coordinates of the action points;

[0146] Perform dimensionality reduction on the initial matrix to obtain a target matrix, where the number of rows of the target matrix is less than the number of rows of the initial matrix;

[0147] Use the multiple action points that form the target matrix as multiple target points.

[0148] Here, the number of groups of coordinates in the example of the present disclosure is two groups, which are the abscissa and ordinate depicted in the mobile phone coordinate system. In other examples, the number of groups of coordinates can also be multiple groups. For example, in the world coordinate system, the number of groups of coordinates can also be three groups.

[0149] In the present disclosure, the dimensionality reduction method used to perform dimensionality reduction on the initial matrix to obtain the target matrix can be the principal component analysis method. Specifically: form an initial matrix X based on the action trajectories of n rows and 2 columns; determine the average vector M of each row in the initial matrix X; subtract the average vector M from each row of the initial matrix X to obtain a zero-mean initial matrix X; calculate the covariance matrix of the zero-mean initial matrix and obtain the eigenvalues and eigenvectors of the covariance matrix; arrange all the eigenvectors in ascending order of eigenvalues by row, and take the first k rows to form the basis vector P; multiply the basis vector P by the initial matrix X to obtain the target matrix after dimensionality reduction. Here, k is less than n; after obtaining the target matrix, the action trajectory after dimensionality reduction can be determined, that is, multiple target points can be determined.

[0150] In the present disclosure, constructing an initial matrix based on multiple action points collected within a preset time period can be respectively based on the touch trajectory ζ touch ={ζ t1 ,ζ t2 ,...ζ tn} to form a touch initial matrix according to n rows and 2 columns, and according to the fixation point trajectory φ eye ={φ t1 ,φ t2 ,...φ tn} to form a saccade initial matrix according to n rows and 2 columns; the present disclosure can respectively perform dimensionality reduction on the touch initial matrix and the saccade initial matrix to obtain a touch target matrix and a saccade target matrix, and determine the saccade trajectory φ' after dimensionality reduction based on the touch target matrix and the saccade target matrix eye ={φ' t1 ,φ' t2 ,...φ' tn}, and the touch trajectory ζ' after dimensionality reduction touch ={ζ' t1 ,ζ' t2 ,...ζ' tn} Correspondingly, a plurality of target fixation points and a plurality of target touch points are determined.

[0151] In the embodiments of the present disclosure, the action points are analyzed and processed by the principal component analysis method. After removing redundant and coupled variables, a set of principal component variables with the smallest coupling coefficient is obtained, which can count how many different action points are noticed within a preset time period, so that the present disclosure can select appropriate action points to represent the attention position of the target object to the content of interest.

[0152] In the present disclosure, after determining a plurality of target points, with the position of the target point as the center of the circle and the length corresponding to the continuous appearance duration of the target points collected at the same position as the diameter, a plurality of corresponding initial circular regions are marked on the screen, and corresponding regions of interest are determined based on the plurality of initial circular regions respectively.

[0153] Here, according to the coordinates of the plurality of target points, the acquisition time and the acquisition order of the target points, an action sequence is determined. This action sequence Seq eye ={λ T1 , λ T2 ,... λ Tk}, where λ T1 ={x T1 , y T1 , T1}, x T1 , y T1 represent the abscissa and ordinate of the target point gazed at during the time period T1, and T1 represents the continuous appearance duration of the target point; according to this action sequence, an action trajectory area diagram is drawn; here, the action trajectory area diagram is characterized by a plurality of initial circular regions and arrows between adjacent initial circular regions; the center of the initial circular region corresponds to a target point, the diameter of the initial circular region is positively correlated with the continuous appearance duration of the target point, and the arrows between the initial circular regions represent the displacement direction of the target point.

[0154] Combined with the above embodiments of the present disclosure, a plurality of target fixation points are determined from the dimension-reduced fixation point trajectory φ' eye ={φ' t1 , φ' t2 ,... φ' tn}, and a plurality of target touch points are determined from the dimension-reduced touch trajectory ζ' touch ={ζ' t1 , ζ' t2 ,... ζ' tnAfter determining multiple target touch points, respectively using the positions of multiple annotation viewpoints and multiple target touch points as the centers, and using the length corresponding to the continuous appearance duration of the target fixation point and the target touch point collected at the same position as the diameter, calibrate multiple initial circular regions corresponding to the touch screen and multiple initial circular regions corresponding to the fixation screen on the screen, and based on the multiple initial circular regions respectively, determine the region of interest corresponding to the touch screen and the region of interest corresponding to the fixation screen.

[0155] Here, the present disclosure can determine the fixation point trajectory region map based on the dimensionality-reduced fixation point trajectory φ′ eye ={φ′ t1 , φ′ t2 ,... φ′ tn} and the acquisition time of the target fixation point, and determine the region of interest corresponding to the fixation screen based on the fixation point trajectory region map; and, based on the dimensionality-reduced touch trajectory ζ′ touch ={ζ′ t1 , ζ′ t2 ,... ζ′ tn} and the acquisition time of the target touch point to determine the touch trajectory region map, and determine the region of interest corresponding to the touch screen based on the touch trajectory region map.

[0156] In some embodiments, for the above determination of the region of interest based on multiple initial circular regions, it can be implemented through the following steps:

[0157] Take the smallest circular region including multiple initial circular regions as the region of interest.

[0158] Exemplarily, a judgment window can be determined. The shape of the judgment window is circular. Determine whether multiple initial circular regions are all within the judgment window. If so, determine the current judgment window as the smallest circular region; if not, expand the judgment window according to a specified ratio and then judge again until the expanded judgment window can contain all the initial circular regions. At this time, determine the expanded judgment window as the smallest circular region, that is, as the region of interest.

[0159] The embodiments of the present disclosure can take the smallest circular region including multiple initial circular regions as the region of interest, thereby being able to reasonably determine the local refresh range, which can not only effectively meet the user's needs, but also minimize the power consumption of the screen refresh as much as possible and reduce the screen power consumption.

[0160] In some embodiments, for the above taking the smallest circular region including multiple initial circular regions as the region of interest, it can be implemented through the following steps:

[0161] When the data N of the initial circular region is equal to 2, the (N - 1)-th minimum circumscribed circle region is obtained based on the (N - 1)-th initial circular region and the N-th initial circular region;

[0162] When N is greater than 2, the (N - 1)-th minimum circumscribed circle region is obtained based on the N-th initial circular region and the (N - 2)-th minimum circumscribed circle region;

[0163] The (N - 1)-th minimum circumscribed circle region is used as the region of interest;

[0164] Where N is a positive integer greater than 1.

[0165] In the embodiments of the present disclosure, first, all the initial circular regions in the above-mentioned action trajectory region map are preliminarily filtered; the initial circular regions with a radius smaller than the preset radius in the initial circular regions are removed, and the remaining initial circular regions are determined as N initial circular regions; when N is equal to 2, the minimum circumscribed circle region of the two initial circular regions is determined as the region of interest; when N is greater than 2, the N initial circular regions are traversed in sequence, and the N-th initial circular region and the (N - 2)-th minimum circumscribed circle region are compared in sequence, so as to obtain the minimum circumscribed circle region including all the initial circular regions, and thus the region of interest is determined. Here, the preset radius is determined according to the size of the screen. Exemplarily, the preset radius can be 5 pixel points of the screen.

[0166] Here, taking N greater than 2 as an example, the method for determining the minimum circumscribed circle region can be: first, compare the intersection relationship between the first initial circular region and the second initial circular region to determine the first minimum circumscribed circle region, and then traverse the N initial circular regions in sequence, and compare the intersection relationship between the N-th initial circular region and the (N - 2)-th minimum circumscribed circle region to determine the (N - 1)-th minimum circumscribed circle region. Here, the method for determining a minimum circumscribed circle region is: compare the current two circular regions (which can be the initial circular region or the previous minimum circumscribed circle region). If the two circular regions do not intersect, the radius of their minimum circumscribed circle region is determined by the following formula (3). If the two circular regions intersect, the radius of their minimum circumscribed circle region is determined by the following formula (4).

[0167]

[0168] R = max(r1 + r2)+d / 2 (4)

[0169] Here, R is the radius of the minimum circumscribed circle region, r1 and r2 are the radii of the two circular regions participating in the comparison, and d is the distance between the two circular regions participating in the comparison.

[0170] In combination with the above embodiments of the present disclosure, after respectively determining the region of interest corresponding to the touch screen and the region of interest corresponding to the gaze screen, the weights of the two regions of interest can be adjusted according to the weights output by the Attention mechanism to obtain the final local refresh area.

[0171] An exemplary application of the screen refresh method proposed in the present disclosure is described below in combination with the above-mentioned embodiments of the present disclosure.

[0172] Here, the screen refresh method proposed in the present disclosure can be applied to the fields of mobile phones / tablets / touch-screen TVs / car-mounted electronic terminals, and can also be applied to laptops with touchpads. Normally, when the human eye observes the entire screen, its content of interest (ROI) may only be the display content of a small area of the screen. For example, when playing an animation, the human eye focuses most of the time on the body language of the animated characters rather than the background; when using map software to zoom in and out on the map, the user's finger sliding trajectory is usually only within a fixed small area. Therefore, in these scenarios, the full-screen refresh solution currently used by mobile phones requires rendering the display screen of the entire mobile phone screen. For the display area that does not need to be updated every frame, full-screen refresh wastes computing resources and increases power consumption.

[0173] Therefore, this paper proposes a local refresh scheme for the local refresh area. As for the local refresh scheme, it mainly includes the local refresh scheme based on the support of the screen side and the local refresh scheme supported by the AP side; the local refresh scheme supported by the AP is realized by adjusting the frame rate of the image transmission on the AP side. Specifically, within the specified time, the display area chip DDIC of the screen side can receive the 2A (define the column of the image transmission) / 2B (define the end row of the image transmission) instructions sent by the AP side to determine which area of the screen the AP image data is updated in. The local refresh method supported by the screen side can be to add additional masking circuits and clock signals to the driving circuit of the screen side to set whether the next level or next levels of signals generated by the boundary of the corresponding partition to continue to be transmitted.

[0174] However, the local refresh solution on the AP side usually needs to determine the frame refresh difference based on the previous and next frame images. The system needs to constantly cache and read the previous frame image to calculate the image difference, which can easily cause I / O busy and CPU to work at high load for a long time. For high-resolution videos / pictures and scenes where the changes between the previous and next frames are not obvious, it is not friendly to memory usage and memory reading and writing. The local refresh solution on the screen side has strict requirements on the type of display screen. It is currently limited to e-paper scenes and cannot be applied to LED screens or OLED screens. It does not have good universality.

[0175] Based on the above problems, the present disclosure proposes a screen refresh method; seeFigure 3 , Figure 3 is a schematic flow chart of a method for performing partial screen refresh by combining a gaze screen and a touch screen shown according to an exemplary embodiment; as Figure 3 shown, the screen refresh method proposed by the present disclosure can be implemented through the following steps:

[0176] Step S301, start;

[0177] Step S302, collect the coordinates of the screen gaze point formed by the target object's gaze on the screen and the coordinates of the screen touch point formed by the target object's touch on the screen;

[0178] In the present disclosure, collecting the eye movement data of the target object's gaze on the screen and determining the gaze point trajectory can be implemented in the following manner: taking an eye image through a front camera within a preset time period, and calibrating the center position of the pupil of the eye through an image processing method; using the corneal reflection point (yellow spot) of the eye as the base point of the relative position between the camera and the eye, and at this time, according to the calibrated pupil center, eye movement data such as the line-of-sight vector indicating the position change of the pupil of the eye relative to the cornea of the eye can be obtained; then, according to the mapping relationship between the eye movement data and the coordinates of the screen gaze point (such as the mapping function between the vector formed by the pupil and the corneal reflection point and the screen gaze point), the coordinates of the screen gaze point are obtained, and a gaze point trajectory φ eye ={φ t1 ,φ t2 ,...φ tn}.

[0179] Here, collecting the touch trajectory of the target object's touch on the screen can be implemented in the following manner: when a touch event is reported by an under-screen sensor of the screen, such as a capacitive sensor, according to the touch points of the target object's finger touching the screen collected at a preset sampling frequency, such as 240 Hz, within a preset time period, the coordinates of the screen touch point formed on the screen are determined. A touch trajectory φ eye ={φ t1 ,φ t2 ,...φ tn} is formed based on the coordinates of the screen touch point.

[0180] Step S303, respectively form corresponding initial matrices based on the screen touch points and the screen gaze points, and perform feature dimensionality reduction on the initial matrices to obtain corresponding target matrices;

[0181] Here, respectively based on the touch trajectory ζ touch ={ζ t1 ,ζ t2 ,...ζ tn}, a touch initial matrix is formed according to n rows and 2 columns, and based on the gaze point trajectory φ eye={φ t1 , φ t2 ,... φ tn} is composed into an initial eye movement matrix according to n rows and 2 columns; the touch initial matrix and the eye movement initial matrix are respectively dimensionally reduced according to the principal component analysis method to obtain a touch target matrix and an eye movement target matrix.

[0182] Step S304, obtaining a target touch point corresponding to the touch screen based on multiple action points forming a target matrix corresponding to the touch screen, and obtaining a target fixation point corresponding to the fixation screen based on multiple action points forming a target matrix corresponding to the fixation screen;

[0183] Based on the eye movement target matrix, the reduced fixation point trajectory φ′ eye ={φ′ t1 , φ′ t2 ,... φ′ tn} is obtained, and the target fixation point is obtained therefrom. At the same time, based on the touch target matrix, the reduced touch trajectory ζ′ touch ={ζ′ t1 , ζ′ t2 ,... ζ′ tn} is obtained, and the target touch point is obtained therefrom.

[0184] See Figure 4 , Figure 4 which is a schematic flowchart of feature dimensional reduction shown according to an exemplary embodiment; as Figure 4 shown, the dimensional reduction method proposed above in the present disclosure can be implemented through the following steps:

[0185] Step S401, forming an initial matrix based on the action trajectories of n rows and 2 columns;

[0186] Here, the action trajectories are respectively the fixation point trajectory φ′ eye ={φ′ t1 , φ′ t2 ,... φ′ tn} and the touch trajectory ζ′ touch ={ζ′ t1 , ζ′ t2 ,... ζ′ tn}.

[0187] Step S402, determining the average vector M of each row in the initial matrix X;

[0188] Step S403, subtracting each row of the initial matrix X from the average vector M to obtain a zero-meaned initial matrix X;

[0189] Step S404, obtaining the covariance matrix C of the zero-meaned initial matrix X, and obtaining the eigenvalues and eigenvectors of the covariance matrix C;

[0190] Here, the covariance matrix can be determined by the following formula (5):

[0191]

[0192] Step S405: Arrange all the eigenvectors in ascending order of eigenvalues row by row, and take the first k rows to form the basis vector P; here, k is a positive integer less than n.

[0193] Step S406: Multiply the basis vector P by the initial matrix X to obtain the target matrix after dimensionality reduction. Here, k is less than n; after obtaining the target matrix, the action trajectory after dimensionality reduction can be determined.

[0194] Combining Step S303 and Step S304, through the Figure 4 feature method shown, the fixated point trajectory φ' after dimensionality reduction is obtained eye ={φ' t1 , φ' t2 ,... φ' tn}, and the target fixated point is determined therefrom; at the same time, the touch trajectory ζ' after dimensionality reduction is obtained touch ={ζ' t1 , ζ' t2 ,... ζ' tn}, and the target touch point is obtained therefrom.

[0195] Step S305: Generate a fixated point trajectory area map and a touch trajectory area map based on the target fixated point and the target touch point;

[0196] The present disclosure can draw a fixated point trajectory area map based on the coordinates of the target fixated point and the acquisition time in the fixated point trajectory φ' after dimensionality reduction eye ={φ' t1 , φ' t2 ,... φ' tn}; referring to Figure 5 , Figure 5 is a schematic diagram of a fixated point trajectory area map shown according to an exemplary embodiment; as shown in Figure 5 , Figure 5 is used to characterize the range corresponding to the content of interest on the screen where the target object's eyes fixate. Here, the coordinates of the center point of each initial circular area in the fixated point trajectory area map are a fixated point trajectory φ' after dimensionality reduction eye ={φ' t1 , φ' t2 ,... φ' tn}, the coordinates of the target fixation point in it, and the diameter of the initial circular area is determined by the continuous appearance duration of the target fixation point. Specifically, the longer the target object continuously fixates on a certain point on the screen, the more attention is invested. Correspondingly, the larger the diameter of the initial circular area on the fixation point trajectory area graph; as Figure 5 shown, the initial circular area is shown with numbers (1, 2, 3, 4...), and the numbers can be used to represent the diameter size.

[0197] Similarly, based on the touch trajectory ζ′ after dimensionality reduction touch ={ζ′ t1 , ζ′ t2 ,... ζ′ tn}, the coordinates of the target touch point and the acquisition time are used to draw the touch trajectory area graph; see Figure 6 , Figure 6 is a schematic diagram of the touch trajectory area graph shown according to an exemplary embodiment; as Figure 6 shown, Figure 6 is used to characterize the range corresponding to the content of interest in which the target object's finger touches the screen.

[0198] Here, the coordinates of the center point of each initial circular area in the touch trajectory area graph are the coordinates of a target touch point in a touch trajectory ζ′ after dimensionality reduction touch ={ζ′ t1 , ζ′ t2 ,... ζ′ tn}, and the diameter of the initial circular area is determined by the continuous appearance duration of the target touch point. Specifically, the longer the target object continuously touches a certain point on the screen, the more attention is invested. Correspondingly, the larger the diameter of the initial circular area on the touch trajectory area graph; as Figure 6 shown, the initial circular area is shown with numbers (1, 2, 3, 4...), and the numbers can be used to represent the diameter size.

[0199] Step S306, respectively combine the fixation point trajectory area graph and the touch trajectory area graph according to the attention mechanism model to determine the local refresh area;

[0200] In the present disclosure, the area of interest corresponding to the touch screen is determined based on the touch trajectory area graph, and the area of interest corresponding to the fixation screen is determined based on the touch trajectory area graph.

[0201] Here, the method for determining the region of interest is as follows: initially filter all the initial circular regions; remove the initial circular regions with a radius smaller than the preset radius from the initial circular regions, and determine the remaining initial circular regions as N initial circular regions; when N is equal to 2, determine the minimum circumscribed circle region of the two initial circular regions as the region of interest; when N is greater than 2, sequentially traverse the N initial circular regions, and sequentially compare the Nth initial circular region with the (N - 2)th minimum circumscribed circle region to obtain the minimum circumscribed circle region including all the initial circular regions, and thereby determine the final minimum circumscribed circle region as the region of interest. Here, the preset radius is determined according to the size of the screen. Exemplarily, the preset radius can be 5 pixel points of the screen.

[0202] Here, the method for determining a minimum circumscribed circle region is as follows: compare the current two circular regions (which can be the initial circular regions or the previous minimum circumscribed circle region). If the two circular regions do not intersect, the radius of their minimum circumscribed circle region is determined by formula (3) of the present disclosure. If the two circular regions intersect, the radius of their minimum circumscribed circle region is determined by formula (4) of the present disclosure.

[0203] After determining the region of interest corresponding to the touch screen and the region of interest corresponding to the gaze screen, input the parameter S of the region of interest corresponding to the touch screen touch and the parameter S of the region of interest corresponding to the gaze screen eye into the attention mechanism model; here, corresponding to S eye and S touch can be the center coordinates and radius representing the range of the region of interest; the attention mechanism model can output the weight ω of the region of interest corresponding to the gaze screen eye and the weight ω of the region of interest corresponding to the touch screen touch . After determining ω eye and ω touch , adjust the weight of the region of interest corresponding to the touch screen to obtain the second adjusted region ω eye S eye and the first adjusted region ω touch S touch ; the present disclosure takes the minimum rectangular region including the first adjusted region and the second adjusted region as the local refresh region.

[0204] See Figure 7a , Figure 7b and Figure 7c , Figure 7a which is a schematic diagram showing the principle of determining the local refresh region according to an exemplary embodiment Figure 1 ; Figure 7b which is a schematic diagram showing the principle of determining the local refresh region according to an exemplary embodimentFigure 2 ; Figure 7c It is a schematic diagram showing the principle of determining a local refresh area according to an exemplary embodiment Figure 3 ; combined with Figure 7a 、 Figure 7b and Figure 7c shown, if only the region of interest W corresponding to the gaze screen or the region of interest E corresponding to the touch screen is determined within a preset time period, according to the Attention mechanism model, the weight of the corresponding region of interest within the preset time period is set to the highest (e.g., 1), and the weight of the other region of interest is set to 0, and a minimum rectangular region that can cover the circular region is determined as the local refresh area; if both the region of interest W corresponding to the gaze screen and the region of interest E corresponding to the touch screen exist and have no intersection, then the weights of the two regions of interest are adjusted according to the weights output by the Attention mechanism. In the present disclosure, if the weight is less than the preset weight threshold, the weight of the region of interest is set to 0, and the weight of the other region of interest is set to 1, and then a minimum rectangle that can cover the two regions of interest is determined as the local refresh area; if both the region of interest W corresponding to the gaze screen and the region of interest E corresponding to the touch screen exist and have an intersection, then a minimum rectangle that can accommodate the union of the two regions of interest is determined as the local refresh area. Here, the preset weight threshold is preset. Exemplarily, the preset weight threshold is 0.025.

[0205] See Figure 8 , Figure 8 It is a schematic diagram showing the principle of jointly determining a local refresh area by combining the gaze screen and the touch screen according to an exemplary embodiment; as Figure 8 shown, the present disclosure determines the local refresh area by combining the gaze point trajectory area map formed by the target object gazing at the screen and the touch trajectory area map formed by the target object touching the screen

[0206] Step S307, the AP side generates a map sending instruction based on the local refresh area;

[0207] Here, after the AP side determines the local refresh area according to the data collected by the camera or the under-screen capacitance and other acquisition components, the diagonal coordinates of the local refresh area are determined on the screen. Taking the lower left corner of the screen as the origin to establish a plane rectangular coordinate system as an example, the diagonal coordinates are the coordinates O1 = {a1, b1} of the upper left vertex and the coordinates O2 = {a2, b2} of the lower right vertex of the local refresh area.

[0208] Based on the coordinates of the upper left vertex O1 = {a1, b1} and the coordinates of the lower right vertex O2 = {a2, b2}, determine the pixel rows and pixel columns for local rendering at the AP end. The AP end generates a picture sending instruction of 2A (defining the column of picture sending) / 2B (defining the end row of picture sending) based on the pixel rows and pixel columns, and sends it to the screen end.

[0209] Step S308, the display driver chip at the screen end obtains the picture sending instruction to update the content of interest in the local refresh area;

[0210] Here, within 1 frame time, the display driver chip at the screen end receives the picture sending instruction sent by the AP end, determines which area of the screen the image data of the AP is updated in, and parses the instruction to obtain pixel data, thereby refreshing the specified local refresh area to update the content of interest.

[0211] Step S309, end.

[0212] Through the screen refresh method proposed in the present disclosure, it is possible to directly delimit the local refresh range of the screen according to the fixation point trajectory and touch trajectory formed on the screen by the target object's attention to the screen, without storing the information of the current frame and the previous frame, reducing memory consumption and the frequency of I / O reading and writing, reducing power consumption and improving the smoothness of mobile phone operation; moreover, getting rid of the bondage of the screen medium for picture drawing, it can directly control the picture sending at the AP end, which has good universality for local refreshing of display screens such as LCD display screens and OLED display screens; furthermore, the present disclosure collects eye data through a camera, broadens the usage scenarios of the camera, and maximally exerts the functions of the camera.

[0213] See Figure 9 , Figure 9 is a schematic structural diagram of an electronic device shown according to an exemplary embodiment; as Figure 9 shown, the electronic device 900 proposed by the present disclosure includes:

[0214] An acquisition module 901, configured to acquire the position of the action point formed on the screen when the target object pays attention to the content of interest on the screen; wherein, when the display positions of the content of interest are different, the positions of the action points are different;

[0215] A determination module 902, configured to determine the local refresh area of the screen based on the position of the action point;

[0216] An update module 903, configured to update the content of interest in the local refresh area.

[0217] In some embodiments, the determination module 902 is further configured to determine the area of interest where the target object pays attention to the content of interest based on the positions of multiple action points collected within a preset time period;

[0218] Define the local refresh area on the screen according to the region of interest.

[0219] In some embodiments, the position of the action point includes the coordinates of the screen touch point formed on the screen when the target object touches the screen and / or the coordinates of the screen gaze point formed on the screen when the target object gazes at the screen; the determination module 902 is further configured to determine the region of interest corresponding to touching the screen based on the coordinates of multiple screen touch points collected within the preset time period; and / or determine the region of interest corresponding to gazing at the screen based on the coordinates of multiple screen gaze points collected within the preset time period.

[0220] In some embodiments, the determination module 902 is further configured to use the smallest rectangular area including the region of interest corresponding to touching the screen as the local refresh area; or use the smallest rectangular area including the region of interest corresponding to gazing at the screen as the local refresh area; or determine the local refresh area based on the region of interest corresponding to touching the screen and the region of interest corresponding to gazing at the screen.

[0221] In some embodiments, the determination module 902 is further configured to perform weight adjustment on the region of interest corresponding to touching the screen to obtain a first adjusted area; perform weight adjustment on the region of interest corresponding to gazing at the screen to obtain a second adjusted area; and use the smallest rectangular area including the first adjusted area and the second adjusted area as the local refresh area.

[0222] In some embodiments, the determination module 902 is further configured to determine multiple target points among multiple action points collected within the preset time period; use the positions of the target points as the centers, and use the lengths corresponding to the continuous appearance durations of the target points collected at the same positions as the diameters to mark multiple initial circular areas on the screen; and determine the region of interest based on the multiple initial circular areas.

[0223] In some embodiments, the determination module 902 is further configured to construct an initial matrix based on multiple action points collected within the preset time period, where the number of rows of the initial matrix is the number of multiple action points, and the number of columns of the initial matrix is the number of coordinate groups of the action points; perform dimensionality reduction processing on the initial matrix to obtain a target matrix, where the number of rows of the target matrix is less than the number of rows of the initial matrix; and use the multiple action points forming the target matrix as the multiple target points.

[0224] In some embodiments, the determining module 902 is further configured to use the smallest circular region including a plurality of the initial circular regions as the region of interest.

[0225] In some embodiments, when the data N of the initial circular region is equal to 2, the determining module 902 is further configured to obtain the (N - 1)-th minimum circumscribed circle region based on the (N - 1)-th initial circular region and the N-th initial circular region; when N is greater than 2, obtain the (N - 1)-th minimum circumscribed circle region based on the N-th initial circular region and the (N - 2)-th minimum circumscribed circle region; and use the (N - 1)-th minimum circumscribed circle region as the region of interest, where N is a positive integer greater than 1.

[0226] In some embodiments, the position of the action point includes the coordinates of the screen fixation point formed on the screen when the target object gazes at the screen. The obtaining module 901 is further configured to collect the eyes of the target object based on the camera assembly of the electronic device, and obtain eye movement data of the pupil of the eye changing its position relative to the cornea of the eye; and obtain the coordinates of the screen fixation point based on the eye movement data and a preset mapping relationship between the eye movement data and the coordinates of the screen fixation point.

[0227] In some embodiments, the updating module 903 is further configured to determine the pixel rows and pixel columns for image refreshing according to the focusing coordinates of the partial refresh region on the screen; and update the content of interest based on the pixel rows and the pixel columns.

[0228] Figure 10 FIG. is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0229] Refer to Figure 10 , the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0230] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-described methods. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0231] The memory 804 is configured to store various types of data to support the operation of the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, and the like. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0232] The power component 806 provides power to the various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0233] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0234] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a Microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0235] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0236] The sensor assembly 814 includes one or more sensors for providing an assessment of the status of the electronic device 800 in various aspects. For example, the sensor assembly 814 can detect the on / off state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 can also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0237] The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices in a wired or wireless manner. The electronic device 800 can access a wireless network based on communication standards, such as WiFi, 3G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0238] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0239] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including a computer program or instructions. The computer program or instructions can be executed by a processor 820 of the electronic device 800 to complete the above screen refreshing method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0240] A non-transitory computer-readable storage medium, when the computer program or instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the screen refreshing method proposed in the above embodiments of the present disclosure.

[0241] The embodiments of the present disclosure provide a computer program product, which includes: a computer program or instructions, and the computer program or instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or instructions from the computer-readable storage medium, and the processor executes the computer program or instructions, so that the computer device executes any one of the above screen refreshing methods in the embodiments of the present disclosure.

[0242] Those skilled in the art will readily conceive of other implementations of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and embodiments are only to be regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0243] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A screen refreshing method, characterized in that, The method includes: Obtaining the position of an action point formed on the screen when a target object focuses on an interesting content on the screen; wherein, when the display position of the interesting content is different, the position of the action point is different; Determining a local refresh area of the screen based on the position of the action point; Updating the interesting content in the local refresh area.

2. The method according to claim 1, wherein The determining the local refresh area of the screen based on the position of the action point includes: Determining an interesting area where the target object focuses on the interesting content based on the positions of multiple action points collected within a preset time period; Defining the local refresh area on the screen according to the interesting area.

3. The method according to claim 2, wherein The position of the action point includes the coordinates of a screen touch point formed on the screen when the target object touches the screen and / or the coordinates of a screen gaze point formed on the screen when the target object gazes at the screen; the determining the interesting area where the target object focuses on the interesting content based on the positions of multiple action points collected within a preset time period includes: Determining the interesting area corresponding to touching the screen based on the coordinates of multiple screen touch points collected within the preset time period; and / or, determining the interesting area corresponding to gazing at the screen based on the coordinates of multiple screen gaze points collected within the preset time period.

4. The method according to claim 3, characterized in that, The defining the local refresh area on the screen according to the interesting area includes: Taking the smallest rectangular area including the interesting area corresponding to touching the screen as the local refresh area; or, Taking the smallest rectangular area including the interesting area corresponding to gazing at the screen as the local refresh area; or, Determining the local refresh area based on the interesting area corresponding to touching the screen and the interesting area corresponding to gazing at the screen.

5. The method according to claim 4, wherein The determining the local refresh area based on the interesting area corresponding to touching the screen and the interesting area corresponding to gazing at the screen includes: Performing weight adjustment on the interesting area corresponding to touching the screen to obtain a first adjusted area; Performing weight adjustment on the interesting area corresponding to gazing at the screen to obtain a second adjusted area; Taking the smallest rectangular area including the first adjusted area and the second adjusted area as the local refresh area.

6. The method according to claim 2, characterized in that, The determining the interesting area where the target object focuses on the interesting content based on the positions of multiple action points collected within a preset time period includes: Determining multiple target points among the multiple action points collected within the preset time period; Taking the position of the target point as the center and taking the length corresponding to the continuous appearance duration of the target points collected at the same position as the diameter, calibrating multiple initial circular areas on the screen; Determining the interesting area based on the multiple initial circular areas.

7. The method according to claim 6, characterized in that The determining multiple target points among the multiple action points collected within the preset time period includes: Construct an initial matrix based on multiple action points collected within the preset time period, where the number of rows of the initial matrix is the number of multiple action points, and the number of columns of the initial matrix is the number of coordinate groups of the action points; Perform dimensionality reduction processing on the initial matrix to obtain a target matrix, where the number of rows of the target matrix is less than the number of rows of the initial matrix; Use the multiple action points that form the target matrix as multiple target points.

8. The method according to claim 6, characterized in that, The determining the region of interest based on multiple initial circular regions includes: Use the smallest circular region that includes multiple initial circular regions as the region of interest.

9. The method according to claim 8, characterized in that The using the smallest circular region that includes multiple initial circular regions as the region of interest includes: When the data N of the initial circular region is equal to 2, obtain the (N - 1)-th minimum circumscribed circle region based on the (N - 1)-th initial circular region and the N-th initial circular region; When N is greater than 2, obtain the (N - 1)-th minimum circumscribed circle region based on the N-th initial circular region and the (N - 2)-th minimum circumscribed circle region; Use the (N - 1)-th minimum circumscribed circle region as the region of interest; where N is a positive integer greater than 1.

10. The method according to any one of claims 1 to 9, characterized in that, The position of the action point includes the coordinates of the screen fixation point formed on the screen when the target object gazes at the screen. The obtaining the position of the action point formed on the screen when the target object focuses on the interesting content on the screen includes: Collect the eyes of the target object based on the camera component of the electronic device, and obtain the eye movement data of the pupil of the eye relative to the cornea of the eye; Based on the eye movement data and the preset mapping relationship between the eye movement data and the coordinates of the screen fixation point, obtain the coordinates of the screen fixation point.

11. The method according to any one of claims 1 to 9, characterized in that, The updating the interesting content in the local refresh region includes: Determine the pixel rows and pixel columns for image refreshing according to the diagonal coordinates of the local refresh region on the screen; Based on the pixel rows and the pixel columns, update the interesting content.

12. An electronic device, characterized in that, Includes: An acquisition module configured to acquire the position of the action point formed on the screen when the target object focuses on the interesting content on the screen; where when the display position of the interesting content is different, the position of the action point is different; A determination module configured to determine the local refresh region of the screen based on the position of the action point; An update module configured to update the interesting content in the local refresh region.

13. An electronic device, characterized in that, Includes: A processor; A memory for storing computer programs or instructions of the processor; where the processor is configured to execute the computer program or instructions to implement the steps of the screen refreshing method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by the processor, the steps of the screen refreshing method according to any one of claims 1 to 11 are implemented.

15. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, the steps of the screen refreshing method according to any one of claims 1 to 11 are implemented.

Images