Page turning method, electronic equipment and storage medium

Abstract

The invention is used for providing a page turning method, electronic equipment and a storage medium, the page turning method is used for the electronic equipment, the page turning method comprises the steps that an application interface of a first application is displayed, the application interface comprises a first display area and a second display area, and the first display area and the second display area are partially overlapped or completely not overlapped; determining page turning waiting time of the application interface of the first application in response to determining that the fixation point of the user relative to the application interface is moved from the first display area to the second display area; and in response to determining that the fixation point of the user is moved from the first display area to the second display area, after the page turning waiting time and in response to determining that the fixation point of the user is moved from the second display area to the first display area, executing page turning operation for an application interface of the first application. According to the method, accurate and automatic page turning can be carried out, and the reading experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of software technology, and in particular to a page turning method, an electronic device and a storage medium. Background Art

[0002] With the rapid development of network technology, smart terminal devices such as smart phones and tablet computers are becoming more and more popular, bringing great convenience to people's lives, studies and work. In recent years, digital reading has become very popular, and most of the current reading software, such as reading applications (Application, APP), browser APP, information flow APP, etc., all use manual page turning interaction methods, such as sliding fingers to turn pages, clicking fingers to turn pages, etc. This manual page turning method requires users to use their hands frequently for a long time to operate, which is easy to fatigue and requires one-handed operation. When it is inconvenient for users to use their hands to operate or the screen of the device is too large to operate, it will reduce the user's interactive experience.

[0003] In addition, some reading software also provides an automatic page turning interactive mode, such as scrolling up and down. The scrolling up and down interactive mode can scroll the page down or up at a uniform speed according to the scrolling speed set by the user. Since the scrolling speed is fixed, but the content density and difficulty of the page are variable, that is, the user's reading speed is variable, the page scrolling speed cannot be well matched with the user's reading speed, resulting in a poor reading experience for the user in the scrolling up and down mode. Summary of the invention

[0004] Some embodiments of the present application provide a page turning method, an electronic device, and a storage medium. The present application is introduced from multiple aspects below, and the embodiments and beneficial effects of the following aspects can be referenced to each other.

[0005] In a first aspect, an embodiment of the present application provides a page turning method for an electronic device, the method comprising:

[0006] Displaying an application interface of a first application, the application interface comprising a first display area and a second display area, the first display area and the second display area partially overlapping or completely non-overlapping;

[0007] In response to determining that a user's gaze point relative to the application interface moves from the first display area to the second display area, determining a page turning waiting time of the application interface of the first application;

[0008] In response to determining that the user's gaze point moves from the first display area to the second display area, after a page turning waiting time has passed, and in response to determining that the user's gaze point moves from the second display area to the first display area, a page turning operation is performed on the application interface of the first application.

[0009] According to the embodiment of the present application, the electronic device can divide the application interface into an upper area (as an example of the first display area) and a lower area (as an example of the second display area), and can detect changes in the user's gaze point based on eye tracking technology. After determining that the user's gaze point moves from the upper area to the lower area, the electronic device can estimate the remaining reading time required for the user to finish reading the page content in the lower area as the page turning waiting time, and enter a waiting time window. After the waiting time window, when it is determined that the user's gaze point moves from the lower area to the upper area again, the electronic device can determine that the user's intention to turn the page is detected, and then perform an automatic page turning operation. In this way, when the user decides to turn the page after reading the page content in the lower area, he only needs to move the gaze point up to the upper area, and the page can be automatically turned without any mechanical movement of the user's hand, which greatly reduces fatigue and improves the user's reading experience. In addition, the electronic device recognizes the user's reading progress and the intention to turn the page on the current page by detecting changes in the user's gaze point, and has high recognition accuracy, good fault tolerance and strong applicability.

[0010] In addition, the electronic device sets the overlapping part between the upper area and the lower area as a transition area. When the user's gaze point moves from the upper grid (as an example of the area in the upper area other than the transition area) to the transition area, no change in the area where the user's gaze point is located is detected. When the user's gaze point moves from the lower grid (as an example of the area in the lower area other than the transition area) to the transition area, no change in the area where the user's gaze point is located is detected. The size of the transition area can affect the sensitivity of the gaze point switching between the upper and lower areas. The larger the transition area, the lower the sensitivity, and the smaller the transition area, the higher the sensitivity. Therefore, by setting an intermediate transition area, the electronic device can adjust the sensitivity of the gaze point switching between the upper and lower areas, and avoid the problem of frequent detection of the switching of the area where the user's gaze point is located when the user's gaze point jumps in the middle area, causing intention recognition errors.

[0011] In some embodiments, the method further comprises:

[0012] In response to determining that the user's gaze point moves from the first display area to the second display area, after a page turning waiting time has elapsed, displaying page turning prompt information in the application interface of the first application;

[0013] The page turning prompt information is used to prompt the user to move the gaze point from the second display area to the first display area to turn the page.

[0014] According to the implementation mode of the present application, after determining that the user's gaze point moves from the upper area to the lower area and after the waiting time window, the electronic device can determine that the user has finished reading the page content of the current page (i.e., read to the bottom of the page). At this time, the electronic device can give a page turning prompt to guide the user to look up and turn the page, so that the user can get a clear psychological hint that the intention has been accurately recognized, further improving the user's reading experience.

[0015] In some implementations, determining that a user's gaze point relative to the application interface moves from the first display area to the second display area includes:

[0016] Determine a display area where the user's gaze point is currently located as the second display area, and determine a display area where the user's gaze point was previously located as the first display area;

[0017] Wherein, determining that the user's gaze point moves from the second display area to the first display area includes:

[0018] Determine a display area where the user's gaze point is currently located as a first display area, and determine a display area where the user's gaze point was previously located as a second display area;

[0019] The display area where the user's gaze point is currently located and the display area where the user's gaze point was last located are determined based on at least one frame of the user's facial image.

[0020] According to the implementation mode of the present application, the electronic device can use one or more frames of facial images to determine the coordinates of the user's gaze point, and then determine the specific display area where the user's gaze point is concentrated on the application interface. Through this multi-frame smoothing processing method, the accuracy of determining the display area where the user's gaze point is located can be improved, the problem of gaze point jump can be effectively suppressed, and the accuracy of reading progress and page turning intention recognition can be further improved.

[0021] In some implementations, determining a page turning waiting time of an application interface of a first application includes:

[0022] Determining a page reading speed of a user in a first display area;

[0023] estimating the user's remaining reading time in the second display area based on the page reading speed;

[0024] The page turning waiting time is determined based on the remaining reading time.

[0025] According to the implementation mode of the present application, the electronic device dynamically calculates the reading speed of the user reading the upper area, and estimates the reading time of the user reading the lower area based on the reading speed, thereby converting the judgment of the reading progress into the judgment of the reading time. In this way, the electronic device does not need to accurately identify which word or line on the page the user's gaze point falls on, which provides favorable conditions for the solution of using low-power cameras for human eye tracking, so that the battery life of the electronic device can be unaffected or less affected when reading for a long time, so that users are willing to use it for a long time, reducing the pressure, interference and worries of users when using it for a long time.

[0026] In some implementations, determining a page reading speed of a user in a first display area includes:

[0027] Obtaining a first cumulative time that the user's gaze point has stayed in the first display area;

[0028] The page reading speed is determined according to the first accumulated time and the amount of page information in the first display area.

[0029] In some implementations, estimating the remaining reading time of the user in the second display area based on the page reading speed includes:

[0030] When the page reading speed is within the preset speed range, determining the estimated reading time of the user in the second display area according to the amount of page information in the second display area and the page reading speed;

[0031] When the page reading speed is outside the preset speed range, determining the estimated reading time of the user in the second display area according to the amount of page information in the second display area and the preset reading speed or the user's historical reading speed;

[0032] The remaining reading time is determined according to the estimated reading time and the second accumulated time that the user's gaze point has stayed in the second display area.

[0033] According to the implementation mode of the present application, the electronic device dynamically calculates the user's reading speed in the upper area based on the amount of page information in the upper area of ​​the current page and the cumulative time that the user's gaze point stays in the upper area. Then, based on the calculated reading speed, the expected time for the user to finish reading the page content in the lower area and the remaining reading time for the user to finish reading the page content in the lower area are estimated. In this way, the electronic device can accurately determine the user's reading progress based on the user's reading time, solving the problem of being unable to accurately identify the user's gaze point position due to the accuracy limitations of low-power camera technology and being unable to determine the user's reading progress based on the accurate position of the user's gaze point.

[0034] At the same time, by counting the cumulative time the user spends reading the upper half of the screen and the cumulative time the user spends reading the lower half of the screen, the waiting time window for detecting the user's intention to turn the page is dynamically determined, which can further reduce the impact of the problem of the user's gaze point jumping in the middle area on the accuracy of page turning intention recognition.

[0035] In addition, the electronic device determines whether the calculated reading speed is within the normal range, or judges whether there is a large error in combination with the user's historical reading speed, discards data that does not meet the normal range or has a large error, and uses a preset reading speed or the user's historical reading speed to estimate the reading time, thereby further ensuring the accuracy of identifying the user's reading progress.

[0036] In some implementations, determining the page turning waiting time based on the remaining reading time includes:

[0037] When the remaining reading time is within a preset time range, the page turning waiting time is determined as the remaining reading time;

[0038] When the remaining reading time is less than the minimum value of the preset time range, determining the page turning waiting time to be the first preset time;

[0039] When the remaining reading time is greater than the maximum value of the preset time range, the page turning waiting time is determined to be a second preset time.

[0040] According to the implementation mode of the present application, the electronic device determines whether the estimated remaining reading time is within a reasonable time range, discards data that does not meet the reasonable time range, and uses a preset time as the page turning waiting time. This can avoid turning pages too quickly (or prompting the user to turn pages too quickly, resulting in triggering false page turning) or not turning pages for too long (or not prompting the user to turn pages for too long, resulting in the user waiting time being too long), so that the time for automatic page turning is within a reasonable time range, further improving the user's reading experience.

[0041] In some implementations, performing a page turning operation on an application interface of a first application includes:

[0042] In response to determining that the user's gaze point moves from the second display area to the first display area, after a third preset time, a page turning operation is performed on the application interface of the first application.

[0043] In some embodiments, the method further comprises:

[0044] Within a third preset time, in response to determining that the user's gaze point moves from the first display area to the second display area, the page turning operation on the application interface of the first application is abandoned.

[0045] According to the implementation mode of the present application, after the electronic device determines that the user's gaze point has moved from the lower area to the upper area, it enters a delayed execution time window. When there is no switching of the user's gaze point between the upper and lower areas in the delayed execution time window, it is considered that the user's intention to turn the page by moving the gaze point upward is real, and the electronic device performs the page turning operation. If the user's gaze point moves to the lower area again in the delayed execution time window, it is considered that the previous upward movement of the user's gaze point was an erroneous operation and is suppressed. Through this delayed execution time window, the problem of wrong page turning caused by the user mistakenly looking at the upper area when the user has not finished reading the page content in the lower area, and the problem of wrong page turning caused by the gaze point jumping due to algorithm recognition errors can be avoided.

[0046] In a second aspect, an embodiment of the present application provides a page turning method for an electronic device, the method comprising:

[0047] Displaying an application interface of a second application, the application interface comprising a first display area and a second display area, the first display area and the second display area partially overlapping or completely non-overlapping;

[0048] In response to determining that the user's gaze point relative to the application interface moves from the first display area to the second display area, and after a fourth preset time, the page scrolling speed of the application interface of the second application is adjusted from the first scrolling speed to the second scrolling speed.

[0049] According to an embodiment of the present application, the electronic device may divide the application interface into an upper area and a lower area (as an example of a first display area and a second display area), and may detect changes in the user's gaze point based on eye tracking technology. In the case where the user is accustomed to gazing at the upper half of the application interface, if it is determined that the user's gaze point moves from the upper area to the lower area, it can be determined that the scrolling speed of the page is slower than the user's reading speed, and the electronic device can increase the page scrolling speed. In the case where the user is accustomed to gazing at the lower half of the application interface, if it is determined that the user's gaze point moves from the lower area to the upper area, it can be determined that the scrolling speed of the page is faster than the user's reading speed, and the electronic device can reduce the page scrolling speed. In this way, the electronic device can dynamically adjust the scrolling speed of the page according to the user's reading speed, so that the scrolling speed of the page matches the user's reading habits and reading speed, without the need for the user to manually adjust, which greatly improves the user's reading experience.

[0050] In addition, the electronic device sets the overlapping part between the upper area and the lower area as a transition area. When the user's gaze point moves from the upper grid (as an example of the area in the upper area other than the transition area) to the transition area, no change in the area where the user's gaze point is located is detected. When the user's gaze point moves from the lower grid (as an example of the area in the lower area other than the transition area) to the transition area, no change in the area where the user's gaze point is located is detected. The size of the transition area can affect the sensitivity of the gaze point switching between the upper and lower areas. The larger the transition area, the lower the sensitivity, and the smaller the transition area, the higher the sensitivity. Therefore, by setting an intermediate transition area, the electronic device can adjust the sensitivity of the gaze point switching between the upper and lower areas, and avoid the problem of frequent detection of the switching of the area where the user's gaze point is located when the user's gaze point jumps in the middle area, causing intention recognition errors.

[0051] In some embodiments, the method further comprises:

[0052] Determine the adjustment coefficient corresponding to the user;

[0053] A second scroll speed is determined based on the adjustment factor.

[0054] In some embodiments, the method further comprises:

[0055] In response to determining that the user's gaze point moves from the second display area to the first display area, and after a fifth preset time has passed, the page scrolling speed of the application interface of the second application is adjusted from the second scrolling speed to the first scrolling speed.

[0056] According to the implementation mode of the present application, in the case where the user is accustomed to looking at the upper part of the application interface, if it is determined that the user's gaze point has moved from the lower area to the upper area, it can be determined that the scrolling speed of the page is faster than the user's reading speed, and the electronic device can restore the default scrolling speed. In the case where the user is accustomed to looking at the lower part of the application interface, if it is determined that the user's gaze point has moved from the upper area to the lower area, it can be determined that the scrolling speed of the page is slower than the user's reading speed, and the electronic device can also restore the default scrolling speed. In this way, the electronic device can dynamically adjust the scrolling speed of the page according to the user's reading speed, so that the scrolling speed of the page matches the user's reading habits and reading speed, without the need for the user to manually adjust, which greatly improves the user's reading experience.

[0057] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device; a processor, when the processor executes the instructions in the memory, the electronic device can execute the page turning method provided by any embodiment of the first aspect or the second aspect of the present application. The beneficial effects that can be achieved in the third aspect can refer to the beneficial effects of any embodiment of the first aspect or the second aspect of the present application, and will not be repeated here.

[0058] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed on a computer, the computer executes the page turning method provided by any embodiment of the first aspect or the second aspect of the present application. The beneficial effects that can be achieved in the fourth aspect can refer to the beneficial effects of any embodiment of the first aspect or the second aspect of the present application, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of an application scenario provided for an implementation example of this application;

[0060] Figure 2a A schematic diagram of a setting interface for the air sliding screen function provided in some embodiments;

[0061] Figure 2b A schematic diagram of an interactive method of turning pages by air gestures provided in some embodiments;

[0062] Figure 2c and Figure 2d A schematic diagram of a setting interface for setting a scrolling speed provided in some embodiments;

[0063] Figure 3a A schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0064] Figure 3b A software structure block diagram of an electronic device provided in one embodiment of the present application;

[0065] Figure 3c A schematic diagram of the system architecture of an electronic device provided in one embodiment of the present application;

[0066] Figure 4 A flow chart of a page turning method provided for one embodiment of the present application;

[0067] Figure 5a A schematic diagram of an application interface of a reading APP provided in one embodiment of the present application;

[0068] Figure 5b A schematic diagram of a transition region provided for one embodiment of the present application;

[0069] Figure 5c A schematic diagram of an application interface of a reading APP provided in another embodiment of the present application;

[0070] Figure 5d A schematic diagram of an application interface of a reading APP provided in another embodiment of the present application;

[0071] Figure 5e A schematic diagram of an application interface of a reading APP provided in another embodiment of the present application;

[0072] Figure 6 A flow chart of a page turning method provided for a specific embodiment of the present application;

[0073] Figure 7a A schematic diagram of gaze point jump provided for an embodiment of the present application;

[0074] Figure 7b A schematic diagram of gaze point jump provided in another embodiment of the present application;

[0075] Figure 8 A flowchart of a process for triggering a fence event provided for a specific embodiment of the present application;

[0076] Fig. 9 A schematic diagram of changes in the display area where the user's gaze point is located provided in one embodiment of the present application;

[0077] Fig.10 A processing flow chart for opening a new page provided by an embodiment of the present application;

[0078] Fig.11 A processing flow chart for when a user switches from gazing at a lower area to gazing at an upper area provided in one embodiment of the present application;

[0079] Fig.12 A processing flow chart for when a user switches from gazing at an upper area to gazing at a lower area provided in one embodiment of the present application;

[0080] Fig.13 A flow chart of a page turning method provided for another embodiment of the present application;

[0081] Fig.14a A schematic diagram of a page turning process provided by an embodiment of the present application;

[0082] Fig.14b A schematic diagram of a page turning process of the upper and lower half screens provided by an embodiment of the present application;

[0083] Fig.15 A schematic diagram of switching a mobile phone from a portrait mode to a landscape mode provided in accordance with an embodiment of the present application;

[0084] Fig.16 A flow chart of a page turning method provided for another specific embodiment of the present application;

[0085] Fig.17 A flow chart of a page turning method provided for another embodiment of the present application;

[0086] Fig.18 A flow chart of a page turning method provided for another specific embodiment of the present application;

[0087] Fig.19 A schematic diagram of accelerated scrolling and page turning provided by an embodiment of the present application;

[0088] Fig. 20 A flow chart of a page turning method provided for another embodiment of the present application;

[0089] Fig.21 A flow chart of a page turning method provided for another specific embodiment of the present application;

[0090] Fig. 22 A schematic diagram of decelerating scrolling and page turning provided by an embodiment of the present application;

[0091] Fig.23 A block diagram of an electronic device provided by an embodiment of the present application is shown;

[0092] Fig.24 A block diagram of a system on chip (SOC) provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0093] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0094] The page turning method provided in the embodiment of the present application can be applied to the scene where the user reads the page content in the application interface of the reading software on the electronic device, and can control the page turning process of the application interface of the reading software, especially in the scene where it is inconvenient for the user to turn the page by hand, such as when the user is lying down, has dirty hands, or wears gloves.

[0095] In conjunction with the reference manual Figure 1 , Figure 1 The following is a schematic diagram of an application scenario provided by an implementation example of the present application. Figure 1As shown, in this scenario, the user 101 can read the page content in the application interface 102 of the reading software on the electronic device at a certain speed. When the user finishes reading the current page content, it is necessary to perform a page turning operation on the application interface 102 of the reading software. The reading software can be various types of APPs that provide information to users, for example, it can include but not limited to reading APPs, browser APPs, information flow APPs, etc., and the embodiments of the present application do not make specific restrictions on this.

[0096] In some embodiments, the user can set whether to enable the air sliding screen function in the setting interface of the electronic device. Figure 2a , Figure 2a A schematic diagram of a setting interface for the air sliding screen function provided by some embodiments is shown. Figure 2a As shown, the setting interface 201 of the air sliding screen function may include a control 202 for turning on the air sliding screen function. The electronic device may turn on the air sliding screen function in response to a user's click operation on the control 202.

[0097] When the air sliding screen function is turned on, the electronic device can use the air gesture page turning interaction method to realize page turning. Figure 2b , Figure 2b A schematic diagram of an interactive method of turning pages by air gestures provided in some embodiments is shown. Figure 2b As shown, when the air sliding screen function is turned on, the electronic device can recognize the user's gesture information based on machine vision technology, and when a user gesture indicating page turning is detected (such as Figure 2b When the downward swiping gesture 203 is performed in the air, a page turning operation is performed on the application interface 204 of the reading software.

[0098] Although this page turning method does not require the finger to touch the screen, it can solve the problem of inconvenient hand touch screen page turning. However, it still requires users to perform gesture operations frequently for a long time, which is easy to fatigue and requires one-handed operation, and does not really free the user's hand. In addition, the recognition speed and accuracy of this page turning method are low, there are certain recognition failures, the interaction process is relatively complicated, and the power consumption is also high.

[0099] In some embodiments, the electronic device can realize automatic page turning by using the interactive method of scrolling and turning pages. Specifically, the user can set the scrolling speed in the setting interface of the electronic device, and the electronic device can obtain the scrolling speed set by the user accordingly. Figure 2c and Figure 2d , Figure 2c and Figure 2d FIG. 4 is a schematic diagram showing a setting interface for setting the scroll speed provided in some embodiments. Figure 2cAs shown, the electronic device can provide the user with a setting interface 205 for setting the scrolling speed, and obtain the scrolling speed set by the user based on the setting interface 205. Or, as Figure 2d As shown, the electronic device can provide the user with a setting interface 206 for setting the scrolling speed, and obtain the scrolling speed set by the user based on the setting interface 206 .

[0100] After that, the electronic device can control the page to scroll down or up at a uniform speed to turn the page according to the scrolling speed set by the user. Since the scrolling speed is fixed, but the content density and difficulty of the page are variable, the user's reading speed is also variable. Therefore, the scrolling speed of the page cannot be well matched with the user's reading speed, resulting in a poor reading experience for the user in the scrolling page turning mode. In addition, the interactive mode of scrolling page turning is mainly applicable to the up and down scrolling page turning mode. In the simulated page turning and left and right covering page turning modes, there is a problem of early or delayed page turning, and the user experience is not good.

[0101] In some embodiments, the electronic device can collect user facial image information, determine the direction of the user's line of sight based on the collected user facial information, obtain the coordinates of the user's line of sight on the display screen based on the direction of the user's line of sight, and determine the display area where the user's line of sight is located based on the coordinates of the user's line of sight on the display screen. The electronic device can compare the display area where the user's current line of sight is located with the display area where the user's line of sight was located last time. When the area where the user's current line of sight is different from the area where the user's line of sight was located last time, the content displayed in the display area where the user's line of sight was located last time is refreshed to the next page of the page content displayed in the display area where the user's current line of sight is located, and the page includes the page of the electronic reader display area, and the next page content includes the content of the next page displayed in the electronic reader display area. This page turning method is mainly suitable for a page turning mode that divides the display screen into two parts and controls the two parts of the display screen to turn pages separately. It is not applicable to other page turning modes, and there is a problem of line of sight jump between the two display areas, which is easy to trigger page turning by mistake, and the user experience is not good.

[0102] To this end, an embodiment of the present application provides a page turning method to achieve automatic page turning and enhance the user's reading experience.

[0103] According to an embodiment of the present application, the electronic device can divide the application interface into an upper area (as an example of the first display area) and a lower area (as an example of the second display area), and can detect changes in the user's gaze point based on eye tracking technology. After determining that the user's gaze point moves from the upper area to the lower area, the electronic device can estimate the remaining reading time required for the user to finish reading the page content in the lower area as the page turning waiting time, and enter a waiting time window. After the waiting time window, when it is determined that the user's gaze point moves from the lower area to the upper area again, the electronic device can determine that the user's intention to turn the page is detected, and then perform an automatic page turning operation. In this way, when the user decides to turn the page after reading the page content in the lower area, he only needs to move the gaze point up to the upper area, and the page can be automatically turned without any mechanical movement of the user's hand, which greatly reduces fatigue and improves the user's reading experience. In addition, the electronic device recognizes the user's reading progress and the intention to turn the page on the current page by detecting changes in the user's gaze point, and has high recognition accuracy, good fault tolerance and strong applicability.

[0104] In addition, the electronic device sets the overlapping part between the upper area and the lower area as a transition area. When the user's gaze point moves from the upper grid (as an example of the area in the upper area other than the transition area) to the transition area, no change in the area where the user's gaze point is located is detected. When the user's gaze point moves from the lower grid (as an example of the area in the lower area other than the transition area) to the transition area, no change in the area where the user's gaze point is located is detected. The size of the transition area can affect the sensitivity of the gaze point switching between the upper and lower areas. The larger the transition area, the lower the sensitivity, and the smaller the transition area, the higher the sensitivity. Therefore, by setting an intermediate transition area, the electronic device can adjust the sensitivity of the gaze point switching between the upper and lower areas, and avoid the problem of frequent detection of the switching of the area where the user's gaze point is located when the user's gaze point jumps in the middle area, causing intention recognition errors.

[0105] According to another embodiment of the present application, after determining that the user's gaze point moves from the upper area to the lower area and after the waiting time window, the electronic device can determine that the user has finished reading the page content of the current page (i.e., read to the bottom of the page). At this time, the electronic device can give a page turning prompt to guide the user to look up and turn the page, so that the user can get a clear psychological hint that the intention has been accurately recognized, further improving the user's reading experience.

[0106] According to another embodiment of the present application, the electronic device may divide the application interface into an upper area and a lower area (as an example of a first display area and a second display area), and may detect changes in the user's gaze point based on eye tracking technology. In the case where the user is accustomed to looking at the upper half of the application interface, if it is determined that the user's gaze point moves from the upper area to the lower area, it can be determined that the scrolling speed of the page is slower than the user's reading speed, and the electronic device can increase the page scrolling speed. In the case where the user is accustomed to looking at the lower half of the application interface, if it is determined that the user's gaze point moves from the lower area to the upper area, it can be determined that the scrolling speed of the page is faster than the user's reading speed, and the electronic device can reduce the page scrolling speed. In this way, the electronic device can dynamically adjust the scrolling speed of the page according to the user's reading speed, so that the scrolling speed of the page matches the user's reading habits and reading speed, without the need for the user to manually adjust, which greatly improves the user's reading experience.

[0107] In the above embodiments, the electronic device may be a single screen, a folding screen, a curved screen, etc. Among them, the folding screen may be a structure with two physical screens, or a structure with one physical screen, which is not limited here. In other embodiments of the present application, the page turning method of the present application may also be applied to electronic devices with more than three screens, which is not limited here.

[0108] In the above embodiments, the application interface is divided in the upper and lower regions in the portrait mode of the electronic device. In other embodiments of the present application, the application interface can also be divided in the left and right regions in the landscape mode of the electronic device. In the left and right region division mode of the landscape mode of the electronic device, the left region on the left side of the display interface is the first display region, and the right region on the right side of the display interface is the second display region.

[0109] The embodiments of the present application do not limit the form of the electronic device. The electronic device may be a mobile phone, a tablet, a laptop computer, a smart screen, a vehicle-mounted device (for example, a vehicle computer, a vehicle-mounted navigation system), or other electronic device with a display screen, which is not limited here.

[0110] The page turning method of the embodiment of the present application is described below in conjunction with the specific structure of the electronic device.

[0111] Figure 3aThe structure diagram of an electronic device 300 provided by an embodiment of the present application is shown. The electronic device 300 may include a processor 310, an external memory interface 320, an internal memory 321, a Universal Serial Bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a mobile communication module 350, a wireless communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a button 390, a motor 391, an indicator 392, a camera 393, a display screen 394, and a Subscriber Identification Module (SIM) card interface 395, etc. The sensor module 380 may include a pressure sensor 380A, a gyroscope sensor 380B, an air pressure sensor 380C, a magnetic sensor 380D, an acceleration sensor 380E, a distance sensor 380F, a proximity light sensor 380G, a fingerprint sensor 380H, a temperature sensor 380J, a touch sensor 380K, an ambient light sensor 380L, a bone conduction sensor 380M, etc.

[0112] The processor 310 may include one or more processing units, for example: the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor ISP, a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor NPU, etc. Among them, different processing units can be independent devices or integrated into one or more processors. For example, the processor 310 can be used to execute the various steps of the page turning method in the embodiment of the present application.

[0113] The processor 310 can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0114] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory. The memory may store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0115] In some embodiments, the processor 310 can detect changes in the user's gaze point based on eye tracking technology. After determining that the user's gaze point has moved from the upper area to the lower area, the remaining reading time required for the user to finish reading the page content in the lower area can be estimated as the page turning waiting time, and a waiting time window is entered. After the waiting time window, when it is determined that the user's gaze point has moved from the lower area to the upper area again, it can be determined that the user's intention to turn the page is detected, and then the automatic page turning operation is performed. In this way, when the user decides to turn the page after reading the page content in the lower area, he only needs to move the gaze point up to the upper area, and the page can be automatically turned without any mechanical movement of the user's hand, which greatly reduces fatigue and improves the user's reading experience.

[0116] In some embodiments, the processor 310 can detect changes in the user's gaze point based on eye tracking technology. In the case where the user is accustomed to looking at the upper half of the application interface, if it is determined that the user's gaze point moves from the upper area to the lower area, it can be determined that the scrolling speed of the page is slower than the user's reading speed, and the page scrolling speed can be increased. In the case where the user is accustomed to looking at the lower half of the application interface, if it is determined that the user's gaze point moves from the lower area to the upper area, it can be determined that the scrolling speed of the page is faster than the user's reading speed, and the page scrolling speed can be reduced. In this way, the scrolling speed of the page can be dynamically adjusted according to the user's reading speed, so that the scrolling speed of the page matches the user's reading habits and reading speed, without the need for the user to manually adjust, which greatly improves the user's reading experience.

[0117] In some embodiments, the processor 310 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, and a Subscriber Identity Module (SIM) interface.

[0118] The MIPI interface can be used to connect the processor 310 with peripheral devices such as the display screen 394 and the camera 393. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 310 and the camera 393 communicate via the CSI interface to realize the shooting function of the electronic device 300. The processor 310 and the display screen 394 communicate via the DSI interface to realize the display function of the electronic device 300.

[0119] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 310 with the camera 393, the display screen 394, the wireless communication module 360, the audio module 370, the sensor module 380, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0120] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0121] The electronic device 300 implements the display function through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or change display information.

[0122] The display screen 394 is used to display images, videos, etc. The display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include 1 or N display screens 394, where N is a positive integer greater than 1.

[0123] The electronic device 300 can realize the shooting function through ISP, camera 393, video codec, GPU, display screen 394 and application processor.

[0124] The ISP is used to process the data fed back by the camera 393. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 393.

[0125] The camera 393 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 300 may include 1 or N cameras 393, where N is a positive integer greater than 1.

[0126] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 300 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0127] The internal memory 321 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 321 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 300 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 310 executes various functional applications and data processing of the electronic device 300 by running instructions stored in the internal memory 321, and / or instructions stored in a memory provided in the processor.

[0128] In some embodiments, instructions for a page turning method may be stored in the internal memory 321, and the processor 310 runs the page turning method instructions so that the processor 310 can detect changes in the user's gaze point based on eye tracking technology. After determining that the user's gaze point moves from the upper area to the lower area, the remaining reading time required for the user to finish reading the page content in the lower area can be estimated as the page turning waiting time, and a waiting time window is entered. After the waiting time window, when it is determined that the user's gaze point moves from the lower area to the upper area again, it can be determined that the user's intention to turn the page is detected, and then the automatic page turning operation is performed. In this way, when the user decides to turn the page after reading the page content in the lower area, he only needs to move the gaze point up to the upper area, and the page can be automatically turned without any mechanical movement of the user's hand, which greatly reduces fatigue and improves the user's reading experience.

[0129] In some embodiments, the processor 310 runs the page turning method instruction so that the processor 310 can detect changes in the user's gaze point based on eye tracking technology. In the case where the user is accustomed to looking at the upper half of the application interface, if it is determined that the user's gaze point moves from the upper area to the lower area, it can be determined that the scrolling speed of the page is slower than the user's reading speed, and the page scrolling speed can be increased at this time. In the case where the user is accustomed to looking at the lower half of the application interface, if it is determined that the user's gaze point moves from the lower area to the upper area, it can be determined that the scrolling speed of the page is faster than the user's reading speed, and the page scrolling speed can be reduced at this time. In this way, the scrolling speed of the page can be dynamically adjusted according to the user's reading speed, so that the scrolling speed of the page matches the user's reading habits and reading speed, without the need for manual adjustment by the user, which greatly improves the user's reading experience.

[0130] The pressure sensor 380A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 380A can be set on the display screen 394. There are many types of pressure sensors 380A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 380A, the capacitance between the electrodes changes. The electronic device 300 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 394, the electronic device 300 detects the touch operation intensity according to the pressure sensor 380A. The electronic device 300 can also calculate the touch position according to the detection signal of the pressure sensor 380A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0131] The gyro sensor 380B can be used to determine the motion posture of the electronic device 300. In some embodiments, the angular velocity of the electronic device 300 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 380B. The gyro sensor 380B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 380B detects the angle of the electronic device 300 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 300 through reverse movement to achieve anti-shake. The gyro sensor 380B can also be used for navigation and somatosensory game scenes.

[0132] In some embodiments, when the user rotates the electronic device 300, the gyroscope sensor 380B can determine the motion posture of the electronic device 300, so that the processor 310 determines whether the electronic device 300 is in a horizontal screen state or a vertical screen state according to the data provided by the gyroscope sensor 380B.

[0133] The distance sensor 380F is used to measure the distance. The electronic device 300 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 300 can use the distance sensor 380F to measure the distance to achieve fast focusing.

[0134] The ambient light sensor 380L is used to sense the ambient light brightness. The electronic device 300 can adaptively adjust the brightness of the display screen 394 according to the perceived ambient light brightness. The ambient light sensor 380L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 380L can also cooperate with the proximity light sensor 380G to detect whether the electronic device 300 is in a pocket to prevent accidental touch.

[0135] The touch sensor 380K is also called a "touch control device". The touch sensor 380K can be set on the display screen 394, and the touch sensor 380K and the display screen 394 form a touch screen, also called a "touch control screen". The touch sensor 380K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 394. In other embodiments, the touch sensor 380K can also be set on the surface of the electronic device 300, which is different from the position of the display screen 394.

[0136] The software system of the electronic device 300 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 300. Among them, the present application does not limit the type of operating system of the electronic device. For example, Android system, Linux system, Windows system, iOS system, Harmony Operating System (Harmony OS), etc.

[0137] Figure 3b It is a software structure block diagram of an electronic device 300 provided in one embodiment of the present application.

[0138] like Figure 3bAs shown, the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer (APP), the application framework layer (APP Framework), the Android runtime (Android Runtime) and the system library (Libraries), and the kernel layer (Kernel).

[0139] The application layer can include a series of application packages.

[0140] like Figure 3b As shown, the application package can include gallery, map, navigation, music, video, game, chat, shopping, reading and other application apps. The application layer can also include Figure 3b Other application APPs besides the applications shown, such as camera, calendar, call, WLAN, Bluetooth, short message, etc., are not described one by one in the embodiments of the present application.

[0141] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0142] like Figure 3b As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0143] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0144] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0145] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0146] The phone manager is used to provide communication functions of the electronic device 300, such as management of call status (including connecting, hanging up, etc.).

[0147] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0148] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0149] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system.

[0150] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0151] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0152] The system library may include multiple functional modules, such as: Surface Manager, Media Libraries, 3D graphics processing library (such as OpenGLES), 2D graphics engine (such as SGL), gaze point interaction engine, etc.

[0153] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0154] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0155] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc.

[0156] A 2D graphics engine is a drawing engine for 2D drawings.

[0157] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0158] Combine the following Figure 3c The system architecture adopted by the electronic device 300 according to the embodiment of the present application is explained.

[0159] In conjunction with the reference manual Figure 3c , Figure 3c A schematic diagram of the system architecture of an electronic device provided by an embodiment of the present application is shown. Figure 3c As shown, the system architecture adopted by the electronic device 300 of the embodiment of the present application may include an image sensor 301, a gaze point interaction engine 302, a gaze point interaction algorithm module 303, an artificial intelligence (AI) chip 304, an application APP 305, a gaze intention processing module 306 and a reader module 307.

[0160] The image sensor 301 may be a front-mounted low-power camera for capturing and outputting facial images of the user.

[0161] The gaze point interaction engine 302 is an execution software module for gaze point interaction, which is used to receive the facial image collected by the image sensor 301, and input the facial image and related parameters into the gaze point interaction algorithm module 303 for reasoning to identify the user's gaze point.

[0162] The gaze point interaction algorithm module 303 may include a pre-trained inference model for identifying gaze points, which can run in the AI ​​chip 304 and the inference container, and consume system computing power to infer the input facial image to identify the user's gaze point.

[0163] The AI ​​chip 304 can be integrated on the main board of the electronic device 300 to provide AI computing power and operators.

[0164] Application APP 305 is an APP running on the electronic device 300, which can mainly include reading APPs such as news information, e-books, magazines and comics. The main adaptability of this type of APP is that the entire screen area can be used as a reading interface, and the graphics and text are laid out from top to bottom on the screen according to natural layout rules. The graphics and text parts beyond the screen area support left and right or up and down page turning operations. It should be noted that in some possible embodiments, the application APP can also include other types of APPs, such as browser APPs, information flow APPs, etc., and the embodiments of this application do not make specific restrictions on this.

[0165] The gaze intention processing module 306 is a software module for gaze point intention processing running in a reading APP. This module can assist in registering and monitoring events with the gaze point interaction engine 302 on the APP's external operating system; it can also assist in calculating the amount of information on the reading page, the reading speed, and executing page turning with the reader module 307 within the APP.

[0166] The reader module 307 runs in the reading APP, provides typesetting and rendering of the reading page content, and provides operations on the reading page such as turning pages. The reader module 307 can also open APIs to the outside to query parameters related to the reading page, such as the page information query interface, the page type mixed page turning execution interface, etc.

[0167] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 300. In other embodiments of the present application, the electronic device 300 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0168] The specific process of the page turning method provided in the embodiment of the present application is described in detail below.

[0169] The page turning method provided in the embodiment of the present application can be applied to Figure 3a The hardware structure shown in Figure 3b In an electronic device with the software structure shown in the figure, or with more or fewer components than shown in the figure, or with some components combined, or some components separated, or with different component arrangements, etc., and with similar hardware structures and software structures.

[0170] The following uses a mobile phone as an example to describe in detail a page turning method provided by an embodiment of the present application. The mobile phone may be in a vertical screen display state, and multiple APPs are installed in the mobile phone, some or all of which are APPs that provide information to users. Figure 4 As shown, Figure 4 A flow chart of a page turning method provided by an embodiment of the present application is shown, and the method may include steps S401-S405.

[0171] S401: The mobile phone displays an application interface of a reading APP (as an example of a first application), which includes an upper area (as an example of a first display area) and a lower area (as an example of a second display area), and the upper area and the lower area partially overlap or do not overlap at all.

[0172] In the embodiment of the present application, the first application can be any application installed in the mobile phone that can provide information to the user. The first application can be various types of applications, including but not limited to reading apps, browser apps, information flow apps, etc. The embodiment of the present application takes a reading app as the first application as an example for illustration, but the present application is not limited thereto. The reading app can be an app with page turning modes such as left and right page turning, simulated page turning, overlay page turning, or upper and lower half screen page turning.

[0173] In the embodiment of the present application, the upper area and the lower area together constitute the application interface of the reading APP. The upper area and the lower area may partially overlap or not overlap at all. The application interface of the reading APP may display a page of the reading APP, such as the homepage of the reading APP, or the reading page of a book in the reading APP, and the present application does not make any specific restrictions on this.

[0174] In a possible embodiment, in conjunction with the reference specification, Figure 5a , Figure 5a A schematic diagram of an application interface of a reading APP provided by an embodiment of the present application is shown. Figure 5a As shown, when the mobile phone is in the vertical screen display state, the upper area of ​​the application interface 510 may include an upper grid 501 and a transition area 502, and the lower area of ​​the application interface 510 may include a transition area 502 and a lower grid 503. In other words, the upper area and the lower area of ​​the application interface 510 may partially overlap, and the overlapping area is the transition area 502.

[0175] The transition area 502 may be set in the middle of the application interface, and the size of the transition area 502 may affect the sensitivity of the user's gaze point switching between the upper and lower areas. Specifically, the larger the transition area 502, the lower the sensitivity, and the smaller the transition area 502, the greater the sensitivity.

[0176] For example, in conjunction with the reference specification Figure 5b , Figure 5b FIG. 5 shows a schematic diagram of a transition region 502 provided by an embodiment of the present application. Figure 5b As shown, the transition area 502 may include an area in the middle of the application interface 510 with a width of [Xa, X+b], where X represents the position of half the height of the application interface 510. The size of the transition area 502 may be determined according to the values ​​of a and b, and the values ​​of a and b may be determined according to actual needs, for example, they may be input by a reading APP when registering a gaze point switching fence event or may be preset by the system, and the embodiment of the present application does not impose any specific limitation on this.

[0177] In another possible embodiment, in conjunction with the reference specification, Figure 5c , Figure 5cFIG. 1 is a schematic diagram showing an application interface of a reading APP provided by another embodiment of the present application. Figure 5c As shown, the upper area of ​​the application interface 510 may include only the upper grid 501, and the lower area of ​​the application interface 510 may include only the lower grid 503. In other words, the upper area and the lower area of ​​the application interface 510 may not overlap at all.

[0178] It should be noted that in the above embodiments, the display mode of the mobile phone is in the vertical screen state. In other embodiments of the present application, it can also be displayed in the horizontal screen state. When the mobile phone is displayed in the horizontal screen state, the area above the application interface is the first display area, and the area below the application interface is the second display area. Alternatively, when the mobile phone is displayed in the horizontal screen state, the area on the left side of the application interface is the first display area, and the area on the right side of the application interface is the second display area.

[0179] In one possible embodiment, Figure 5d FIG. 1 is a schematic diagram showing an application interface of a reading APP provided by another embodiment of the present application. Figure 5d As shown, when the mobile phone is in the horizontal screen display state, the positional relationship between the two areas of the application interface can remain unchanged, and is still in an upper and lower relationship. That is, the application interface 520 can include an upper area and a lower area, the upper area of ​​the application interface 520 can include an upper grid 504 and a transition area 505, and the lower area of ​​the application interface 520 can include a transition area 505 and a lower grid 506. In other words, the upper area and the lower area of ​​the application interface 520 can partially overlap, and the overlapping area is the transition area 505.

[0180] In another possible embodiment, when the mobile phone is in the horizontal display state, the upper area of ​​the application interface 520 may also include only the upper grid 504, and the lower area of ​​the application interface 520 may also include only the right grid 506. In other words, the upper area and the lower area of ​​the application interface 520 may not overlap at all.

[0181] In another possible embodiment, Figure 5e FIG. 1 is a schematic diagram showing an application interface of a reading APP provided by another embodiment of the present application. Figure 5e As shown, when the mobile phone is in the horizontal display state, the positional relationship between the two areas of the application interface can be changed to a left-right relationship. That is, the application interface 520 can include a left area (as another example of the first display area) and a right area (as another example of the second display area). The left area of ​​the application interface 520 can include a left grid 507 and a transition area 508, and the right area of ​​the application interface 520 can include a transition area 508 and a right grid 509. In other words, the left area and the right area of ​​the application interface 520 can partially overlap, and the overlapping area is the transition area 508.

[0182] In another possible embodiment, when the mobile phone is in a horizontal display state, the left area of ​​the application interface 520 may also include only the left grid 507, and the right area of ​​the application interface 520 may also include only the right grid 509. In other words, the left area and the right area of ​​the application interface 520 may not overlap at all.

[0183] S402: The mobile phone determines that the user's gaze point relative to the application interface moves from the upper area to the lower area.

[0184] In the embodiment of the present application, the mobile phone can obtain the user's facial image in real time, and perform reasoning based on the user's facial image to identify the user's gaze point relative to the application interface and determine the display area where the gaze point is located, thereby determining whether the user's gaze point has moved from the upper area to the lower area. Wherein, when it is determined that the display area where the user's gaze point is currently located is the lower area, and it is determined that the display area where the user's gaze point was previously located is the upper area, it can be determined that the user's gaze point has moved from the upper area to the lower area.

[0185] Specifically, the mobile phone can recognize the face and eyes based on the reasoning of the gaze point interaction algorithm model, and determine the relationship between the user's gaze point and the display area, and detect the switching of the user's gaze point between the upper and lower areas in combination with the display area where the user's gaze point was last located. Among them, the gaze point interaction algorithm model can be predetermined according to actual needs, and the embodiments of the present application do not impose specific restrictions on this.

[0186] S403: In response to determining that the user's gaze point relative to the application interface moves from the upper area to the lower area, the mobile phone determines a page turning waiting time of the application interface of the reading APP.

[0187] In the embodiment of the present application, for the reading scenario, the areas where the intention needs to be mainly identified include the top of the page (representing the beginning) and the bottom of the page (representing the completion), and there is no actual intention in the middle area. Considering that when a user reads, they usually read the content of the lower half of the page after reading the content of the upper half of the page, so when the mobile phone determines that the user is reading the content of the lower half of the page, it can dynamically estimate the time required for the user to finish reading the content of the lower half of the page according to the user's real-time reading speed, that is, estimate the page turning waiting time. The mobile phone can judge the reading progress based on the judgment of the reading time. When the user's reading page turning waiting time is determined, it can be determined that the user has finished reading the content of the lower half of the page (that is, read to the bottom of the page).

[0188] Specifically, after the mobile phone determines that the user's gaze point has moved from the upper area to the lower area, it can be determined that the user has switched from reading the page content in the upper area to reading the page content in the lower area. At this time, the remaining reading time required to finish reading the page content in the lower area can be estimated based on the page reading speed of the user reading the page content in the upper area, and then the page turning waiting time can be determined. The specific process of determining the page turning waiting time will be described in detail later.

[0189] S404: In response to determining that the user's gaze point moves from the upper area to the lower area, the mobile phone determines that the user's gaze point moves from the lower area to the upper area after a page turning waiting time.

[0190] In the embodiment of the present application, in response to determining that the user's gaze point has moved from the upper area to the lower area, the mobile phone can determine the reading progress, thereby determining whether the user has finished reading all the contents of the page (i.e., read to the bottom of the page). Specifically, the mobile phone can determine whether the current waiting time t reaches the page turning waiting time T after determining that the user's gaze point has moved from the upper area to the lower area. If so, it can be determined that the user has finished reading all the contents of the page. If not, it can be determined that the user has not finished reading all the contents of the page, and continues to wait.

[0191] In the embodiment of the present application, after determining that the user has finished reading all the contents of the page, the mobile phone can judge the intention of turning the page, so as to automatically turn the page when recognizing the intention of the user to turn the page. Specifically, the mobile phone can determine whether the user's gaze point moves from the lower area to the upper area. If so, it can be determined that the user's intention to turn the page is recognized. If not, it can continue to wait.

[0192] Specifically, the specific content of detecting that the user's gaze point moves from the lower area to the upper area can refer to the relevant content of detecting that the user's gaze point moves from the upper area to the lower area in step S402, and the embodiments of the present application will not be repeated here. Among them, when it is determined that the display area where the user's gaze point is currently located is the upper area, and it is determined that the display area where the user's gaze point was previously located is the lower area, it can be determined that the user's gaze point moves from the lower area to the upper area.

[0193] S405: In response to determining that the user's gaze point moves from the lower area to the upper area, the mobile phone performs a page turning operation on the application interface of the reading APP.

[0194] In the embodiment of the present application, when the mobile phone determines that the user's gaze point moves from the lower area to the upper area, it can be determined that the user's intention to turn the page is recognized, and the page can be automatically turned. Specifically, the mobile phone can perform a page turning operation to display the page content of the next page of the current page in the application interface of the reading APP.

[0195] It is understandable that the page turning method provided in the embodiment of the present application can be used for reading apps with multiple page turning modes, for example, including but not limited to left and right page turning, simulated page turning, overlay page turning, and half-screen page turning. For reading apps with different page turning modes, the page turning operations performed may be different.

[0196] For example, for a reading app with a left-right page turning mode, you can perform a left-right page turning action to display the page content of the next page of the current page in the application interface of the reading app. For a reading app with a simulated page turning mode, you can perform a simulated page turning action to display the page content of the next page of the current page in the application interface of the reading app. For a reading app with an overlay page turning mode, you can perform an overlay page turning action to overwrite the page content of the current page with the page content of the next page of the current page. For a reading app with an upper and lower half-screen page turning mode, you can perform a half-screen flip overlay page turning action to display the upper half of the page content of the next page of the current page on the upper half of the page, and display the lower half of the page content of the next page of the current page on the lower half of the page.

[0197] It should be noted that the details of executing the page turning operation on the application interface of the reading APP are not repeated here in the embodiment of the present application.

[0198] In one possible embodiment, executing a page-turning operation on an application interface of a reading APP may include: in response to determining that the user's gaze point moves from a lower area to an upper area, after 200ms (as an example of a third preset time), executing a page-turning operation on the application interface of the reading APP.

[0199] In a possible embodiment, the page turning method may further include: within 200ms, the mobile phone, in response to determining that the user's gaze point moves from the upper area to the lower area, abandons the page turning operation on the application interface of the reading APP.

[0200] In the embodiment of the present application, due to the following recognition result jump scenario: the user gazes at the lower area of ​​the application interface, and after determining that the user's gaze point has moved from the upper area to the lower area, the current waiting time t has reached the page turning waiting time T, the user does not look up but the algorithm recognition error occurs, resulting in the user's gaze point jumping from bottom to top and then down. In this scenario, the page turning will be triggered by mistake, affecting the user experience. Therefore, in order to avoid this jump scenario, a page turning delay of 200ms can be added.

[0201] If the user's gaze does not move from the upper area to the lower area within the delay time, it can be considered that the user's intention to turn the page is real, and the page turning operation can be performed. If the user's gaze moves from the upper area to the lower area again within the delay time, it can be considered that the user's intention to turn the page is a misidentification, and the page turning operation is not performed.

[0202] It should be noted that the above example of using 200 ms as the third preset time is only an exemplary implementation. In actual applications, the third preset time can also be set to other values, and the embodiment of the present application does not impose specific limitations on this.

[0203] The following further describes the page turning method provided by the embodiment of the present application in combination with the system architecture of the mobile phone and the modules therein. Figure 6 As shown, Figure 6 A flowchart of a page turning method provided by a specific embodiment of the present application is shown. Figure 4 To further illustrate the illustrated embodiment, the method may include steps S601 - S621 .

[0204] S601: The gaze intention processing module may register a gaze point switching fence event with the gaze point interaction engine.

[0205] Specifically, when the user is using a reading APP, the reader module can display the application interface of the reading APP. At this time, the gaze intention processing module can register the gaze fences of the two display areas of the application interface and the user gaze point switching fence event. Specifically, the upper area gaze fence and the lower area gaze fence can be registered, as well as the fence event of the user's gaze point switching from the upper area gaze fence to the lower area gaze fence and the fence event of the user's gaze point switching from the lower area gaze fence to the upper area gaze fence.

[0206] S602: When the user gazes at the upper area of ​​the application interface, the image sensor can collect images and output frames in real time to obtain a facial image of the user.

[0207] Specifically, the image sensor may be a front-mounted low-power camera of a mobile phone, for example, a low-power camera with a frame rate of 10fps to 20fps. The low-power camera may run on a security chip to ensure the privacy and security of the user.

[0208] It is understandable that by adopting a low-power camera, the power consumption is increased by no more than 5% compared to manual page turning, and the recognition accuracy of the area where the user's gaze point is located is high (greater than 95%), which can effectively solve the power consumption problem based on image recognition technology. Even when reading for a long time, the impact on the battery life of the mobile phone is very small.

[0209] It should be noted that, in actual applications, the low power consumption mode of the front camera of the mobile phone can also be directly turned on to implement the page turning method of the embodiment of the present application, and the embodiment of the present application does not impose any specific restrictions on this.

[0210] S603: The image sensor may input the collected facial image into the gaze point interaction engine.

[0211] Accordingly, the gaze interaction engine can receive the facial image of the user sent by the image sensor.

[0212] S604: The gaze point interaction engine may input the user's facial image into the gaze point interaction algorithm module for processing.

[0213] Specifically, after the gaze point interaction engine receives the user's facial image, it can input the facial image and related parameters (such as the position and angle of the image sensor relative to the user's eyes) into the gaze point interaction algorithm module.

[0214] S605: The gaze point interaction algorithm module may perform gaze point recognition processing on the facial image to obtain the user's gaze point coordinates relative to the application interface.

[0215] Specifically, the gaze point interaction algorithm module can recognize human faces and human eyes based on the reasoning of the gaze point interaction algorithm model. The specific recognition method will be described in detail later.

[0216] S606: The gaze point interaction algorithm module can make fence entry and exit judgments based on the gaze point coordinates to determine whether to trigger a fence event in which the user's gaze point switches from the upper area to the lower area (i.e., determine that the user's gaze point moves from the upper area to the lower area).

[0217] Specifically, the gaze point interaction algorithm module can determine the relationship between the identified user gaze point and the display area, and detect the switching of the user gaze point between the upper and lower areas in combination with the display area where the user's gaze point was last located. Wherein, when it is determined that the display area where the user's gaze point is currently located is the lower area, and when it is determined that the display area where the user's gaze point was last located is the upper area, it can be determined that the user's gaze point has moved from the upper area to the lower area. When it is determined that the display area where the user's gaze point is currently located is the upper area, and when it is determined that the display area where the user's gaze point was last located is the lower area, it can be determined that the user's gaze point has moved from the lower area to the upper area.

[0218] Specifically, when the gaze point interaction algorithm module determines that the current mobile phone is in a vertical display state, the gaze point interaction algorithm in the upper and lower area recognition mode can be used to identify the user's gaze point and determine the display area of ​​the application interface where the gaze point is located. In the upper and lower area recognition mode, the mobile phone can directly determine whether the display area of ​​the application interface where the user's gaze point is located is the upper area or the lower area of ​​the display interface using the gaze point interaction algorithm.

[0219] S607: When determining a fence event that triggers the user's gaze point to switch from the upper area gaze fence to the lower area gaze fence, the gaze point interaction algorithm module may also report the fence event to the gaze point interaction engine.

[0220] Accordingly, the gaze point interaction engine may receive a fence event in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence.

[0221] S608: The gaze point interaction engine may report a fence event of the user's gaze point switching from the upper area gaze fence to the lower area gaze fence to the gaze intention processing module.

[0222] Accordingly, the gaze intention processing module may receive a fence event in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence.

[0223] It should be noted that step S602 to step S608 correspond to Figure 4 Step S402 in the illustrated embodiment.

[0224] S609: The gaze intention processing module may calculate the page reading speed and the page turning waiting time in response to the fence event in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence.

[0225] Specifically, the gaze intention processing module can estimate the remaining reading time required to finish reading the page content in the lower area according to the page reading speed of the user reading the page content in the upper area, and then determine the page turning waiting time. The method for determining the page turning waiting time will be described in detail later.

[0226] It should be noted that step S609 corresponds to Figure 4 Step S403 in the illustrated embodiment.

[0227] S610: The gaze intention processing module may determine the reading progress in response to a fence event in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence.

[0228] Specifically, after receiving a fence event in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence, the gaze intention processing module can also judge the reading progress based on the calculated page turning waiting time, thereby determining whether the user has finished reading all the contents of the page.

[0229] S611: When the user gazes at the lower area of ​​the application interface, the image sensor can collect images and output frames in real time to obtain a facial image of the user.

[0230] S612: The image sensor may input the collected facial image into the gaze point interaction engine.

[0231] Accordingly, the gaze interaction engine can receive the facial image of the user sent by the image sensor.

[0232] S613: The gaze point interaction engine may input the user's facial image into the gaze point interaction algorithm module for processing.

[0233] S614: The gaze point interaction algorithm module can perform gaze point recognition processing on the facial image to obtain the user's gaze point coordinates relative to the application interface.

[0234] S615: The gaze point interaction algorithm module can make fence entry and exit judgments based on the gaze point coordinates to determine whether to trigger a fence event in which the user's gaze point switches from the lower area to the upper area (i.e., determine that the user's gaze point moves from the lower area to the upper area).

[0235] S616: When determining a fence event that triggers the user's gaze point to switch from the lower area gaze fence to the upper area gaze fence, the gaze point interaction algorithm module may also report the fence event to the gaze point interaction engine.

[0236] Accordingly, the gaze point interaction engine may receive a fence event in which the user's gaze point switches from the lower area gaze fence to the upper area gaze fence.

[0237] It should be noted that the specific contents of steps S611 to S616 can refer to the relevant contents of steps S602 to S607, and the embodiments of the present application will not be repeated here.

[0238] S617: The gaze point interaction engine may also report a fence event of the user's gaze point switching from the lower area gaze fence to the upper area gaze fence to the gaze intention processing module.

[0239] Accordingly, the gaze intention processing module may receive a fence event in which the user's gaze point switches from the lower area gaze fence to the upper area gaze fence.

[0240] S618: The gaze intention processing module may determine the intention to turn the page.

[0241] Specifically, after determining that the user has finished reading all the contents of the page and receiving a fence event in which the user's gaze point switches from the lower area gaze fence to the upper area gaze fence, the gaze intention processing module can determine and recognize the user's intention to turn the page.

[0242] In actual applications, during the process of judging the intention to turn a page, the gaze intention processing module may enter a waiting time window of a page turning waiting time T after receiving a fence event in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence. After the waiting time window (i.e., after the current waiting time t reaches the page turning waiting time T), the judgment of the intention to turn a page begins, and the page turning action is performed in response to the fence event in which the user's gaze point switches from the lower area gaze fence to the upper area gaze fence, so as to reduce the impact of the gaze point jump. Within the waiting time window (i.e., before the current waiting time t reaches the page turning waiting time T), even if a fence event in which the user's gaze point switches from the lower area gaze fence to the upper area gaze fence is received, the page turning is not performed.

[0243] It should be noted that step S610 to step S618 correspond to Figure 4 Step S404 in the illustrated embodiment.

[0244] S619: When the gaze intention processing module receives a fence event in which the user's gaze point switches from the lower area gaze fence to the upper area gaze fence, the gaze intention processing module may perform event smoothing suppression.

[0245] Specifically, after receiving the fence event that the user's gaze point switches from the lower area gaze fence to the upper area gaze fence, the gaze intention processing module can enter a 200ms delayed execution time window. If no fence event that the user's gaze point switches from the upper area gaze fence to the lower area gaze fence is received in the delayed execution time window, it is considered that the user's intention to turn the page is real.

[0246] Specifically, in the delayed execution time window, if a fence event is received again in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence, it can be considered that the fence event of the user's gaze point switching from the lower area gaze fence to the upper area gaze fence received last time is a misidentification and needs to be suppressed. In other words, the page turning operation for the application interface of the reading APP can be abandoned, and the next fence event of the user's gaze point switching from the lower area gaze fence to the upper area gaze fence is waited for.

[0247] S620: If it is determined that the user's intention to turn the page is real, the gaze intention processing module may send a page turning instruction to the reader module.

[0248] Accordingly, the reader module can receive the page turning instruction sent by the gaze intention processing module.

[0249] S621: The reader module may execute a page turning operation on the application interface of the reading APP in response to a page turning instruction.

[0250] It should be noted that step S619 to step S621 correspond to Figure 4 Step S405 in the illustrated embodiment. Other contents in steps S601 to S621 can be referred to Figure 4 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0251] In actual applications, since the embodiment of the present application uses a low-power camera, it is limited by its sampling rate, image clarity, and the position of the camera, the angle between the camera and the human eye, the user's eye characteristics, light, and the power consumption of the gaze point interaction algorithm. The accuracy of gaze point recognition cannot be achieved at the pixel level, and the recognition accuracy deviation for some areas of the screen is large. In areas with large deviations, the gaze point recognition algorithm will have a sudden jump in the recognition coordinates, and abnormal jump events will interfere with the recognition of the user's formal intention.

[0252] For example, in conjunction with the reference specification Figure 7a , Figure 7a FIG. 2 shows a schematic diagram of a gaze point jump provided by an embodiment of the present application. Figure 7a As shown in FIG. 1 , when the user gazes at the upper area, some gaze points may jump (i.e., within a time window, the user's gaze points will be identified as mainly concentrated in the upper area, but some individual gaze points will be identified as being in the lower area). Figure 7b , Figure 7b FIG. 2 shows a schematic diagram of a gaze point jump provided by another embodiment of the present application. Figure 7b As shown, similar to the upper area, when the user is gazing at the lower area, due to the constraint of recognition accuracy, some gaze points may still jump (i.e., within a time window, the user's gaze points will be recognized as mainly concentrated in the lower area, but some individual gaze points will also be recognized as being in the upper area).

[0253] To solve the problem of gaze point jump, in a possible embodiment, a transition area can be set in the middle of the application interface, and the upper area and the lower area of ​​the application interface both contain the transition area. When the user's gaze point falls on the middle transition area, the gaze point interaction algorithm module and the gaze point interaction engine will suppress the triggering of the gaze point switching fence event. In other words, Figure 5a As shown, when it is determined that the user's gaze point falls on the middle transition area 502, no matter whether the user's gaze point last fell on the upper grid 501 or the lower grid 503 (it can be the left grid or the right grid when the mobile phone is in landscape display mode), the gaze point switching fence event will not be triggered.

[0254] In addition to setting the transition area, in a possible embodiment, multi-frame smoothing processing can also be performed when determining the coordinates of the gaze point. That is, the display area where the user's gaze point is currently located and the display area where it was previously located can both be determined based on at least one frame of the user's facial image. Specifically, Figure 6 As shown, before step S606, step S605' may also be performed first (in Figure 6 ), and / or, before step S615, step S614′ (in Figure 6 In other words, the gaze point interaction algorithm module can weight the gaze point coordinates obtained by the recognition in combination with at least one historical gaze point coordinate to obtain the final gaze point coordinates. The historical gaze point coordinates may be the coordinates of the gaze point obtained by the recognition based on the historical frame data, and the historical frame data may be the facial image collected before the current frame data.

[0255] In a possible embodiment, when identifying the user's gaze point, the gaze point interaction algorithm module may also directly use the continuously acquired multi-frame image recognition to obtain multiple gaze point coordinates, and then perform weighted analysis to obtain the final gaze point coordinates.

[0256] Below Figure 6 The process of gaze point recognition and fence entry and exit judgment from step S602 to step S607 in the illustrated embodiment is further described in detail. Figure 8 As shown, Figure 8 A flowchart of a process for triggering a fence event provided by a specific embodiment of the present application is shown, and the process may include steps S801-S821.

[0257] like Figure 8As shown, in step S801, it can be determined whether the screen of the mobile phone is off. In step S802, if it is determined that the screen is not off, it can be determined whether the current light intensity is higher than the preset light intensity threshold. If it is determined that the screen is off, the entire process ends. In step S803, if it is determined that it is higher than the preset light threshold, the image sensor collects the current frame data and inputs it into the gaze point interaction engine. If it is determined that it is not higher than the preset light threshold, the entire process ends. Among them, the preset light intensity threshold can be pre-set according to actual conditions, for example, it can be set to 5lux, etc., and the embodiment of the present application does not impose specific restrictions on this. Steps S801 to S802 correspond to Figure 6 Steps S602 to S603 in the illustrated embodiment.

[0258] In step S804, the gaze point interaction engine may input the received current frame data into the gaze point interaction algorithm module for image processing before face recognition. In step S805, the gaze point interaction algorithm module may perform face recognition on the image after the image processing before face recognition. In step S806, the gaze point interaction algorithm module may determine whether the face is valid based on the face recognition result obtained. If it is determined to be invalid, the current frame data may be determined to be an invalid frame, and the entire process is terminated. In step S807, if it is determined to be valid, the gaze point interaction algorithm module may perform image processing before face orientation detection on the face recognition result. In step S808, the gaze point interaction algorithm module may perform face orientation detection on the image after the image processing before face orientation detection. In step S809, the gaze point interaction algorithm module may determine whether the face is facing the screen based on the face orientation detection result obtained by the detection. If it is determined that the face is not facing the screen, the current frame data may be determined to be an invalid frame, and the entire process is terminated.

[0259] In step S810, if it is determined that the face is facing the screen, the gaze point interaction algorithm module can perform image processing before visual detection on the face recognition result. In step S811, the gaze point interaction algorithm module can perform visual detection on the image after the image processing before visual detection. In step S812, the gaze point interaction algorithm module can determine whether the user is looking at the screen based on the visual detection result obtained by the detection. If it is determined that the user is not looking at the screen, it can be determined that the current frame data is an invalid frame, and the whole process ends. In step S813, if it is determined that the user is looking at the screen, the gaze point interaction algorithm module can perform image processing before facial key point detection on the face recognition result. In step S814, the gaze point interaction algorithm module can perform facial key point detection on the image after the image processing before facial key point detection. In step S815, the gaze point interaction algorithm module can determine whether the facial key points are valid based on the facial key point detection results obtained by the detection. If it is determined to be invalid, it can be determined that the current frame data is an invalid frame, and the whole process ends. In step S816, if it is determined to be valid, the gaze point interaction algorithm module can perform gaze point prediction on the image after the gaze prediction image processing. In step S817, the gaze point interaction algorithm module can perform gaze point prediction on the image after the gaze prediction image processing.

[0260] In step S818, the gaze point interaction algorithm module can also determine whether the distance between the user and the mobile phone screen is less than a preset distance threshold. If it is judged to be greater than or equal to the preset distance threshold, it can be determined that the current frame data is an invalid frame, and the entire process ends. In step S819, if it is judged to be less than the preset distance threshold, the gaze point interaction algorithm module can weight the predicted gaze point coordinates in combination with the historical gaze point coordinates corresponding to the historical frame data to obtain the final gaze point coordinates. Among them, the preset distance threshold can be pre-set according to actual needs, for example, it can be set to 50cm, etc., and the embodiment of the present application does not impose specific restrictions on this. Steps S804 to S819 correspond to Figure 6 Steps S604 to S605' in the illustrated embodiment.

[0261] It should be noted that in the embodiment of the present application, the distance between the user and the mobile phone screen can be 0 to 50 cm, and the optimal distance can be 35 cm, but is not limited to this.

[0262] In step S820, the gaze point interaction algorithm module can also determine the relationship between the final gaze point coordinates and the upper area and the lower area, that is, determine whether the display area where the user's gaze point is currently located is the upper area or the lower area. Then, it can be determined whether it needs to be reported in combination with the display area where the user's gaze point was last located. In step S821, when it is determined that the display area where the user's gaze point is currently located is the lower area, and when it is determined that the display area where the user's gaze point was last located is the upper area, the gaze point interaction algorithm module can determine that the fence event that triggers the user's gaze point to switch from the upper area gaze fence to the lower area gaze fence needs to be reported. At this time, the fence event can be reported to the context-aware software development kit (SDK) of the gaze point interaction engine. Otherwise, it can be determined that it does not need to be reported, and the entire process ends. Steps S820 to S821 correspond to Figure 6 Step S606 and step S607 in the illustrated embodiment. Thereafter, the context-aware software development kit may inform the gaze intention processing module of the reading APP of the fence event.

[0263] It should be noted that Figure 8 The other contents of each step in the illustrated embodiment will not be described in detail in the embodiment of the present application.

[0264] In one possible embodiment, if no face is recognized in step S805 (i.e., the number of faces is 0), it can be determined that the user has left. If it is determined in step S809 that the face is not facing the screen, it can be determined that the user is not reading. At this time, the page turning can be automatically paused, and the screen can be controlled to dim or turn off to save power on the phone. If multiple faces are recognized in step S805, one of the faces can be selected as the target face for subsequent operations. Among them, the target face can be the face closest to the camera, or it can be the face with the highest usage rate. The embodiment of the present application does not specifically limit the method for selecting a face.

[0265] Below Figure 6 The calculation process of the page reading speed and the page turning waiting time in step S609 in the illustrated embodiment is further described in detail.

[0266] In a possible embodiment, step S609 may include: determining the page reading speed of the user in the upper area; estimating the remaining reading time of the user in the lower area based on the page reading speed; and determining the page turning waiting time based on the remaining reading time.

[0267] In a specific embodiment, the gaze intention processing module can calculate the page reading speed of the user in the upper area in real time based on the amount of page information in the upper area and the time the user spends reading the page content in the upper area. In other words, determining the page reading speed of the user in the upper area can include: obtaining the cumulative time (as an example of the first cumulative time) that the user's gaze point has stayed in the upper area while reading the upper half of the screen; determining the page reading speed based on the cumulative time of reading the upper half of the screen and the amount of page information in the upper area. The amount of page information can include, but is not limited to, the amount of information such as the number of words, the proportion of blank space and illustrations, and the embodiments of the present application do not impose specific restrictions on this.

[0268] In an embodiment of the present application, when a user opens a reading APP to display the corresponding application interface, or after performing operations such as page turning and chapter cutting, the amount of information of the page above the position of half the height of the application interface can be calculated according to the page content of the current page as the amount of page information in the upper area, and the amount of information of the page below the position of half the height of the application interface can be calculated as the amount of page information in the lower area. Exemplarily, if the current page is a text page, the coordinates of each word of the current page can be obtained, and the amount of text above the position of half the height of the application interface can be calculated as the amount of page information in the upper area. The amount of text below the position of half the height of the application interface is calculated as the amount of page information in the lower area.

[0269] In a possible embodiment, the amount of information of the page in the upper region may be calculated according to the page content of the current page as the amount of page information in the upper region, and the amount of information of the remaining pages may be taken as the amount of page information in the lower region.

[0270] In an embodiment of the present application, in order to obtain the user's cumulative time reading the upper half of the screen and the cumulative time reading the lower half of the screen, when the user opens a reading APP to display the corresponding application interface, or after performing operations such as turning pages and cutting chapters, the gaze intention processing module can record the current time as the time for reading the upper area or the time for reading the lower area according to the display area where the user's gaze point is located.

[0271] Specifically, when a user opens a reading APP to display the corresponding application interface, or after performing operations such as turning pages or cutting chapters, if the display area where the user's gaze point is located is the upper area, the gaze intention processing module can record the current time as the time for reading the upper area; if the display area where the user's gaze point is located is the lower area, the gaze intention processing module can record the current time as the time for reading the lower area.

[0272] In the embodiment of the present application, when the user gazes at the middle area of ​​the display interface, there may be a problem of the gaze point jumping caused by the inaccurate algorithm recognition result, and / or the user may scan up and down, or read the content of the page in the lower half and then re-read the content of the page in the upper half. Therefore, in order to obtain the cumulative time of the user reading the upper half screen and the cumulative time of reading the lower half screen, each time it is determined that the user's gaze point moves from the upper area to the lower area, the time when the user's gaze point moves from the upper area to the lower area can be recorded as the time of reading the lower area. Each time it is determined that the user's gaze point moves from the lower area to the upper area, the time when the user's gaze point moves from the lower area to the upper area can also be recorded as the time of reading the upper area. The time difference between the time of the last reading of the lower area and the time of the last reading of the upper area can also be added to the cumulative time of the user reading the upper half screen or the cumulative time of the user reading the lower half screen.

[0273] From the above analysis, it can be seen that when the user is reading a new page, the gaze intention processing module can obtain an event sequence indicating the change of the display area where the user's gaze point is located. Based on the event sequence, according to the time difference between the corresponding recorded time of reading the upper area and the time of reading the lower area, the cumulative time that the user's gaze point has stayed in the upper area for reading the upper half of the screen, and the cumulative time that the user's gaze point has stayed in the lower area for reading the lower half of the screen can be accumulated and calculated.

[0274] For example, in conjunction with the reference specification Fig. 9 , Fig. 9 A schematic diagram showing the change of the display area where the user's gaze point is located according to an embodiment of the present application is shown. Fig. 9 As shown, A→B→C→D→E→F is the change of the display area where the user's gaze point is located, and the time of each change can be recorded as t1, t2, t3, t4, t5, and t6. Then it can be determined that the sum of the time from point A→point B, point C→point D, point E→point F is T1=(t2-t1)+(t4-t3)+(t6-t5) for the user's gaze point staying in the upper area to read the upper half of the screen, and the sum of the time from point B→point C, point D→point E is T2=(t3-t2)+(t5-t4) for the user's gaze point staying in the lower area to read the lower half of the screen.

[0275] In the embodiment of the present application, the user's page reading speed can be estimated based on the cumulative reading time of the upper half of the screen and the amount of page information in the upper area. The specific calculation formula is as follows:

[0276]

[0277] Among them, V1 represents the page reading speed, X1 represents the amount of page information in the upper area, and T1 represents the cumulative time of reading the upper half of the screen.

[0278] In a specific embodiment, estimating the remaining reading time of the user in the lower area based on the page reading speed may include: when the page reading speed is within a preset speed range, determining the estimated reading time of the user in the lower area according to the amount of page information in the lower area and the page reading speed; when the page reading speed is outside the preset speed range, determining the estimated reading time of the user in the lower area according to the amount of page information in the lower area and the preset reading speed or the user's historical reading speed; determining the remaining reading time according to the estimated reading time and the cumulative time that the user's gaze has stayed in the lower area for reading the lower half of the screen (as an example of the second cumulative time).

[0279] The preset speed range can be preset according to actual conditions, for example, it can be determined according to the average reading speed range of a certain user group, or it can be determined according to the user's historical reading speed, and the present application embodiment does not impose specific restrictions on this. Exemplarily, the preset speed range can be set to a speed range greater than or equal to 5 words / second and less than or equal to 15 words / second.

[0280] The preset reading speed can be preset according to actual conditions, for example, it can be set to a default reading speed of 10 words per second, etc., and the present application embodiment does not impose specific restrictions on this. The historical reading speed can also be determined according to actual conditions, for example, it can be the average reading speed of all page contents of the page before the current page read by the user, or it can be the estimated reading speed of the user in the upper area when the user reads the page before the current page, and the present application embodiment does not impose specific restrictions on this.

[0281] In the embodiment of the present application, after determining the page reading speed, it is also possible to determine whether the page reading speed is within the normal range, or to determine whether there is a large error in combination with the user's historical reading speed. The page reading speed that does not meet the normal range or has a large error can be discarded, and the preset reading speed or the user's historical reading speed is used to estimate the user's estimated reading time in the lower area, and then the user's remaining reading time in the lower area is estimated. The specific estimation method of the estimated reading time and the remaining reading time will be described in detail later.

[0282] In a specific embodiment, determining the page turning waiting time based on the remaining reading time may include: when the remaining reading time is within a preset time range, determining the page turning waiting time to be the remaining reading time; when the remaining reading time is less than the minimum value of the preset time range, determining the page turning waiting time to be 2s (as an example of the first preset time); when the remaining reading time is greater than the maximum value of the preset time range, determining the page turning waiting time to be 9s (as an example of the second preset time).

[0283] Among them, the preset time range can be pre-set according to actual conditions, for example, it can be set to a time range greater than or equal to 3s and less than or equal to 9s, and the embodiment of the present application does not make specific restrictions on this.

[0284] In the embodiment of the present application, in order to avoid turning pages too quickly or not turning pages for too long, the gaze intention processing module can also check the estimated remaining reading time to determine whether it meets the reasonable time interval. The gaze intention processing module can discard the remaining reading time that does not meet the reasonable time interval and use the preset page turning waiting time.

[0285] It should be noted that the above example of using 2s as the first preset time and 9s as the second preset time is only an exemplary implementation. In actual applications, the first preset time and the second preset time can also be set to other values, and the embodiment of the present application does not impose specific restrictions on this.

[0286] The following is a detailed description of the process when a mobile phone opens a new page. Fig.10 As shown, Fig.10 A processing flow chart for opening a new page provided by an embodiment of the present application is shown, and the processing flow may include steps S1001-S1011.

[0287] like Fig.10 As shown, in step S1001, when the user opens the reading APP to display the corresponding application interface, or after triggering operations such as page turning and chapter cutting, the gaze intention processing module can first remove all delay methods, that is, all delay times can be reset. In step S1002, the gaze intention processing module can determine the amount of page information in the upper area and the amount of page information in the lower area. In step S1003, the gaze intention processing module can also determine the display area where the user's gaze point is located, and determine whether the display area where the user's gaze point is located is the upper area. In step S1004, if the display area where the user's gaze point is located is determined to be the upper area, the current time can be recorded as the time of the last reading of the upper area. And in step S1005, the gaze intention processing module can clear the cumulative time of reading the upper half of the screen and the cumulative time of reading the lower half of the screen, and reset the time of the last reading of the lower area.

[0288] In step S1006, the gaze intention processing module can also determine whether the amount of page information in the lower area is less than a preset information amount threshold. Among them, the preset information amount threshold can be preset according to actual needs, for example, it can be set to 30 words, etc., and the embodiment of the present application does not impose specific restrictions on this. In step S1007, if it is determined that the amount of page information in the lower area is less than the preset information amount threshold, the page turning waiting time of this page can be directly estimated, and subsequent steps can be executed. Exemplarily, the page turning waiting time can be directly calculated based on the page information amount of the entire page and the preset reading speed or the user's historical reading speed. Specifically, if it is determined that the amount of page information in the lower area is greater than or equal to the preset information amount threshold, no operation is performed.

[0289] In step S1008, if it is determined that the display area where the user's gaze point is located is not the upper area, it can be determined whether the display area where the user's gaze point is located is the lower area. In step S1009, if it is determined that the display area where the user's gaze point is located is the lower area, the current time can be recorded as the time of the most recent reading of the lower area. And in step S1010, the gaze intention processing module can clear the cumulative time of reading the upper half of the screen and the cumulative time of reading the lower half of the screen, and reset the time of the most recent reading of the upper area. In step S1011, the gaze intention processing module can also directly estimate the page turning waiting time of this page and execute subsequent steps. The estimation method is similar to that in step S1007, and the embodiments of the present application will not be repeated here. Specifically, if it is determined that the display area where the user's gaze point is located is not the lower area, it can be determined that the user is gazing at the area outside the application interface, and no operation can be performed at this time.

[0290] It should be noted that Fig.10 The other contents of each step in the illustrated embodiment will not be described in detail in the embodiment of the present application.

[0291] The following is a detailed description of the process when the user switches from gazing at the lower area to gazing at the upper area. Fig.11 As shown, Fig.11 A processing flow chart provided by an embodiment of the present application when a user switches from gazing at a lower area to gazing at an upper area is shown. The processing flow may include steps S1101-S1106.

[0292] like Fig.11As shown, in step S1101, when the user switches from gazing at the lower area to gazing at the upper area, all delay methods can be removed first, that is, all delay times can be reset. In step S1102, the gaze intention processing module can record the current time as the time of the last reading of the upper area. In step S1103, the gaze intention processing module can also determine whether the user has read all the contents of the page. For example, when the user is not prompted to look up and turn the page, it is determined whether the page turning waiting time calculated last time has passed after the user's gaze point was moved from the upper area to the lower area last time. Or, when the user is prompted to look up and turn the page, it can be directly determined whether the user has been prompted to look up and turn the page. In step S1104, if it is determined that the user has not read all the contents of the page, it can be determined whether the time of the last reading of the lower area is 0. In step S1105, if it is determined that the time of the last reading of the lower area is not 0, the time difference between the time of the last reading of the lower area and the time of the last reading of the upper area can be calculated, and added to the cumulative time of the user reading the lower half screen. If it is determined that the last reading time of the lower area is 0, no operation is performed. In step S1106, if it is determined that the user has finished reading all the contents of the page, a page turning instruction can be sent to the reader module after a delay of 200ms to perform a page turning operation on the application interface of the reading APP.

[0293] The following is a detailed description of the process when the user switches from gazing at the upper area to gazing at the lower area. Fig.12 As shown, Fig.12 A processing flow chart provided by an embodiment of the present application when a user switches from gazing at an upper area to gazing at a lower area is shown. The processing flow may include steps S1201-S1212.

[0294] like Fig.12 As shown, in step S1201, when the user switches from gazing at the upper area to gazing at the lower area of ​​the display interface, the gaze intention processing module can first remove all delay methods, that is, all delay times can be reset. In step S1202, the gaze intention processing module can record the current time as the time when the lower area was last read. In step S1203, the gaze intention processing module can also determine whether the time when the upper area was last read is 0. In step S1204, if it is determined that the time when the upper area was last read is 0, it can be determined that the user is looking directly at the lower area. At this time, the page turning waiting time of this page can be directly estimated, and subsequent steps can be executed. Estimation method and Fig.10 The estimation method in step S1007 in the illustrated embodiment is similar and will not be described in detail in the embodiment of the present application.

[0295] In step S1205, if it is determined that the time of the most recent reading of the upper area is not 0, the time difference between the time of the most recent reading of the lower area and the time of the most recent reading of the upper area can be calculated and added to the cumulative time of the user reading the upper half of the screen. In step S1206, the gaze intention processing module can also estimate the user's page reading speed based on the cumulative time of reading the upper half of the screen and the amount of page information in the upper area.

[0296] In step S1207, the gaze intention processing module can determine whether the page reading speed is within the preset speed range. If so, it can be determined that the page reading speed is within the normal range or there is no large error. If not, it can be determined that the page reading speed is not within the normal range or there is a large error.

[0297] In step S1208, when it is determined that the page reading speed is within the preset speed range (i.e., it is determined that the page reading speed is within the normal range or there is no large error), the gaze intention processing module can estimate the user's estimated reading time in the lower area according to the amount of page information in the lower area and the page reading speed, and then estimate the user's remaining reading time in the lower area. The specific calculation formula is as follows:

[0298]

[0299] Among them, T ′ Indicates the remaining reading time, X2 indicates the amount of page information in the lower area, and T2 indicates the cumulative time for reading the lower half of the screen.

[0300] Optionally, in step S1209, when it is determined that the page reading speed is outside the preset speed range (i.e., it is determined that the page reading speed does not meet the normal range or has a large error), the gaze intention processing module may discard the calculated page reading speed, and estimate the estimated reading time of the user in the lower area according to the amount of page information in the lower area and the preset reading speed, and then estimate the remaining reading time of the user in the lower area. The specific calculation formula is as follows:

[0301]

[0302] Among them, V2 represents the preset reading speed.

[0303] Optionally, in step S1209, when it is determined that the page reading speed is outside the preset speed range (i.e., it is determined that the page reading speed does not meet the normal range or has a large error), the gaze intention processing module may also discard the calculated page reading speed, and estimate the estimated reading time of the user in the lower area according to the amount of page information in the lower area and the user's historical reading speed, and then estimate the remaining reading time of the user in the lower area. The specific calculation formula is as follows:

[0304]

[0305] Among them, V3 represents the user's historical reading speed.

[0306] In step S1210, the gaze intention processing module can also calculate whether the remaining reading time is within the preset time range. In step S1211, when it is judged that the remaining reading time is within the preset time range, the gaze intention processing module can determine that the remaining reading time meets the reasonable time interval. At this time, it can be directly used as the page turning waiting time of the application interface of the reading APP. In step S1212, when it is judged that the remaining reading time is outside the preset time range, the gaze intention processing module can determine that the remaining reading time does not meet the reasonable time interval. At this time, if the remaining reading time is less than the minimum value of the preset time range, it can be determined that the page turning waiting time of the application interface of the reading APP is 2s. If the remaining reading time is greater than the maximum value of the preset time range, it can be determined that the page turning waiting time of the application interface of the reading APP is 9s. In other words, if the remaining reading time is less than the minimum value of the preset time range, it is necessary to wait for 2s at the fastest before executing the subsequent steps. If the remaining reading time is greater than the maximum value of the preset time range, the subsequent steps will also be executed after 9s at the latest.

[0307] The following is a detailed description of the page turning method provided by the embodiment of the present application, taking the scenario of prompting the user to look up and turn the page as an example. In this scenario, when it is determined that the user has finished reading all the contents of the page, the mobile phone can prompt the user to look up and turn the page, so that the user can get a clear psychological hint that the intention has been accurately recognized.

[0308] refer to Fig.13 As shown, Fig.13 A flow chart of a page turning method provided in another embodiment of the present application is shown, and the method may include steps S1301-S1306.

[0309] S1301: The mobile phone displays an application interface of a reading APP (as an example of a first application), which includes an upper area (as an example of a first display area) and a lower area (as an example of a second display area), and the upper area and the lower area partially overlap or do not overlap at all.

[0310] S1302: The mobile phone determines that the user's gaze point relative to the application interface moves from the upper area to the lower area.

[0311] S1303: In response to determining that the user's gaze point relative to the application interface moves from the upper area to the lower area, the mobile phone determines a page turning waiting time of the application interface of the reading APP.

[0312] In the embodiment of the present application, in order to avoid prompting the user to turn the page too quickly, causing the page to be turned incorrectly, or not prompting the user to turn the page for too long, causing the user to wait too long, the estimated remaining reading time can also be checked to determine whether it meets the reasonable time interval, and the page turning waiting time is determined based on the judgment result. The specific judgment method can be referred to Figure 4 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0313] S1304: In response to determining that the user's gaze point moves from the upper area to the lower area, the mobile phone displays page turning prompt information in the application interface of the reading APP after a page turning waiting time has elapsed.

[0314] In the embodiment of the present application, in response to determining that the user's gaze point has moved from the upper area to the lower area, and the current waiting time t reaches the page turning waiting time T (i.e., determining that the user has finished reading all the contents of the page), the mobile phone can also provide a page turning prompt to guide the user to look up and turn the page. Specifically, the page turning prompt information can be displayed in the application interface of the reading APP, and the page turning prompt information can be used to prompt the user to move the gaze point from the lower area to the upper area to automatically turn the page.

[0315] It should be noted that the content and display position of the page turning prompt information can be pre-set according to actual conditions. For example, the page turning prompt information can be set to "Look up, turn a page", and the display position of the page turning prompt information can be set to the bottom of the page. The embodiment of the present application does not impose specific restrictions on this.

[0316] S1305: The mobile phone determines that the user's gaze point moves from the lower area to the upper area.

[0317] In the embodiment of the present application, the mobile phone can judge the intention of turning the page while displaying the page turning prompt information. The specific judgment method can be referred to Figure 4 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0318] S1306: In response to determining that the user's gaze point moves from the lower area to the upper area, the mobile phone performs a page turning operation on the application interface of the reading APP.

[0319] In the embodiment of the present application, due to the following recognition result jump scenario: the user gazes at the lower area of ​​the application interface, and the user has been prompted to look up to turn the page, but the user does not look up but the algorithm recognition error occurs, resulting in the user's gaze point jumping from bottom to top and then down. In this scenario, the page turning will be triggered by mistake, affecting the user experience. Therefore, in order to avoid this jump scenario, a page turning delay of 200ms can also be added. For the specific delayed page turning method, please refer to Figure 4 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0320] For example, in conjunction with the reference specification Fig.14a , Fig.14a FIG. 1 is a schematic diagram showing a left-right page turning process provided by an embodiment of the present application. Fig.14a As shown, when the user moves the gaze point from position 1401 in the upper area to position 1402 in the lower area, the mobile phone can determine that the user's gaze point has moved from the upper area to the lower area. In response to determining that the user's gaze point has moved from the upper area to the lower area, and after the page turning waiting time has passed, the mobile phone can display the page turning prompt information "Look up, turn the page" in the application interface 1410 of the reading APP. Thereafter, when the user moves the gaze point from position 1403 in the lower area to position 1404 in the upper area, the mobile phone can determine that the user's gaze point has moved from the lower area to the upper area. In response to determining that the user's gaze point has moved from the lower area to the upper area, the mobile phone can execute the action of turning a page left or right for the application interface of the reading APP after a delay of 200ms.

[0321] For example, in conjunction with the reference specification Fig.14b , Fig.14b FIG. 1 is a schematic diagram showing a process of turning pages of the upper and lower half screens provided by an embodiment of the present application. Fig.14b As shown, when the user moves the gaze point from position 1405 in the upper area to position 1406 in the lower area, the mobile phone can determine that the user's gaze point has moved from the upper area to the lower area. In response to determining that the user's gaze point has moved from the upper area to the lower area, and after the page turning waiting time has passed, the mobile phone can display a page turning prompt message of "Look up, turn the page" in the application interface 1420 of the reading APP. Thereafter, when the user moves the gaze point from position 1407 in the lower area to position 1408 in the upper area, the mobile phone can determine that the user's gaze point has moved from the lower area to the upper area. In response to determining that the user's gaze point has moved from the lower area to the upper area, the mobile phone can perform a page turning action of half-screen flipping coverage of the application interface of the reading APP after a delay of 200ms.

[0322] It should be noted that other contents in step S1301 to step S1306 in the embodiment of the present application can be referred to Figures 4 to 12 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0323] In one possible embodiment, when the user performs a rotation operation on the mobile phone to switch the mobile phone from a portrait screen to a landscape screen, the mobile phone can obtain the corresponding rotation direction, which may include rotating to the left and rotating to the right. When rotating to the left, the camera (i.e., the image sensor) of the mobile phone will be rotated to the left, and when rotating to the right, the camera of the mobile phone will be rotated to the right. The mobile phone can also obtain the current interface follow mode, which may include two modes: follow and do not follow. In the follow mode, the layout of the application interface can change with the rotation of the mobile phone (i.e., from a top-to-bottom layout to a left-to-right layout), and in the do not follow mode, the layout of the application interface may not change with the rotation of the mobile phone. The mobile phone can switch the display interface according to the rotation direction and the current interface follow mode, and adjust the division of the two display areas in the display interface.

[0324] The following is a detailed description using the example where the rotation direction of the mobile phone is left rotation and the interface following mode is non-following.

[0325] For example, in conjunction with the reference specification Fig.15 , Fig.15 A schematic diagram of switching a mobile phone from portrait to landscape provided by an embodiment of the present application is shown. When the user rotates the mobile phone to the left, assuming that the current interface follow mode is not to follow, the position relationship between the two display areas of the user interface does not change with the rotation of the mobile phone, and remains in an up-and-down relationship. In other words, after the mobile phone is switched from portrait to landscape, the reading page still remains upward. Fig.15 As shown, when the mobile phone is in the vertical screen display state, the vertical screen application interface 1510 of the reading APP may include an upper area and a lower area, the upper area of ​​the vertical screen application interface 1510 may include an upper grid 1501 and a transition area 1502, and the lower area of ​​the vertical screen application interface 1510 may include a transition area 1502 and a lower grid 1503. When the user rotates the mobile phone to the left, the mobile phone enters the horizontal screen display state, and the application interface of the reading APP is switched to a horizontal screen application interface 1520, which may still include an upper area and a lower area, the upper area of ​​the horizontal screen application interface 1520 may include an upper grid 1504 and a transition area 1505, and the lower area of ​​the horizontal screen application interface 1520 may include a transition area 1505 and a lower grid 1506.

[0326] In the embodiment of the present application, when the interface follow mode is not to follow, although the layout of the application interface does not change after the mobile phone is switched from the portrait screen to the landscape screen, the position and angle of the mobile phone's camera (i.e., the image sensor) relative to the user's eyes will change under different display states (including the portrait screen display state and the landscape screen display state). Therefore, the mobile phone can use different modes of gaze point interaction algorithms to identify the user's gaze point and determine the display area where the gaze point is located in different display states. For example, when the mobile phone is in the portrait screen display state, the gaze point interaction algorithm of the upper and lower area recognition mode can be used to identify the user's gaze point and determine the display area where the gaze point is located. When the mobile phone is in the landscape screen display state, the gaze point interaction algorithm of the left and right area recognition mode can be used to identify the user's gaze point and determine the display area where the gaze point is located. Among them, different modes of gaze point interaction algorithms can correspond to different camera position parameters and angle parameters, etc.

[0327] Specifically, in the left and right area recognition mode, the mobile phone can use the gaze point interaction algorithm to determine whether the user's gaze point is located in the left area or the right area of ​​the physical screen, and can map the left area and the right area of ​​the physical screen to the upper area and the lower area of ​​the display interface respectively based on the rotation direction of the mobile phone, thereby determining whether the display area of ​​the application interface where the user's gaze point is located is the upper area or the lower area. Specifically, when the rotation direction is to the left, the left area of ​​the physical screen can be automatically mapped to the lower area of ​​the display interface, and the right area of ​​the physical screen can be mapped to the upper area of ​​the display interface. When the rotation direction is to the right, the left area of ​​the physical screen can be automatically mapped to the upper area of ​​the display interface, and the right area of ​​the physical screen can be mapped to the lower area of ​​the display interface.

[0328] In a specific embodiment, in conjunction with the reference specification, Fig.16 , Fig.16 A flow chart of a page turning method provided by another specific embodiment of the present application is shown, which is a further explanation of an embodiment of switching a mobile phone from a vertical screen to a horizontal screen scene. Fig.16 The contents of step S1601, step S1604 to step S1623 in the embodiment shown are the same as those in the embodiment shown in FIG. Figure 6 The contents of step S601 to step S621 in the illustrated embodiment are similar, and the embodiments of the present application will not repeat the same contents therein, and only describe the different contents therein in detail.

[0329] like Fig.16As shown, in step S1602, when the user performs a rotation operation on the mobile phone to switch the mobile phone from portrait to landscape, the reader module can respond to the rotation operation and determine the rotation direction of the mobile phone and the interface follow mode. And the application interface of the reading APP displayed on the mobile phone is switched based on the interface follow mode. In step S1603, the reader module can notify the gaze point interaction engine of the rotation operation of switching the mobile phone from portrait to landscape, and send the rotation direction of the rotation operation and the interface follow mode.

[0330] In step S1606, after receiving the facial image of the user, the gaze point interaction engine may input the facial image of the user into the gaze point interaction algorithm module for processing. Specifically, the gaze point interaction engine may determine relevant parameters (such as the position and angle of the image sensor relative to the user's eyes) according to the rotation direction, and input the facial image, relevant parameters and interface following mode into the gaze point interaction algorithm module together. In step S1607, the gaze point interaction algorithm module may perform gaze point recognition processing on the facial image to obtain the user's gaze point coordinates relative to the application interface. In step S1608, the gaze point interaction algorithm module may also perform fence entry and exit judgment according to the gaze point coordinates to determine whether to trigger a fence event in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence. Specifically, when determining that the interface following mode is not to follow, the gaze point interaction algorithm module may use the gaze point interaction algorithm of the left and right area recognition mode to identify the user's gaze point and determine the area of ​​the physical screen where the gaze point is located. The gaze point interaction algorithm module can also determine the rotation direction of the rotation operation, and based on the rotation direction, map the left and right areas of the physical screen to the upper and lower areas of the display interface respectively, thereby determining the display area of ​​the application interface where the gaze point is located, and then determine whether to trigger a fence event in which the user's gaze point switches from the upper area gaze fence to the lower area gaze fence.

[0331] In step S1615, after the gaze point interaction engine receives the user's facial image, the user's facial image can be input into the gaze point interaction algorithm module for processing. The specific implementation process is similar to step S1606, and the embodiments of the present application will not be repeated here. In step S1616, the gaze point interaction algorithm module can perform gaze point recognition processing on the facial image to obtain the user's gaze point coordinates relative to the application interface. In step S1617, the gaze point interaction algorithm module can also perform fence entry and exit judgment based on the gaze point coordinates to determine whether to trigger a fence event in which the user's gaze point switches from the lower area gaze fence to the upper area gaze fence (i.e., determine that the user's gaze point moves from the lower area to the upper area). The specific judgment method is similar to steps S1607 and S1608, and the embodiments of the present application will not be repeated here.

[0332] It should be noted that Fig.16 For other contents in step S1601 to step S1623 in the embodiment shown, please refer to Figure 6 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0333] Taking a mobile phone as an example, the following is a detailed description of the page turning method provided by another embodiment of the present application. The mobile phone can be in a vertical screen display state, and multiple APPs are installed in the mobile phone, some or all of which are APPs that provide information to users. In the initial state, the user can choose whether to look at the upper half or the lower half of the application interface, and the corresponding page turning method is determined according to the result of the user's selection.

[0334] The following first describes in detail the page turning method provided by the embodiment of the present application in the case where the user chooses to focus on the upper half of the application interface. Fig.17 As shown, Fig.17 A flow chart of a page turning method provided in another embodiment of the present application is shown, and the method may include steps S1701-S1705.

[0335] S1701: The mobile phone displays an application interface of a reading APP (as an example of a second application), which includes an upper area (as an example of a first display area) and a lower area (as an example of a second display area), and the upper area and the lower area partially overlap or do not overlap at all.

[0336] In the embodiment of the present application, the second application can be any application installed in the mobile phone that can provide information to the user. The second application can be various types of applications, including but not limited to reading apps, browser apps, information flow apps, etc. In the embodiment of the present application, a reading app is used as an example of the first application, but the present application is not limited thereto. The reading app can be an app in a scrolling page turning mode or a sliding page turning mode.

[0337] S1702: The mobile phone determines that the user's gaze point relative to the application interface moves from the upper area to the lower area.

[0338] S1703: In response to determining that the user's gaze point relative to the application interface moves from the upper area to the lower area, and after 1s (as an example of the fourth preset time), the mobile phone adjusts the page scrolling speed of the application interface of the reading APP from the default scrolling speed (as an example of the first scrolling speed) to the accelerated scrolling speed (as an example of the second scrolling speed).

[0339] In the embodiment of the present application, since the reading APP is an APP in a scrolling page turning mode or a sliding page turning mode, and the user chooses to gaze at the upper half of the application interface, when the mobile phone determines that the user's gaze point has moved from the upper area to the lower area, it can be determined that the scrolling speed of the page is slower than the user's reading speed and the scrolling speed needs to be increased. In response to determining that the user's gaze point relative to the application interface has moved from the upper area to the lower area, the mobile phone can adjust the page scrolling speed of the application interface of the reading APP from the default scrolling speed to the accelerated scrolling speed after 1 second, thereby accelerating the scrolling of the pages of the application interface.

[0340] The default scrolling speed and the accelerated scrolling speed can be preset according to actual conditions. For example, the default scrolling speed can be set by the user, and the accelerated scrolling speed can be calculated based on the default scrolling speed set by the user. For different users, the corresponding default scrolling speed and accelerated scrolling speed can be the same or different, and the embodiments of the present application do not impose specific restrictions on this.

[0341] It should be noted that the above example of using 1s as the fourth preset time is only an exemplary implementation. In actual applications, the fourth preset time can also be set to other values, and the embodiment of the present application does not impose specific limitations on this.

[0342] In a possible embodiment, the page turning method may further include: the mobile phone determines the adjustment coefficient corresponding to the user, and determines the accelerated scrolling speed based on the adjustment coefficient. The adjustment coefficient corresponding to the user may be greater than 1 (i.e., the acceleration coefficient). The adjustment coefficient corresponding to the user may be preset according to actual conditions, for example, it may be set to 1.5. For different users, the same adjustment coefficient may be set, or different adjustment coefficients may be set, and the embodiment of the present application does not specifically limit this.

[0343] Specifically, the default scrolling speed may be multiplied by the adjustment coefficient corresponding to the user to obtain the accelerated scrolling speed. Since the adjustment coefficient is greater than 1, the calculated accelerated scrolling speed is greater than the default scrolling speed.

[0344] S1704: The mobile phone determines that the user's gaze point moves from the lower area to the upper area.

[0345] S1705: In response to determining that the user's gaze point moves from the lower area to the upper area, and after 1 second (as an example of the fifth preset time), the mobile phone adjusts the page scrolling speed of the application interface of the reading APP from the accelerated scrolling speed to the default scrolling speed.

[0346] In the embodiment of the present application, when the mobile phone determines that the user's gaze point moves from the lower area to the upper area, it can be determined that the scrolling speed of the page is faster than the user's reading speed and the scrolling speed needs to be reduced. In response to determining that the user's gaze point moves from the lower area to the upper area, the mobile phone can adjust the page scrolling speed of the application interface of the reading APP from the accelerated scrolling speed to the default scrolling speed after 1 second, so that the page of the application interface returns to the default speed.

[0347] It should be noted that the above example of using 1s as the fifth preset time is only an exemplary implementation. In actual applications, the fifth preset time can also be set to other values, and the embodiment of the present application does not impose specific limitations on this.

[0348] It should be noted that other contents in step S1701 to step S1705 in the embodiment of the present application can be referred to Figures 4 to 12 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0349] In a specific embodiment, in conjunction with the reference specification, Fig.18 , Fig.18 A flow chart of a page turning method provided by another specific embodiment of the present application is shown. Fig.17 Further description of the illustrated embodiment. Fig.18 The contents of steps S1801 to S1808 and steps S1811 to S1817 in the embodiment shown are respectively the same as Figure 6 The contents of steps S601 to S608 and steps S611 to S617 in the embodiment shown are similar, and the embodiments of the present application will not be repeated here. Figure 6 The different contents in the illustrated embodiments are described in detail.

[0350] like Fig.18 As shown, in step S1809, the gaze intention processing module can respond to the fence event of the user's gaze point switching from the upper area gaze fence to the lower area gaze fence, and after 1 second, send an accelerated scrolling instruction to the reader module. In step S1810, the reader module can respond to the accelerated scrolling instruction and perform an accelerated scrolling operation on the application interface of the reading APP.

[0351] It should be noted that step S1809 to step S1810 correspond to Fig.17 Step S1703 in the illustrated embodiment.

[0352] like Fig.18As shown, in step S1818, the gaze intention processing module can respond to the fence event of the user's gaze point switching from the lower area gaze fence to the upper area gaze fence, and after 1 second, send a scroll speed recovery instruction to the reader module. In step S1819, the reader module can respond to the scroll speed recovery instruction and perform a scroll speed recovery operation for the application interface of the reading APP.

[0353] It should be noted that step S1818 to step S1819 correspond to Fig.17 Step S1705 in the illustrated embodiment.

[0354] It should be noted that Fig.18 For other contents in step S1801 to step S1819 in the embodiment shown, please refer to Figure 6 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0355] For example, in conjunction with the reference specification Fig.19 , Fig.19 FIG. 1 is a schematic diagram showing accelerated scrolling and page turning provided by an embodiment of the present application. Fig.19 As shown, when the user moves the gaze point from position 1901 of the upper area to position 1902 of the lower area, the mobile phone can determine that the user's gaze point has moved from the upper area to the lower area. In response to determining that the user's gaze point has moved from the upper area to the lower area, the mobile phone adjusts the page scrolling speed of the application interface 1910 from the default scrolling speed to the accelerated scrolling speed after 1 second, thereby accelerating the scrolling of the page of the application interface 1910.

[0356] like Fig.19 As shown, when the user moves the gaze point from position 1903 of the lower area to position 1904 of the upper area, the mobile phone can determine that the user's gaze point has moved from the lower area to the upper area. In response to determining that the user's gaze point has moved from the lower area to the upper area, the mobile phone can adjust the page scrolling speed of the application interface 1910 from the accelerated scrolling speed to the default scrolling speed after 1 second, thereby restoring the page scrolling speed of the application interface 1910.

[0357] The following is a detailed description of the page turning method provided in the embodiment of the present application for the case where the user chooses to focus on the lower half of the application interface. Fig. 20 As shown, Fig. 20 A flow chart of a page turning method provided in another embodiment of the present application is shown. The method may include steps S2001-S2005.

[0358] S2001: The mobile phone displays an application interface of a reading APP (as an example of a second application), which application interface includes an upper area (as an example of a first display area) and a lower area (as an example of a second display area), and the upper area and the lower area partially overlap or do not overlap at all.

[0359] S2002: The mobile phone determines that the user's gaze point relative to the application interface moves from the lower area to the upper area.

[0360] S2003: In response to determining that the user's gaze point relative to the application interface moves from the lower area to the upper area, and after 1s (as an example of the fourth preset time), the mobile phone adjusts the page scrolling speed of the application interface of the reading APP from the default scrolling speed (as an example of the first scrolling speed) to the decelerated scrolling speed (as an example of the second scrolling speed).

[0361] In the embodiment of the present application, since the reading APP is an APP in a scrolling page turning mode or a sliding page turning mode, and the user chooses to gaze at the lower half of the application interface, when the mobile phone determines that the user's gaze point has moved from the lower area to the upper area, it can be determined that the scrolling speed of the page is faster than the user's reading speed and the scrolling speed needs to be reduced. In response to determining that the user's gaze point relative to the application interface has moved from the lower area to the upper area, the mobile phone can adjust the page scrolling speed of the application interface of the reading APP from the default scrolling speed to the decelerated scrolling speed after 1 second, thereby causing the pages of the application interface to scroll at a reduced speed.

[0362] The default scrolling speed and the deceleration scrolling speed can be preset according to actual conditions. For example, the default scrolling speed can be set by the user, and the deceleration scrolling speed can be calculated based on the default scrolling speed set by the user. For different users, the corresponding default scrolling speed and deceleration scrolling speed can be the same or different, and the embodiments of the present application do not impose specific restrictions on this.

[0363] It should be noted that the above example of using 1s as the fourth preset time is only an exemplary implementation. In actual applications, the fourth preset time can also be set to other values, and the embodiment of the present application does not impose specific limitations on this.

[0364] In a possible embodiment, the page turning method may further include: the mobile phone determines the adjustment coefficient corresponding to the user, and determines the deceleration scrolling speed based on the adjustment coefficient. The adjustment coefficient corresponding to the user may be less than 1 (i.e., the deceleration coefficient). The adjustment coefficient corresponding to the user may be preset according to actual conditions, for example, it may be set to 0.5. For different users, the same adjustment coefficient may be set, or different adjustment coefficients may be set, and the embodiment of the present application does not specifically limit this.

[0365] Specifically, the deceleration scrolling speed may be obtained by multiplying the default scrolling speed by the adjustment coefficient corresponding to the user. Since the adjustment coefficient is less than 1, the calculated deceleration scrolling speed is less than the default scrolling speed.

[0366] S2004: The mobile phone determines that the user's gaze point moves from the upper area to the lower area.

[0367] S2005: In response to determining that the user's gaze point moves from the upper area to the lower area, and after 1 second (as an example of the fifth preset time), the mobile phone adjusts the page scrolling speed of the application interface of the reading APP from the decelerated scrolling speed to the default scrolling speed.

[0368] In the embodiment of the present application, when the mobile phone determines that the user's gaze point moves from the upper area to the lower area, it can be determined that the scrolling speed of the page is slower than the user's reading speed and the scrolling speed needs to be increased. In response to determining that the user's gaze point moves from the upper area to the lower area, the mobile phone can adjust the page scrolling speed of the reading APP application interface from the decelerated scrolling speed to the default scrolling speed after 1 second, so that the page of the application interface returns to the default speed.

[0369] It should be noted that the above example of using 1s as the fifth preset time is only an exemplary implementation. In actual applications, the fifth preset time can also be set to other values, and the embodiment of the present application does not impose specific limitations on this.

[0370] It should be noted that other contents in step S2001 to step S2005 in the embodiment of the present application can be referred to Figures 4 to 12 , Figures 17 to 19 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0371] In a specific embodiment, in conjunction with the reference specification, Fig.21 , Fig.21 A flow chart of a page turning method provided by another specific embodiment of the present application is shown. Fig. 20 Further description of the illustrated embodiment. Fig.21 The contents of steps S2101 to S2108 and steps S2111 to S2117 in the embodiment shown are respectively the same as Figure 6 The contents of steps S601 to S608 and steps S611 to S617 in the embodiment shown are similar, and the embodiments of the present application will not be repeated here. Figure 6 The different contents in the illustrated embodiments are described in detail.

[0372] like Fig.21As shown, in step S2109, the gaze intention processing module can respond to the fence event of the user's gaze point switching from the lower area gaze fence to the upper area gaze fence, and after 1 second, send a deceleration scrolling instruction to the reader module. In step S2110, the reader module can respond to the deceleration scrolling instruction and perform a deceleration scrolling operation on the application interface of the reading APP.

[0373] It should be noted that step S2109 to step S2110 correspond to Fig. 20 Step S2003 in the illustrated embodiment.

[0374] like Fig.21 As shown, in step S2118, the gaze intention processing module can respond to the fence event that the user's gaze point switches from the upper area gaze fence to the lower area gaze fence, and after 1 second, send a scroll speed recovery instruction to the reader module. In step S2119, the reader module can respond to the scroll speed recovery instruction and perform a scroll speed recovery operation for the application interface of the reading APP.

[0375] It should be noted that step S2118 to step S2119 correspond to Fig. 20 Step S2005 in the illustrated embodiment.

[0376] It should be noted that Fig.21 For other contents in step S2101 to step S2119 in the embodiment shown, please refer to Figure 6 The relevant contents in the illustrated embodiments will not be repeated in the embodiments of the present application.

[0377] For example, in conjunction with the reference specification Fig. 22 , Fig. 22 A schematic diagram of decelerating scrolling and page turning provided by an embodiment of the present application is shown. Fig. 22 As shown, when the user moves the gaze point from position 2201 of the lower area to position 2202 of the upper area, the mobile phone can determine that the user's gaze point has moved from the lower area to the upper area. In response to determining that the user's gaze point has moved from the lower area to the upper area, the mobile phone adjusts the page scrolling speed of the application interface 2210 from the default scrolling speed to the decelerated scrolling speed after 1 second, thereby causing the page of the application interface 2210 to scroll at a decelerated speed.

[0378] like Fig. 22As shown, when the user moves the gaze point from position 2203 of the upper area to position 2204 of the lower area, the mobile phone can determine that the user's gaze point has moved from the upper area to the lower area. In response to determining that the user's gaze point has moved from the upper area to the lower area, the mobile phone can adjust the page scrolling speed of the application interface 2210 from the decelerated scrolling speed to the default scrolling speed after 1 second, thereby restoring the page scrolling speed of the application interface 2210.

[0379] In addition, in the above embodiment, only the vertical screen state is described. The page turning method in the horizontal screen state can be the same as the page turning method in the vertical screen state. For details, please refer to the page turning process in the vertical screen state, which will not be repeated here.

[0380] The present application also provides an electronic device, comprising:

[0381] a memory for storing instructions to be executed by one or more processors of the electronic device, and

[0382] The processor, when the processor executes the instructions in the memory, can make the electronic device execute the above embodiment Figures 4 to 22 The page turning method shown.

[0383] The present application also provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor executes the above-mentioned embodiment. Figures 4 to 22 The page turning method shown.

[0384] The present application also provides a computer program product comprising instructions, which, when executed on an electronic device, enables a processor to execute the above-mentioned embodiments. Figures 4 to 22 The page turning method shown.

[0385] Reference now Fig.23, shown is a block diagram of an electronic device 2300 according to one embodiment of the present application. The electronic device 2300 may include one or more processors 2301 coupled to a controller hub 2303. For at least one embodiment, the controller hub 2303 communicates with the processor 2301 via a multi-drop bus such as a front side bus (FSB), a point-to-point interface such as a Quick Path Interconnect (QPI), or a similar connection 2306. The processor 2301 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 2303 includes, but is not limited to, a graphics memory controller hub (Graphics Memory Controller Hub, GMCH) (not shown) and an input / output hub (Input Output Hub, IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes a memory and a graphics controller and is coupled to the IOH.

[0386] The electronic device 2300 may also include a coprocessor 2302 and a memory 2304 coupled to the controller hub 2303. Alternatively, one or both of the memory and the GMCH may be integrated within the processor (as described in the present application), with the memory 2304 and the coprocessor 2302 being directly coupled to the processor 2301 and the controller hub 2303, with the controller hub 2303 being in a single chip with the IOH.

[0387] The memory 2304 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of the two. The memory 2304, as a computer-readable storage medium, may include one or more tangible, non-temporary computer-readable media for storing data and / or instructions. For example, the memory 2304 may include any suitable non-volatile memory such as a flash memory and / or any suitable non-volatile storage device, such as one or more hard disk drives (HDD(s)), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0388] According to some embodiments of the present application, the memory 2304 as a computer-readable storage medium stores instructions, which, when executed on the computer, cause the system 2300 to execute the method for displaying the floating ball control according to the above embodiment. For details, please refer to the above embodiment. Figures 4 to 22The page turning method shown will not be described in detail here.

[0389] In one embodiment, the coprocessor 2302 is a special purpose processor, such as, for example, a high throughput Many Integrated Core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, a general purpose graphics processor (GPGPU), or an embedded processor. The optional nature of the coprocessor 2302 is indicated by a dashed line in Fig.23 middle.

[0390] In one embodiment, the electronic device 2300 may further include a network interface (Network Interface Controller, NIC) 2306. The network interface 2306 may include a transceiver for providing a radio interface for the electronic device 2300, thereby communicating with any other suitable device (such as a front-end module, an antenna, etc.). In various embodiments, the network interface 2306 may be integrated with other components of the electronic device 2300. The network interface 2306 may implement the functions of the communication unit in the above-mentioned embodiments.

[0391] The electronic device 2300 may further include an input / output (I / O) device 2305. The I / O 2305 may include: a user interface designed to enable a user to interact with the electronic device 2300; a peripheral component interface designed to enable peripheral components to interact with the electronic device 2300; and / or a sensor designed to determine environmental conditions and / or location information related to the electronic device 2300.

[0392] It is worth noting that Fig.23 This is for illustrative purposes only. Fig.23 It is shown that the electronic device 2300 includes multiple devices such as a processor 2301, a controller hub 2303, and a memory 2304. However, in actual applications, the device using the methods of the present application may only include a part of the devices of the electronic device 2300, for example, it may only include the processor 2301 and the network interface 2306. Fig.23 The properties of optional devices are shown with dotted lines.

[0393] Reference now Fig.24 , which is a block diagram of a SoC (System on Chip) 2400 according to an embodiment of the present application. Fig.24 In the FIG, similar components have the same reference numerals. In addition, the dashed boxes are optional features of more advanced SoCs. Fig.24In the embodiment, SoC 2400 includes: an interconnect unit 2450, which is coupled to a processor 2410; a system agent unit 2470; a bus controller unit 2480; an integrated memory controller unit 2440; a group or one or more coprocessors 2420, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 2430; and a direct memory access (DMA) unit 2460. In one embodiment, the coprocessor 2420 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high throughput MIC processor, or an embedded processor.

[0394] The static random access memory (SRAM) unit 2430 may include one or more tangible, non-transitory computer-readable storage media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of the instructions. The instructions may include: when executed by at least one unit in the processor, the Soc 2400 executes the display method of the floating ball control in the above embodiment, which may be specifically referred to in the above embodiment. Figures 4 to 22 The page turning method shown will not be described in detail here.

[0395] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0396] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0397] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0398] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or by other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, compact disc read-only memories (Compact Disc Read Only Memory, CD-ROMs), magneto-optical disks, read-only memories (Read Only Memory, ROM), random access memories (RAM), erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), electrically erasable programmable read-only memories (Electrically Erasable Programmable Read Only Memory, EEPROM), magnetic cards or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0399] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0400] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0401] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprise one" do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0402] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A page turning method, used in an electronic device, characterized in that: The method comprises: Displaying an application interface of a first application, the application interface comprising a first display area and a second display area, wherein the first display area and the second display area partially overlap or do not overlap at all; In response to determining that the user's gaze point relative to the application interface moves from the first display area to the second display area, determining a page turning waiting time of the application interface of the first application; In response to determining that the user's gaze point moves from the first display area to the second display area, after the page turning waiting time has passed, and in response to determining that the user's gaze point moves from the second display area to the first display area, a page turning operation is performed on the application interface of the first application.

2. The method according to claim 1, characterized in that: The method further comprises: In response to determining that the gaze point of the user moves from the first display area to the second display area, after the page turning waiting time has elapsed, displaying page turning prompt information in the application interface of the first application; The page turning prompt information is used to prompt the user to move the gaze point from the second display area to the first display area to turn the page.

3. The method according to claim 1, characterized in that The determining that the user's gaze point relative to the application interface moves from the first display area to the second display area includes: Determine a display area where the user's gaze point is currently located as the second display area, and determine a display area where the user's gaze point was previously located as the first display area; Wherein, determining that the user's gaze point moves from the second display area to the first display area includes: Determine a display area where the user's gaze point is currently located as a first display area, and determine a display area where the user's gaze point was previously located as a second display area; The display area where the user's gaze point is currently located and the display area where the gaze point was last located are determined based on at least one frame of facial image of the user.

4. The method according to claim 1, characterized in that The determining the page turning waiting time of the application interface of the first application includes: determining a page reading speed of the user in the first display area; estimating the remaining reading time of the user in the second display area based on the page reading speed; The page turning waiting time is determined based on the remaining reading time.

5. The method according to claim 4, characterized in that The determining the page reading speed of the user in the first display area includes: Acquire a first accumulated time that the user's gaze point has stayed in the first display area; The page reading speed is determined according to the first accumulated time and the amount of page information in the first display area.

6. The method according to claim 4, characterized in that The estimating the remaining reading time of the user in the second display area based on the page reading speed includes: When the page reading speed is within a preset speed range, determining an estimated reading time of the user in the second display area according to the amount of page information in the second display area and the page reading speed; When the page reading speed is outside the preset speed range, determining the estimated reading time of the user in the second display area according to the amount of page information in the second display area and the preset reading speed or the user's historical reading speed; The remaining reading time is determined according to the estimated reading time and the second accumulated time that the user's gaze point has stayed in the second display area.

7. The method according to claim 4, characterized in that The determining the page turning waiting time based on the remaining reading time includes: When the remaining reading time is within a preset time range, determining the page turning waiting time as the remaining reading time; When the remaining reading time is less than the minimum value of the preset time range, determining the page turning waiting time to be a first preset time; When the remaining reading time is greater than the maximum value of the preset time range, the page turning waiting time is determined to be a second preset time.

8. The method according to claim 1, characterized in that: The performing a page turning operation on the application interface of the first application includes: In response to determining that the user's gaze point moves from the second display area to the first display area, after a third preset time, a page turning operation is performed on the application interface of the first application.

9. The method according to claim 8, characterized in that The method further comprises: Within the third preset time, in response to determining that the user's gaze point moves from the first display area to the second display area, abandoning the page turning operation on the application interface of the first application.

10. A page turning method, used in an electronic device, characterized in that: The method comprises: Displaying an application interface of a second application, the application interface comprising a first display area and a second display area, wherein the first display area and the second display area partially overlap or do not overlap at all; In response to determining that the user's gaze point relative to the application interface moves from the first display area to the second display area, and after a fourth preset time, the page scrolling speed of the application interface of the second application is adjusted from the first scrolling speed to the second scrolling speed.

11. The method according to claim 10, characterized in that The method further comprises: Determining an adjustment coefficient corresponding to the user; The second scroll speed is determined based on the adjustment factor.

12. The method according to claim 10, characterized in that The method further comprises: In response to determining that the user's gaze point moves from the second display area to the first display area, and after a fifth preset time has passed, the page scrolling speed of the application interface of the second application is adjusted from the second scrolling speed to the first scrolling speed.

13. An electronic device, characterized in that: include: a memory for storing instructions executed by one or more processors of the electronic device; A processor, when the processor executes the instructions in the memory, can enable the electronic device to execute the page turning method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the page turning method according to any one of claims 1 to 12.

Images