Electronic book control method and device, electronic device, and storage medium

Abstract

Embodiments of the present application relate to the technical field of electronic equipment, and disclose an electronic book control method and device, electronic equipment and storage medium. The method can be applied to electronic equipment, and includes: performing face detection on a photographed image by using a face detection model to obtain a face detection result; if it is determined according to the face detection result that the image includes a face region, starting a human eye detection model, and performing human eye detection on the face region by using the human eye detection model to obtain a human eye detection result; determining a first coordinate of human eye fixation in an electronic page displayed by the electronic equipment according to the human eye detection result by using a line-of-sight estimation model; and controlling the electronic page to perform a corresponding reading operation according to the first coordinate. By implementing the embodiments of the present application, the power consumption of the electronic equipment when running the electronic book can be reduced, and the reading time of the electronic book can be prolonged.

Description

Technical Field

[0001] This application relates to the field of electronic equipment technology, specifically to a control method and device for an e-book, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of electronic device technology, e-books, which can be read on electronic devices, have emerged and are becoming increasingly popular due to their convenience and environmental friendliness.

[0003] In practice, it has been found that, unlike paper books, e-books can only be read on electronic devices. Therefore, how to reduce the power consumption of electronic devices when running e-books in order to increase the reading time of e-books has become an urgent problem to be solved. Summary of the Invention

[0004] This application discloses an e-book control method and device, an electronic device, and a storage medium, which can reduce the power consumption of the electronic device when running an e-book, thereby increasing the reading time of the e-book.

[0005] The first aspect of this application discloses a control method for an e-book, applied to an electronic device, the method comprising:

[0006] The captured images are analyzed using a face detection model to obtain the face detection results.

[0007] If the face detection results indicate that the image contains a face region, then the eye detection model is activated, and the face region is subjected to eye detection using the eye detection model to obtain the eye detection results.

[0008] Based on the human eye detection results, the first coordinates of the human eye's gaze are determined in the electronic page displayed on the electronic device using a gaze estimation model.

[0009] The electronic page is controlled to perform the corresponding reading operation based on the first coordinate.

[0010] A second aspect of this application discloses a control device for an e-book, applied to an electronic device, the device comprising:

[0011] The first detection unit is used to perform face detection on the captured first image using a face detection model to obtain face detection results.

[0012] The second detection unit is used to activate the human eye detection model when it is determined that the first image includes a human face region based on the human face detection result, and to perform human eye detection on the human eye region through the human eye detection model to obtain the human eye detection result.

[0013] The determining unit is used to determine the first coordinates of the human eye's gaze in the electronic page displayed by the electronic device based on the human eye detection results using a gaze estimation model;

[0014] The control unit is used to control the electronic page to perform the corresponding reading operation based on the first coordinate.

[0015] The third aspect of this application discloses an electronic device, including:

[0016] Memory containing executable program code;

[0017] A processor coupled to the memory;

[0018] The processor calls the executable program code stored in the memory to execute the e-book control method disclosed in the first aspect of the embodiments of this application.

[0019] The fourth aspect of this application discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the e-book control method disclosed in the first aspect of this application.

[0020] The fifth aspect of this application discloses a computer program product that, when run on a computer, causes the computer to perform some or all of the steps of any method of the first aspect of this application.

[0021] The sixth aspect of this application discloses an application publishing platform for publishing computer program products, wherein when the computer program products are run on a computer, the computer performs some or all of the steps of any one of the methods of the first aspect of this application.

[0022] Compared with related technologies, the embodiments of this application have the following beneficial effects:

[0023] The e-book control method disclosed in this application can perform face detection on the captured image using a face detection model to obtain face detection results. Furthermore, the eye detection model is only activated when the face detection results determine that the captured image includes a face region, thus avoiding the eye detection model from being constantly active and increasing the power consumption of the electronic device. Further, the electronic device can perform eye detection on the face region using the eye detection model to obtain eye detection results. Then, the electronic device can use a gaze estimation model to determine the first coordinate of the user's gaze on the displayed electronic page based on the eye detection results, and control the electronic page to perform corresponding reading operations based on the first coordinate. Therefore, the e-book control method provided in this application can reduce the power consumption of the electronic device when running an e-book, increase the reading time of the e-book, and perform corresponding reading operations based on the area the user is looking at, thereby reducing the user's operations during e-book reading and improving the user's reading experience. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating an e-book control method disclosed in an embodiment of this application;

[0026] Figure 2 This is a schematic diagram disclosed in an embodiment of this application for illustrating the model recognition process;

[0027] Figure 3 This is a flowchart illustrating another e-book control method disclosed in an embodiment of this application;

[0028] Figure 4 This is a flowchart illustrating another e-book control method disclosed in the embodiments of this application;

[0029] Figure 5 This is a schematic diagram illustrating a page-turning operation as disclosed in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of an e-book control device disclosed in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] It should be noted that the terms "first," "second," "third," and "fourth," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0034] This application discloses an e-book control method and device, an electronic device, and a storage medium, which can reduce the power consumption of the electronic device when running an e-book, thereby increasing the reading time of the e-book.

[0035] The technical solution of this application will be described in detail below with reference to specific embodiments.

[0036] To more clearly illustrate the control method, apparatus, electronic device, and storage medium for the e-book disclosed in this application, an application scenario suitable for the control method of the e-book is first introduced. Optionally, the control method of the e-book can be applied to various electronic devices, including but not limited to portable electronic devices such as mobile phones and tablets, wearable devices such as smartwatches and smart bracelets, and desktop devices such as desktop computers and desktop TVs, without limitation.

[0037] Optionally, this e-book control method can be applied to the process of an electronic device running an e-book, the process of an electronic device running a browser page, or the process of other electronic devices displaying electronic pages; it is not limited here.

[0038] In practice, it has been found that with the rapid development of electronic device technology, e-books, which can now be read on electronic devices, are becoming increasingly popular due to their convenience and environmental friendliness. However, unlike paper books, e-books can only be read on electronic devices. Therefore, how to reduce the power consumption of electronic devices running e-books in order to increase the reading time has become an urgent problem to be solved.

[0039] To address this issue, this application provides an e-book control method. A face detection model can be used to perform face detection on captured images to obtain face detection results. Furthermore, the eye detection model is only activated when the face detection results determine that the captured image includes a face region, thus avoiding the situation where the eye detection model is constantly active, which would increase the power consumption of the electronic device. Further, the electronic device can use the eye detection model to perform eye detection on the face region to obtain eye detection results. Then, the electronic device can use a gaze estimation model to determine the first coordinate of the user's gaze on the displayed electronic page based on the eye detection results, and control the electronic page to perform corresponding reading operations based on the first coordinate. Therefore, the e-book control method provided in this application can reduce the power consumption of the electronic device when running an e-book, increase the reading time of the e-book, and perform corresponding reading operations based on the area the user is looking at, thereby reducing the user's operations during the e-book reading process and improving the user's reading experience.

[0040] Please see Figure 1 , Figure 1 This is a flowchart illustrating an e-book control method disclosed in an embodiment of this application. This method can be applied to the aforementioned electronic device or other executing entity, and is not limited thereto. The method may include the following steps:

[0041] 102. Perform face detection on the captured first image using a face detection model to obtain the face detection results.

[0042] In this embodiment of the application, the electronic device can capture one or more first images through the camera (including but not limited to the front camera, rear camera, etc.) of the electronic device while running the e-book. Then, the electronic device can input the first image into the face detection model to perform face detection on the captured first image to obtain the face detection result.

[0043] The face detection model may include, but is not limited to, DeepFace, DeepID, and FaceNet models. The face detection module may be trained using a large number of first sample images, which are images that include faces.

[0044] For a clearer explanation of the face detection process, please refer to [link / reference]. Figure 2 , Figure 2This is a schematic diagram illustrating the model recognition process disclosed in an embodiment of this application. After the first image 200 is input into the face recognition model, the face recognition model can output a face detection result 210. If a face exists in the first image 200, a face detection box 220 will be present in the face detection result 210, used to select the face region; conversely, if no face exists in the first image 200, the face detection box 220 will not be present in the face detection result 210.

[0045] In another embodiment, the electronic device may also capture one or more first images through its camera while running a browser page, and then perform face detection on the captured first images using a face detection model to obtain face detection results.

[0046] 104. If the face detection results determine that the first image captured contains a face region, then the eye detection model is activated, and the face region is subjected to eye detection through the eye detection model to obtain the eye detection results.

[0047] As described above, if a face exists in the first image 200, a face detection box 220 will exist in the face detection result 210. If the electronic device determines that the face detection result includes a face detection box, the electronic device can determine that the captured first image includes a face region.

[0048] In this embodiment, the human eye detection model may include, but is not limited to, models such as OpenCV, for detecting the human eye region in an image. Similarly, the human eye detection model may be trained using a large number of second sample images, which are images including human eyes.

[0049] In one embodiment, after capturing a first image, the electronic device can first activate a face detection model to perform face detection on the captured first image. Before determining that the first image includes a face region, the electronic device can disable the eye detection model to avoid the eye detection model being constantly active and increasing the power consumption of the electronic device. Furthermore, if the face detection result obtained by the face detection model determines that the first image includes a face region, the electronic device can activate the eye detection model to detect the eye region within the face region.

[0050] Please refer to section 2 again. The electronic device can perform eye detection on the face region in the face detection result 210 using the eye detection model to obtain the eye detection result 230. Optionally, the eye detection result 230 may include both the left eye image and the right eye image.

[0051] 106. Based on the human eye detection results, determine the first coordinate of the human eye's gaze on the electronic page displayed on the electronic device using the gaze estimation model.

[0052] In this embodiment, the gaze estimation model can include, but is not limited to, various models used for gaze estimation such as the Gaze Estimation model and the Gaze AI model. Gait estimation can be understood as the process of determining the region of human gaze in a human eye image based on the human eye image.

[0053] Please refer to 2 again. Assuming that the pupils of the human eyes are all in the middle in the human eye detection result 230, the gaze estimation model can estimate the first coordinate 250 of the human eye looking at the electronic page 240 based on the human eye detection result 230, and the first coordinate 250 is located in the middle of the electronic page 240.

[0054] Understandably, the gaze estimation model can actually determine a more precise first coordinate of 250° based on the pupil's position, the angle between the pupil and the vertical direction, and the angle between the pupil and the horizontal direction. Figure 2 This is only for the purpose of illustrating the general process of line-of-sight estimation and should not be construed as limiting the embodiments of this application.

[0055] 108. Control the electronic page to perform the corresponding reading operation based on the first coordinate.

[0056] Optionally, the electronic page can be divided into multiple trigger areas, and each trigger area can be bound to one or more reading operations. For example, the first trigger area can be bound to the zoom operation, the second trigger area can be bound to the page turning operation, and the third trigger area can be bound to the brightness adjustment operation, etc. There are no restrictions here.

[0057] Therefore, when the electronic device determines that the first coordinate is within a certain target trigger area, it can control the electronic page to perform the reading operation corresponding to the target trigger area.

[0058] By implementing the methods disclosed in the above embodiments, a face detection model can be used to perform face detection on the captured image to obtain face detection results. Furthermore, the eye detection model can be activated only when the face detection results determine that the captured image includes a face region, thus avoiding the eye detection model from being constantly active and increasing the power consumption of the electronic device. Further, the electronic device can use the eye detection model to perform eye detection on the face region to obtain eye detection results. Then, the electronic device can use a gaze estimation model to determine the first coordinate of the user's gaze on the electronic page displayed on the electronic device based on the eye detection results, and control the electronic page to perform corresponding reading operations based on the first coordinate. Therefore, the e-book control method provided in this application can reduce the power consumption of the electronic device when running an e-book, increase the reading time of the e-book, and perform corresponding reading operations based on the area the user is looking at, thereby reducing the user's operations during the e-book reading process and improving the user's reading experience.

[0059] Please see Figure 3 , Figure 3 This is a flowchart illustrating another e-book control method disclosed in this application. This method can be applied to the aforementioned electronic devices or other executing entities, and is not limited thereto. The method may include the following steps:

[0060] 302. Perform face detection on the captured first image using a face detection model to obtain the face detection results.

[0061] 304. If the face detection results determine that the first image captured contains a face region, then the eye detection model is activated, and the face region is subjected to eye detection through the eye detection model to obtain the eye detection results.

[0062] In this embodiment, considering that the assistive functions for electronic pages are more suitable for the elderly, optionally, after the electronic device determines that the captured first image includes a face region based on the face detection results, it can extract features from the image corresponding to the face region to obtain face features; then, when the electronic device determines that the extracted face features match the first feature, it can activate the human eye detection model; wherein, the first feature includes image features corresponding to the elderly user group, including but not limited to image features corresponding to wrinkles, image features corresponding to age spots, etc., which are not limited here.

[0063] Optionally, the electronic device can extract features from the image corresponding to the face region using an image feature extraction algorithm to obtain face features. Optionally, the image feature extraction algorithm may include, but is not limited to, the HOG (Histogram of Oriented Gradient) algorithm, the LBP (Local Binary Pattern) algorithm, etc.

[0064] By implementing the above method, the eye detection model can be activated only when the user of the electronic device is determined to be an elderly user group. This avoids the electronic device from subsequently providing reading assistance functions to users who do not need them, thereby improving the user experience while reducing the power consumption of the electronic device.

[0065] In another embodiment, considering that features such as page zoom are more suitable for users with myopia who do not wear glasses, the electronic device can optionally extract features from the image corresponding to the face region to obtain face features, and then further determine whether the face features include a second feature; only when it is determined that the face features do not include the second feature will the eye detection model be activated. The second feature includes image features corresponding to the wearing of glasses.

[0066] By implementing the above method, electronic devices can activate the human eye detection model only when they determine that the user is not wearing glasses. In other words, if the electronic device determines that the user is wearing glasses, it means the user can clearly see the page information and does not need auxiliary magnification. In fact, forcibly providing magnification would interfere with the user's normal reading. In this case, the electronic device can choose not to activate the human eye detection model, thereby improving the user experience while reducing the power consumption of the electronic device. Conversely, if the device detects that the user is not wearing glasses, it can activate the human eye detection model to perform subsequent auxiliary functions, thus improving the user experience.

[0067] In another embodiment, after determining that the facial features include the second feature, the electronic device can determine whether the second feature matches the third feature, which includes the shape features corresponding to the smart glasses bound to the electronic device. If the second feature matches the third feature, the electronic device can send a wireless connection establishment request to the smart glasses. After establishing a wireless connection with the smart glasses, the electronic device can collect real-time images of the user's eyes through the smart glasses and input the real-time images of the eyes into a gaze estimation model to determine the first coordinates of the gaze in the electronic page displayed on the electronic device.

[0068] By implementing the above method, when the electronic device determines that the user is wearing smart glasses bound to the electronic device based on the face detection results, it can obtain more accurate human eye images through the smart glasses. This can improve the accuracy of the subsequently determined first coordinates without activating the human eye detection model and reducing the power consumption of the electronic device.

[0069] 306. Based on the human eye detection results, determine the first coordinate of the human eye's gaze on the electronic page displayed on the electronic device using the gaze estimation model.

[0070] 308. Determine the target area on the electronic page based on the first coordinate, and zoom in on the page information included in the target area.

[0071] In one embodiment, after displaying an electronic page, the electronic device can identify a valid area in the electronic page that includes page information (including but not limited to: text information, image information, etc.). The electronic device can then bind the valid area to a zoom-in operation, so that when the electronic device determines that the first coordinate is in the valid area, it can zoom in on the target area where the first coordinate is located.

[0072] Optionally, the target area can be a rectangular area, a circular area, etc., without limitation. The area of ​​the target area can be set by the developer or the user, without limitation. The electronic device can zoom in on the page information included in the target area. The zoom factor can be set by the developer or the user, without limitation.

[0073] In one embodiment, the magnification factor can increase as the duration of the first coordinate increases. It is understood that if the magnification factor for the target area is insufficient and the user cannot see it clearly, the user will continue to focus on the first coordinate to try to see the page information included in the target area. Therefore, the electronic device can gradually increase the magnification factor as the user continues to focus on the first coordinate, so that the user can clearly see the page information included in the target area. Optionally, the electronic device can increase the magnification factor by a first factor (e.g., 1x, 2x, etc.) for every additional unit of time (e.g., 5 seconds, 10 seconds) of the duration of the first coordinate.

[0074] Implementing the above method can avoid situations where users cannot see the page information due to insufficient magnification. In addition, considering that using a large magnification at the beginning may result in too many electronic pages being obscured, a gradual magnification method is adopted to avoid users reading and referring to areas other than the magnified area.

[0075] In another embodiment, after determining the target area in the electronic page according to the first coordinates, the electronic device can determine the complexity corresponding to the page information included in the target area. Then the electronic device can determine a target magnification to be applied according to the complexity corresponding to the page information, and magnify the page information included in the target area by the target magnification, wherein the target magnification can be positively correlated with the complexity corresponding to the page information.

[0076] It should be noted that the complexity corresponding to the page information can be understood as the richness of the content included in the page information. For example, in one embodiment, the page information may include text information, and the complexity corresponding to the text information can be positively correlated with the number of strokes of the text information, that is, the more the number of strokes of the text information, the higher the complexity corresponding to the text information. For example, assuming that the text information includes the Chinese character "大", it only has 3 strokes, while if the text information includes the Chinese character "翼", it has 17 strokes. It can be seen that the complexity of "翼" is obviously higher than that of "大", and accordingly the electronic device can magnify "翼" by a larger multiple to facilitate reading by the user.

[0077] In another embodiment, the page information may include image information, and the complexity corresponding to the image information can be positively correlated with the number of image objects included in the image information, that is, the more image objects are included in the image information, the higher the complexity corresponding to the image information. The image objects may include different types of objects such as background, people, vehicles, flowers and trees. It can be understood that if the number of image objects included in the image information is larger, the content depicted in the image is also more complex, and accordingly the electronic device can magnify the image information by a larger multiple, so as to facilitate the user to view the details in the image information.

[0078] By implementing the method disclosed in the above embodiments, the electronic device can adaptively adjust the magnification of the page information according to the complexity corresponding to the page information, which improves the intelligence of the electronic device, and also enables the user to read the magnified page information more clearly, thereby improving the user's experience.

[0079] In one embodiment, in order to avoid automatically magnifying the page information in an area when the user merely glances at the effective area, the electronic device can be provided with a plurality of auxiliary modes, including but not limited to a magnification auxiliary mode, a page turning auxiliary mode, a brightness adjustment auxiliary mode, etc. Wherein, when the electronic device is in the magnification auxiliary mode, the user can control the electronic device to perform a magnification operation through line of sight; similarly, in the page turning auxiliary mode, the user can control the electronic device to perform a page turning operation through line of sight, and in the brightness adjustment auxiliary mode, the user can control the electronic device to perform a brightness adjustment operation through line of sight.

[0080] Optionally, when users need to zoom in on page information, they can turn on the zoom-in assist mode of the electronic device. Then, after the electronic device determines the first coordinate of the human eye's gaze on the electronic page, if it is determined that the electronic device is in zoom-in assist mode, it can determine the target area on the electronic page based on the first coordinate and zoom in on the page information included in the target area.

[0081] By implementing the above method, electronic devices can only magnify page information when in magnification assistance mode, thereby improving the controllability of page magnification, page turning and other auxiliary functions, avoiding unnecessary auxiliary operations from affecting the user's normal reading, and reducing the power consumption of electronic devices.

[0082] Understandably, the gaze estimation model used to determine the initial coordinates of a user's gaze may have performance defects due to the influence of the quality and quantity of training data and the algorithm itself. This can lead to a deviation between the determined initial coordinates and the actual coordinates of the user's gaze. The most obvious manifestation of this in application is that when a user gazes at a certain area and wants to zoom in on that area, the inaccurate initial coordinates determined by the gaze estimation model may cause the electronic device to zoom in on other areas, thus affecting the user's reading experience.

[0083] Optionally, when a user discovers a discrepancy between the magnified area of ​​the electronic device and the area they are looking at, they can perform a correction operation on the target area. When the electronic device detects the correction operation on the target area, it can output guidance information and display an auxiliary cursor on the electronic page (the auxiliary cursor can be displayed at any position on the electronic page, and it can be a light spot that flashes once every first time interval (e.g., 0.1 seconds, 0.2 seconds, etc.)). The guidance information can be used to guide the user to look at the auxiliary cursor.

[0084] Furthermore, the electronic device can capture the user's current facial image via a camera and determine the eye image within that facial image. Optionally, the electronic device can determine the user's current facial image from the captured image using the aforementioned facial detection model, and then determine the eye image from the user's current facial image using the aforementioned eye detection model. The electronic device can then use a gaze estimation model to determine the second coordinate of the user's gaze on the electronic page based on the determined eye image.

[0085] It's understandable that the user's actual gaze coordinates should be the third coordinate corresponding to the auxiliary cursor. However, due to biases in the gaze estimation model, there should be a discrepancy between the second and third coordinates. The electronic device can generate compensation information based on the distance between the second and third coordinates, and then adjust the gaze estimation model accordingly to obtain an optimized gaze estimation model that accurately estimates the user's gaze.

[0086] In another embodiment, considering the difficulty of adjusting the gaze estimation model, the electronic device may not need to optimize the gaze estimation model. Optionally, after determining the fourth coordinate of the human eye's gaze in the electronic page displayed on the electronic device based on the human eye detection results using the gaze estimation model, the electronic device can determine the first coordinate of the human eye's gaze based on the compensation information and the fourth coordinate, thereby reducing the difficulty of implementing the correction process.

[0087] By implementing the above method, the deviation of the first coordinate determined by the line-of-sight estimation model can be avoided through correction operations, which would cause the electronic device to magnify other areas and affect the user's reading. This improves the fault tolerance and controllability of the e-book's control method.

[0088] Implementing the methods disclosed in the above embodiments can reduce the power consumption of electronic devices when running e-books, increase the reading time of e-books, and perform corresponding reading operations according to the area the user is looking at, thereby reducing the number of operations the user needs to perform while reading e-books and improving the user's reading experience. Furthermore, it can only further activate the human eye detection model to detect eyes in the face region when it is determined that the user of the electronic device is an elderly user group, avoiding the electronic device from subsequently providing reading assistance functions to users who do not need them, thereby improving the user experience while reducing the power consumption of the electronic device. It can also improve the user experience while reducing the power consumption of the electronic device. Moreover, it can adaptively adjust the magnification of page information according to the complexity of the page information, improving the intelligence of the electronic device and allowing users to read the magnified page information more clearly, thereby improving the user experience. Finally, it can prevent deviations in the first coordinate determined by the gaze estimation model through correction operations, which could cause the electronic device to magnify other areas and affect the user's reading, thereby improving the fault tolerance and controllability of the e-book control method.

[0089] Please see Figure 4 , Figure 4 This is a flowchart illustrating another e-book control method disclosed in this application. This method can be applied to the aforementioned electronic devices or other executing entities, and is not limited thereto. The method may include the following steps:

[0090] 402. Perform face detection on the captured first image using a face detection model to obtain the face detection results.

[0091] 404. If the face detection results determine that the first image captured contains a face region, then the eye detection model is activated, and the face region is subjected to eye detection through the eye detection model to obtain the eye detection results.

[0092] In one embodiment, after the electronic device performs eye detection on the face region using an eye detection model and obtains the eye detection result, it can further determine whether the eyes in the first image are in a closed state based on the eye detection result. If the eyes in the first image are in a closed state, a second image is acquired through an imaging device, and face detection is performed on the captured second image using the face detection model to obtain a second face detection result. If the second face detection result determines that the second image includes a face region, the eye detection model is activated, and eye detection is performed on the face region using the eye detection model to obtain a second eye detection result.

[0093] Furthermore, if the second eye detection result determines that the human eye in the second image is in an open state, then the gaze estimation model determines the first coordinate of the human eye's gaze on the electronic page displayed on the electronic device based on the second eye detection result.

[0094] Understandably, gaze estimation models typically rely on pupil size to estimate eye gaze. Therefore, if the eyes in the first image are closed, the subsequent gaze estimation model cannot accurately determine the first coordinates of the eye's gaze based on the eye detection results. To address this, the method described above involves acquiring a second image when the eyes are confirmed to be closed, and only inputting the second image into the gaze estimation model when the eyes are open. This improves the accuracy of the first coordinates determined by the gaze estimation model.

[0095] In another embodiment, after determining whether the human eye in the first image is in a closed state based on the human eye detection result, multiple consecutive third images can be acquired by the imaging device; if the human eye in multiple consecutive third images is in a closed state, the electronic device can output an inquiry message, which is used to ask the user whether they need to continue reading the electronic page; then the electronic device can acquire a fourth image by the imaging device; if the human eye in the fourth image is in a closed state, the electronic device can close the electronic page and start a sleep state.

[0096] By implementing the above method, the electronic device can initially determine that the user is in a sleep state when the user's eyes are closed in multiple consecutive images. The electronic device can then output a query to determine whether the user still needs to read the electronic page. Furthermore, when the fourth image determines that the user is still in a closed-eye state, the electronic device can then enter a sleep state to reduce the power consumption of the electronic device.

[0097] 406. Based on the human eye detection results, determine the first coordinate of the human eye's gaze on the electronic page displayed on the electronic device using the gaze estimation model.

[0098] In one embodiment, the electronic page includes a browser's search page. Optionally, if the first coordinate is located in the search bar of the search page, the electronic device can activate a sound acquisition device to collect the user's input search information; then, the electronic device can perform a search operation based on the input search information.

[0099] By implementing the above methods, users can also perform quick searches on the search page using their gaze, reducing the need for hand operations and thus improving the user experience.

[0100] 408. Perform a page-turning operation on the electronic page based on the first coordinate and the page height of the electronic page.

[0101] As mentioned above, electronic devices can be equipped with various assistance modes, including a page-turning assistance mode. To avoid unnecessary page-turning assistance operations affecting the user's normal reading and increasing the power consumption of the electronic device, the electronic device can only perform page-turning operations based on the first coordinate and the page height when it is determined that the electronic device is in page-turning assistance mode.

[0102] In this embodiment of the application, after the electronic device displays the electronic page, it can determine the page turning trigger area corresponding to the page turning operation based on the page height of the electronic page (for example, the electronic device can determine the upper half of the electronic page as the upward page turning trigger area and the lower half of the electronic page as the downward page turning trigger area), and bind the page turning trigger area with the corresponding page turning operation. Then, if it is subsequently determined that the first coordinate is within the page turning trigger area, the electronic device can perform the corresponding page turning operation on the electronic page.

[0103] Optionally, the page-turning trigger area may include an upward page-turning trigger area and a downward page-turning trigger area, which are not limited here. Electronic devices can bind the upward page-turning trigger area to the upward page-turning operation, and similarly, electronic devices can bind the downward page-turning trigger area to the downward page-turning operation.

[0104] Further, please refer to Figure 5 , Figure 5This is a schematic diagram illustrating a page-turning operation, as disclosed in an embodiment of this application. Optionally, the origin 500 of the coordinate system corresponding to the first coordinate can be located at the upper left corner of the electronic page 510, and the vertical axis of this coordinate system (i.e., Figure 5 The y-axis in the coordinate system points downwards and is parallel to the vertical direction of page 510. The horizontal axis of the coordinate system (i.e., Figure 5 The x-axis in the image points to the right and is parallel to the horizontal direction of page 510.

[0105] Optionally, the electronic device can display the vertical coordinate of the electronic page 510 as less than the first value ( Figure 5 (Taking 5 mm as an example) The first area 520 is determined as the page-up trigger area; the area with a vertical coordinate greater than the page height in the electronic device page 510 is... Figure 5 (Taking 100 mm as an example for explanation) and the target difference between the first value (i.e., 95 mm), the second region 530 is determined as the page-down trigger region.

[0106] It should be noted that, Figure 5 The control method for e-books disclosed in this application is for illustrative purposes only, and the construction of the coordinate system and the scale are not rigorous. Figure 5 The embodiments described in this application should not be construed as limiting.

[0107] Optionally, when the ordinate of the first coordinate is less than the first value, the electronic device can perform an up-page operation on the electronic page; and when the ordinate of the first coordinate is greater than the target difference (the target difference is the difference between the page height and the first value), the electronic page can perform a down-page operation.

[0108] The first value can be set by the developers or users according to the page height of the electronic page, but the first value should be less than the page height of the electronic device, typically 1 / 20 or 1 / 25 of the page height, etc., and is not limited here.

[0109] By implementing the above methods, electronic devices can assist users in turning pages based on their line of sight, thereby reducing manual operations and improving the user's reading experience. In addition, by setting the top area of ​​the electronic page as the trigger area for turning pages upward and the bottom area as the trigger area for turning pages downward, the page-turning trigger logic is more in line with the user's normal usage logic, further improving the user experience.

[0110] By implementing the methods disclosed in the above embodiments, the power consumption of electronic devices running e-books can be reduced, and the reading time of e-books can be increased. At the same time, corresponding reading operations can be performed according to the area that the user is looking at, thereby reducing the user's operations during the e-book reading process and improving the user's reading experience. In addition, the user's gaze can be used to assist the user in turning pages, thereby reducing the user's manual operations and improving the user's reading experience. Furthermore, by setting the upper area of ​​the electronic page as the trigger area for turning pages upward and the lower area of ​​the electronic page as the trigger area for turning pages downward, the page turning trigger logic is more in line with the user's normal usage logic, thereby improving the user experience.

[0111] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an e-book control device disclosed in an embodiment of this application. This device can be applied to the aforementioned electronic device or other executing entity, without limitation herein. The device may include a first detection unit 601, a second detection unit 602, a determination unit 603, and a control unit 604, wherein:

[0112] The first detection unit 601 is used to perform face detection on the captured first image using a face detection model to obtain face detection results.

[0113] The second detection unit 602 is used to activate the human eye detection model when it is determined that the first image includes a human face region based on the human face detection result, and to perform human eye detection on the human face region through the human eye detection model to obtain the human eye detection result.

[0114] The determining unit 603 is used to determine the first coordinate of the human eye's gaze in the electronic page displayed on the electronic device based on the human eye detection results using a gaze estimation model;

[0115] The control unit 604 is used to control the electronic page to perform the corresponding reading operation according to the first coordinate.

[0116] By implementing the above-described device, a face detection model can be used to perform face detection on the captured image to obtain face detection results. Furthermore, the eye detection model can be activated only when the face detection results determine that the captured image includes a face region, thus avoiding the eye detection model from being constantly active and increasing the power consumption of the electronic device. Further, the electronic device can use the eye detection model to perform eye detection on the face region to obtain eye detection results. Then, the electronic device can use a gaze estimation model to determine the first coordinate of the user's gaze on the electronic page displayed on the electronic device based on the eye detection results, and control the electronic page to perform corresponding reading operations based on the first coordinate. Therefore, the e-book control method provided in this application embodiment can reduce the power consumption of the electronic device when running an e-book, increase the reading time of the e-book, and perform corresponding reading operations based on the area the user is looking at, thereby reducing the user's operations during the e-book reading process and improving the user's reading experience.

[0117] As an optional implementation, the control unit 604 is also used to determine the target area in the electronic page according to the first coordinates, and to magnify the page information included in the target area.

[0118] By implementing the above device, the page information included in the target area that the user is looking at can be magnified, making it easier for the user to view the page information they are looking at, thereby improving the user experience.

[0119] As an optional implementation, the control unit 604 is also used to determine the complexity of the page information included in the target area; and to determine a target multiplier based on the complexity of the page information, and to magnify the page information included in the target area by the target multiplier, wherein the target multiplier and the complexity are positively correlated.

[0120] By implementing the above device, the magnification of page information can be adaptively adjusted according to the complexity of the page information, which improves the intelligence of electronic devices and allows users to read the magnified page information more clearly, thereby improving the user experience.

[0121] As an optional implementation, the page information may include text information, and the complexity of the text information is positively correlated with the number of strokes in the text information;

[0122] And / or, page information may include image information, and the complexity of the image information is positively correlated with the number of image objects included in the image information.

[0123] By implementing the above device, the magnification of page information can be adaptively adjusted according to the complexity of the page information, which improves the intelligence of electronic devices and allows users to read the magnified page information more clearly, thereby improving the user experience.

[0124] As an optional implementation method, Figure 6 The device shown may further include an output unit (not shown), an acquisition unit, a coordinate determination unit, and an adjustment unit, wherein:

[0125] The output unit is used to output guidance information and output an auxiliary cursor in the electronic page after the control unit 604 magnifies the page information included in the target area. If a correction operation for the target area is detected, the guidance information is used to guide the user to look at the auxiliary cursor.

[0126] The acquisition unit is used to acquire the user's current face image and determine the human eye image in the face image;

[0127] The coordinate determination unit is used to determine the second coordinates of the user's gaze in the electronic page based on the human eye image using a gaze estimation model;

[0128] The adjustment unit is used to generate compensation information based on the distance between the second coordinate and the third coordinate corresponding to the auxiliary cursor, and adjust the line-of-sight estimation model based on the compensation information to obtain an optimized line-of-sight estimation model.

[0129] By implementing the above-mentioned device, the deviation of the first coordinate determined by the line-of-sight estimation model can be avoided through the correction operation, which would cause the electronic device to magnify other areas and thus affect the user's reading. This improves the fault tolerance and controllability of the control method of the e-book.

[0130] As an optional implementation, the control unit 604 is also configured to perform a page-turning operation on the electronic page based on the first coordinates and the page height of the electronic page.

[0131] By implementing the above device, electronic devices can perform corresponding page-turning operations on electronic pages according to the user's gaze position, thereby reducing the user's hand operations during the reading process and improving the user's reading experience.

[0132] As an optional implementation, the origin of the coordinate system corresponding to the first coordinate is located at the upper left corner of the electronic page, the vertical axis of the coordinate system is downward and parallel to the vertical direction of the electronic page, and the horizontal axis of the coordinate system is to the right and parallel to the horizontal direction of the electronic page.

[0133] Furthermore, the control unit 604 is also configured to perform an upward page-turning operation on the electronic page when the vertical coordinate of the first coordinate is less than a first value, wherein the first value is less than the page height; and to perform a downward page-turning operation on the electronic page when the vertical coordinate of the first coordinate is greater than a target difference, wherein the target difference is the difference between the page height and the first value.

[0134] Implementing the above-mentioned device can assist users in turning pages based on their line of sight, thereby reducing manual operation and improving the user's reading experience. In addition, setting the upper area of ​​the electronic page as the trigger area for turning pages upward and the lower area as the trigger area for turning pages downward makes the page-turning trigger logic more in line with the user's normal usage logic, thus improving the user experience.

[0135] As an optional implementation, the second detection unit 602 is further configured to extract features from the portrait region to obtain facial features when it is determined that the first image includes a facial region based on the face detection result; and to activate the human eye detection model when it is determined that the facial features match the first feature, wherein the first feature includes image features corresponding to the elderly user group.

[0136] By implementing the above device, the eye detection model can be activated only when it is determined that the user of the electronic device is an elderly user group. This avoids the electronic device from subsequently providing reading assistance functions to users who do not need them, thereby improving the user experience while reducing the power consumption of the electronic device.

[0137] As an optional implementation method, Figure 6 The device shown also includes a start-up unit (not shown), wherein:

[0138] The activation unit is used to extract features from the portrait region by the second detection unit 602 to obtain facial features. If it is determined that the facial features do not include the second feature, then the human eye detection model is activated. The second feature includes the image features corresponding to wearing glasses.

[0139] Implementing the above device allows the eye detection model to be activated only when it is determined that the user is not wearing glasses. In other words, if the electronic device determines that the user is wearing glasses, it means the user can clearly see the page information and does not need auxiliary magnification. In fact, forcibly providing magnification would interfere with the user's normal reading. In this case, the electronic device can choose not to activate the eye detection model, thereby improving the user experience while reducing the power consumption of the electronic device. Conversely, if the device detects that the user is not wearing glasses, it can activate the eye detection model to perform subsequent auxiliary functions, thus improving the user experience.

[0140] Please see Figure 7 , Figure 7This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0141] like Figure 7 As shown, the electronic device may include:

[0142] Memory 701 storing executable program code;

[0143] Processor 702 coupled to memory 701;

[0144] The processor 702 calls the executable program code stored in the memory 701 to execute the e-book control method disclosed in the above embodiments.

[0145] This application discloses a computer-readable storage medium storing a computer program that causes a computer to execute the e-book control method disclosed in the above embodiments.

[0146] This application also discloses an application publishing platform, which is used to publish computer program products. When the computer program products are run on a computer, the computer performs some or all of the steps of the methods described in the above method embodiments.

[0147] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0148] In the various embodiments of this application, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0149] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-accessible memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of this application.

[0152] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0153] The foregoing has provided a detailed description of an e-book control method, apparatus, electronic device, and storage medium disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A control method for an e-book, characterized in that, Applied to electronic devices, the method includes: The face detection model is used to detect faces in the first captured image to obtain the face detection results. If the face detection results determine that the first image includes a face region, then feature extraction is performed on the face region to obtain face features; If the facial features do not include the second feature, then the human eye detection model is activated. The second feature includes the image features corresponding to wearing glasses. The human eye detection model is used to perform human eye detection on the facial region to obtain the human eye detection result. Based on the human eye detection results, the first coordinates of the human eye's gaze are determined in the electronic page displayed on the electronic device using a gaze estimation model. The electronic page is controlled to perform a corresponding reading operation based on the first coordinate. The method further includes: if the facial feature includes a second feature and the second feature matches a third feature, then a wireless connection establishment request is sent to the smart glasses, wherein the third feature includes the shape feature corresponding to the smart glasses bound to the electronic device; Upon requesting to establish a wireless connection with the smart glasses, real-time human eye images are acquired through the smart glasses, and the real-time human eye images are input into a gaze estimation model to determine the first coordinates of human eye gaze on the electronic page displayed on the electronic device.

2. The method according to claim 1, characterized in that, The step of controlling the electronic page to perform the corresponding reading operation based on the first coordinate includes: The target area is determined in the electronic page based on the first coordinate, and the page information included in the target area is magnified.

3. The method according to claim 2, characterized in that, The step of magnifying the page information included in the target area includes: Determine the complexity of the page information included in the target area; The target multiplier is determined based on the complexity, and the page information included in the target area is magnified by the target multiplier. The target multiplier and the complexity are positively correlated.

4. The method according to claim 3, characterized in that, The page information includes text information, and the complexity of the text information is positively correlated with the number of strokes in the text information. And / or, the page information includes image information, and the complexity of the image information is positively correlated with the number of image objects included in the image information.

5. The method according to any one of claims 2 to 4, characterized in that, After zooming in on the page information included in the target area, the method further includes: If a correction operation is detected targeting the target area, guidance information is output and an auxiliary cursor is displayed on the electronic page. The guidance information is used to guide the user to look at the auxiliary cursor. Obtain the user's current face image and determine the human eye image from the face image; The second coordinates of the user's gaze are determined on the electronic page based on the human eye image using the gaze estimation model. Compensation information is generated based on the distance between the second coordinate and the third coordinate corresponding to the auxiliary cursor, and the line-of-sight estimation model is adjusted based on the compensation information to obtain an optimized line-of-sight estimation model.

6. The method according to claim 1, characterized in that, The step of controlling the electronic page to perform the corresponding reading operation based on the first coordinate includes: Based on the first coordinates and the page height of the electronic page, a page-turning operation is performed on the electronic page.

7. The method according to claim 6, characterized in that, The origin of the coordinate system corresponding to the first coordinate is located at the upper left corner of the electronic page. The vertical axis of the coordinate system is downward and parallel to the vertical direction of the electronic page, and the horizontal axis of the coordinate system is to the right and parallel to the horizontal direction of the electronic page. And, the step of performing a page-turning operation on the electronic page based on the first coordinates and the page height of the electronic page includes: If the ordinate of the first coordinate is less than the first value, then perform an up-page operation on the electronic page, where the first value is less than the page height; If the ordinate of the first coordinate is greater than the target difference, then a page-down operation is performed on the electronic page, where the target difference is the difference between the page height and the first value.

8. The method according to any one of claims 1-4 and 6, 7, characterized in that, After extracting features from the face region to obtain facial features, the method further includes: If the facial features match the first feature, the human eye detection model is activated. The first feature includes image features corresponding to the elderly user group.

9. A control device for an e-book, characterized in that, Applied to electronic devices, the device includes: The first detection unit is used to perform face detection on the captured first image using a face detection model to obtain face detection results. The second detection unit is configured to extract features from the face region to obtain face features when the face detection result determines that the first image includes a face region; and if the face features do not include a second feature, to activate an eye detection model, wherein the second feature includes image features corresponding to wearing glasses, and to perform eye detection on the face region through the eye detection model to obtain an eye detection result. The determining unit is used to determine the first coordinates of the human eye's gaze in the electronic page displayed by the electronic device based on the human eye detection results using a gaze estimation model; A control unit is used to control the electronic page to perform a corresponding reading operation based on the first coordinate. The device is further configured to: if the facial features include a second feature and the second feature matches a third feature, send a wireless connection establishment request to the smart glasses, wherein the third feature includes shape features corresponding to the smart glasses bound to the electronic device; and, in the event of a wireless connection establishment request to the smart glasses, acquire real-time human eye images through the smart glasses and input the real-time human eye images into a gaze estimation model to determine the first coordinates of human eye gaze in the electronic page displayed on the electronic device.

10. An electronic device, characterized in that, The method includes a memory storing executable program code and a processor coupled to the memory; wherein the processor invokes the executable program code stored in the memory to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images