Reading assistant in browser environment

Abstract

A reading assistant tool implemented in a browser application facilitates presentation of textual content in a manner and format that meets the individual user's reading and / or understanding capabilities while maintaining the context of the original content. The reading assistant tool causes the reader view pane to be output alongside the primary content pane. Original content is presented in the main content pane, and simplified / reformatted content corresponding to textual content extracted from the original content is presented in the reader view pane. The scrolling of the extracted textual content in the reader view pane is synchronized with the scrolling of the original content in the primary content pane such that context is maintained when the user browses the content. The concurrent presentation of the original content and the extracted textual content allows the user to consume the extracted textual content more easily while also maintaining context as the original intent.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of and claims priority to U.S. application No. 17 / 931,283, filed on September 12, 2022, entitled “READING ASSISTANT IN A BROWSERENVIRONMENT,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The present specification relates generally to reading assistant tools for computing devices, and in particular to reading assistant tools that allow a user to maintain the context of original content while also facilitating the user's consumption of the content. Background Art

[0003] Some users face difficulties with reading and / or comprehension and / or learning and / or language. For example, conditions such as dyslexia, attention deficit disorder, and other such conditions may affect a user's ability to focus on, process, and / or comprehend content, such as content provided via a web-based application, a web-based source, etc. Existing reading modes replace the content with simplified content, e.g., provide simplified output or present text-intensive content. Summary of the invention

[0004] A reading assistant tool for enhancing reading experience and understanding of web-based content is provided according to the system and method of the implementation described herein. The reading assistant tool provides the output of text content in a manner and format that solves the reading and / or comprehension ability of individual users, while maintaining the context of the original content. The reading assistant tool can be implemented in a browser application to include a browser-based operating system. This can make the reading assistant tool available for substantially any content output by a browser application / browser-based operating system. The availability of the reading assistant tool implemented by the browser application / browser-based operating system can reduce or substantially eliminate the dependence on third-party applications that output content in this manner. Compared with third-party applications or programs, using a browser-level / operating system-level reading assistant tool can improve security because the content accessed by the user and the interaction between the user and the content are isolated within the system. The reading assistant tool can output a reader view that provides a modified or simplified version of the original content. The reading assistant tool can cause the reader view pane to be output together with the main content pane, i.e., next to the main content pane / adjacent to the main content pane, wherein the original content is presented in the main content pane, and the modified or simplified content is presented in the reader view pane. The modified or simplified content can include text content that has been extracted from the original content. The output of the extracted text content can be customized or modified or reformatted according to user settings or preferences to address the needs of a specific user. The concurrent output of the original content in the main content pane and the extracted text content in the reader view pane can enable the user to consume the extracted text content more easily while also maintaining the context as originally intended due to the concurrent output (in one example, side-by-side placement) with the original content. The scrolling of the extracted text content can be synchronized or anchored with the scrolling of the original content so that this continuity and context is maintained as the user browses the content.

[0005] In some aspects, the technology described herein relates to a computer-implemented method comprising: extracting, by a text extraction module of a browser application executed on a computing device, text content included in browser content displayed by the browser application; generating modified text content based on the extracted text content; rendering, by the browser application, a reader view to display the modified text content together with the browser content, comprising: rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content; and providing a reader view for display on a display device of the computing device, the reader view including the second content pane next to the first content pane.

[0006] In some aspects, the technology described herein relates to a computer-implemented method in which generating modified text content includes at least one of: modifying the format of the extracted text content based on previously stored user settings related to displaying the modified text content in a second content pane; or modifying the format of the extracted text content in response to user input detected at a reading assistant toolbar of a browser application.

[0007] In some aspects, the technology described herein relates to a computer-implemented method in which modifying the format of the extracted text content includes modifying at least one of the following: font type, font size, font color, background color, spacing, or line spacing of the extracted text content.

[0008] In some aspects, the technology described herein relates to the computer-implemented method of claim 3, further comprising: rendering browser content by a browser application; analyzing the browser content by a reading assistant module of the browser application; and determining that the browser content is associated with a distillable page in response to detecting extractable text content in the browser content.

[0009] In some aspects, the technology described herein relates to a computer-implemented method that further includes launching a reading assistant tool, including at least one of: launching the reading assistant tool in response to determining that browser content is associated with a refinable page; or launching the reading assistant tool in response to user input selecting the reading assistant tool from a browser application menu.

[0010] In some aspects, the technology described herein relates to a computer-implemented method that further includes: determining that browser content is associated with a non-refinable page; detecting user input selecting a portion of text content from the browser content; and launching a reading assistant tool in response to user input selecting the reading assistant tool from a browser application menu.

[0011] In some aspects, the technology described herein relates to a computer-implemented method in which generating modified textual content includes generating the modified textual content based on a portion of the textual content selected by a user input.

[0012] In some aspects, the technology described herein relates to a computer-implemented method of any preceding claim, the method further comprising: detecting a scroll input for scrolling modified text content displayed in a second content pane; scrolling the modified text content displayed in the second content pane in response to the scroll input; and scrolling browser content displayed in the first content pane in response to the scroll input.

[0013] In some aspects, the technology described herein relates to a computer-implemented method that further includes: detecting a scroll input for scrolling browser content displayed in a first content pane; scrolling the browser content displayed in the first content pane in response to the scroll input; and scrolling modified text content displayed in a second content pane in response to the scroll input.

[0014] In some aspects, the technology described herein relates to a computer-implemented method that scrolls browser content displayed in a first content pane, including synchronizing scrolling of the browser content displayed in the first content pane with scrolling of modified text content displayed in a second content pane.

[0015] In some aspects, the technology described herein relates to a computer-implemented method in which synchronizing scrolling of browser content displayed in a first content pane with scrolling of modified text content displayed in a second content pane includes: associating a first bounding box assigned to a text element in the browser content displayed in the first content pane with a second bounding box assigned to a corresponding text element in the modified text content displayed in the second content pane; and performing synchronization of the scrolling based on the association of the first bounding box with the second bounding box.

[0016] In some aspects, the technology described herein relates to a computer-implemented method in which generating modified text content includes: extracting the text content from browser content; and omitting at least one of image content, table content, links, input fields, or browser controls included in the browser content from the modified text content.

[0017] In some aspects, the technology described herein relates to a non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a computing device, cause the at least one processor to: extract, by a text extraction module of a browser application executed on the computing device, text content included in browser content displayed by the browser application; generate modified text content based on the extracted text content; render, by the browser application, a reader view to display the browser content together with the modified text content, including: rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content; and provide a reader view including the first content pane and the second content pane for display on a display device of the computing device.

[0018] In some aspects, the technology described herein relates to a non-transitory computer-readable medium in which instructions cause at least one processor to: detect previously stored user settings related to displaying modified text content in a second content pane; modify the format of extracted text content based on the previously stored user settings; detect user input at a reading assistant toolbar of a browser application; and modify the format of the extracted text content in response to the user input detected at the reading assistant toolbar.

[0019] In some aspects, the technology described herein relates to a non-transitory computer-readable medium or claim 14, wherein instructions cause at least one processor to: render browser content by a browser application; analyze the browser content by a reading assistant module of the browser application; and determine that the browser content is associated with a refinable page in response to detecting extractable text content in the browser content.

[0020] In some aspects, the technology described herein relates to a non-transitory computer-readable medium in which instructions cause at least one processor to launch a reading assistant tool, including at least one of: launching the reading assistant tool in response to determining that browser content is associated with a refinable page; or launching the reading assistant tool in response to user input selecting the reading assistant tool from a browser application menu.

[0021] In some aspects, the technology described herein relates to a non-transitory computer-readable medium in which instructions cause at least one processor to: determine that browser content is associated with a non-refinable page; detect user input selecting a portion of text content from the browser content; and launch a reading assistant tool in response to user input selecting the reading assistant tool from a browser application menu.

[0022] In some aspects, the technology described herein relates to a non-transitory computer-readable medium in which instructions cause at least one processor to generate modified text content based on a portion of the text content selected by a user input.

[0023] In some aspects, the technology described herein relates to the non-transitory computer-readable medium of claim 18, wherein the instructions cause at least one processor to: detect a scroll input for scrolling modified text content displayed in a second content pane; scroll the modified text content displayed in the second content pane in response to the scroll input; and scroll browser content displayed in the first content pane in response to the scroll input.

[0024] In some aspects, the technology described herein relates to a non-transitory computer-readable medium in which instructions cause at least one processor to: synchronize scrolling of browser content displayed in a first content pane with scrolling of modified text content displayed in a second content pane, including: associating a first bounding box assigned to a text element in the browser content displayed in the first content pane with a second bounding box assigned to a corresponding text element in the modified text content displayed in the second content pane; and synchronizing the scrolling based on the association of the first bounding box with the second bounding box.

[0025] In some aspects, the technology described herein relates to the non-transitory computer-readable medium of claims 13 to 20, wherein the browser content includes text content, and at least one of image content, table content, links, input fields, or browser controls, and wherein the instructions cause at least one processor to modify at least one of the font type, font size, font color, background color, spacing, or line spacing of the extracted text content.

[0026] In some aspects, the technology described herein relates to the non-transitory computer-readable medium of claim 21, wherein the instructions cause at least one processor to generate modified text content, including: extracting the text content from the browser content; and omitting at least one of image content, table content, links, input fields, or browser controls included in the browser content from the modified text content.

[0027] In some aspects, the technology described herein relates to a computing system comprising: a processor; a display; and a memory storing instructions that, when executed by the processor, cause a browser application executing on the computing system to perform operations, the operations comprising: extracting, by a text extraction module of the browser application, text content included in browser content displayed on the display; generating modified text content based on the extracted text content; rendering a reader view to display the browser content together with the modified text content, comprising: rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content; and providing a reader view including the first content pane and the second content pane for display on the display.

[0028] In some aspects, the technology described herein relates to a computing system in which instructions executed by a processor cause a browser application to generate modified text content, including: extracting the text content from browser content; and omitting at least one of image content, table content, links, input fields, or browser controls included in the browser content from the modified text content.

[0029] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1A and Figure 1B An example browser application user interface according to implementations described herein is illustrated.

[0031] Figure 2 is a block diagram of a system according to implementations described herein.

[0032] Figure 3A An example browser application user interface including content displayed in a browser content window is illustrated.

[0033] FIG. 3B to FIG. 3E Illustrated for use in Figure 3A An example alternative for launching the reading assistant tool in the example browser application user interface shown.

[0034] FIG. 3F to FIG. 3H The example is generated by the reading assistant tool and Figure 3A Features of a reader view presented in a browser content window of an example browser application user interface are shown.

[0035] Figure 4A and Figure 4B An example browser application user interface including content displayed in a browser content window is illustrated.

[0036] Figure 4C Illustrated in Figure 4A and Figure 4B Select the Reading Assistant tool in the sample browser application user interface shown.

[0037] Figure 4D The example is generated by the reading assistant tool and Figure 4A and Figure 4B Features of a reader view presented in a browser content window of an example browser application user interface are shown.

[0038] Figure 5 is a flow chart of an example method according to implementations described herein. DETAILED DESCRIPTION

[0039] Web-based content can include many content items (i.e., text content, image content, promotional / advertising content, links, browser control devices, text input fields, etc.). The arrangement of different types of content items may make web-based content difficult to consume, especially for users with reading and / or comprehension difficulties. Existing reading modes may be subject to certain limitations in their ability to extract content, i.e., text content, which is output in a form different from a typical text content structure (i.e., articles, blogs, etc.). In addition, existing reading modes often lose contact with the original content in the process of extracting text content into a reading view. Therefore, some users may lose some contexts such as the original intentions of the author / publisher and a certain degree of understanding of the content. A readily available reading assistant tool can improve the content consumption experience, especially for users with reading and / or comprehension and / or learning and / or language difficulties, and the reading assistant tool is effective in extracting content in various forms, and allows users to maintain the context of the original content while consuming the extracted content. The system and method according to the implementation described herein relate to a tool for providing improved reading and understanding of content output by a computing device. These tools can provide customization to how to present or output content, to adapt to the unique reading and / or understanding and / or learning and / or language ability of a specific user, etc. In some examples, the customization of content can include, for example, arranging content to eliminate redundant content items that are irrelevant to the reading and understanding of content. In some examples, customization can include, for example, modifying the appearance of text content, including, for example, font type, font size, font color, background color, spacing, interval and other such elements that can promote the user to the consumption of text content and understanding. According to the system and method of implementation described herein, by adapting to a wide range of users and for the reading and / or understanding and / or learning and / or language ability of the content of a wide range, improve the accessibility of a wide range of users to a wide range of content. Can make the reading assistant tool available to the user easily and easily via a browser, so that the reading assistant tool is available across the resources of a wide range, to further improve accessibility.

[0040] In some examples, the original, unmodified content can be output in the first pane, and the customized content can be output in the second pane displayed next to the first pane, the customized content including the text content that has been reformatted, or having the appearance that has been modified to facilitate the consumption of a specific user. The customized and / or reformatted text content output in the second pane next to the first pane displaying the original, unmodified content can provide and / or maintain context for the customized and / or reformatted text content. For example, content items such as images that are eliminated when arranging the original content can be visible at the same time as the text content that has been customized and / or reformatted to facilitate the consumption of the user. The scrolling of the customized and / or reformatted text content output in the second pane can be synchronized with the scrolling of the original, unmodified content output in the first pane, to retain context when the user scrolls through the content. In some cases, the original content may be output in a form that is not easy to refine. Therefore, in some examples, the system and method according to the implementation described herein can provide a selection of a part of the original content to be customized or reformatted as described above, and the customized and / or reformatted text content to be output in the second pane as described above.

[0041] Having the original, unmodified content presented alongside the modified text content can help users retain context, while also helping to ensure that publisher intent is not lost when the text content is modified. Customizing the output of content in this manner based on the user's reading and / or learning and / or language abilities can enhance user attention and / or facilitate user comprehension of the content, and improve accessibility to the content to a wide range of users with different reading and / or comprehension and / or language skill levels.

[0042] The systems and methods according to the implementations described herein can be incorporated into the operating system of a computing device. This can make the ability to customize the output of content in this manner available to users of computing devices across a wide range of web-based interfaces / applications in multiple separate browser processes without the use of third-party applications. The availability of reading assistant tools through browser applications / browser-based operating systems can reduce or substantially eliminate dependence on third-party applications, and can improve security compared to third-party applications or programs because the content accessed by the user and the user's interaction with the content are isolated within the system.

[0043] In some examples, controls may be provided to the user that allow the user to make choices about whether and when the systems, programs, or features described herein may enable the collection of user information. Such user information may include, for example, information related to the user's reading and / or learning and / or understanding and / or language skills associated with the use of the system and method; information related to the user's student / professional status; information related to the user's social network, social actions, or activities; the user's current location, and other such information. In some examples, the user information may include information related to the source and / or communication of content (e.g., content to be modified or other content), such as content or communication from a server. In addition, some data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (such as to a city, zip code, or state level) in the case of obtaining location information so that the user's specific location cannot be determined. In some examples, personal information associated with a user's reading and / or comprehension and / or learning and / or language skills, including settings and / or preferences related to modification of content, can be processed before it is stored or used so that such information cannot be associated with a specific user. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0044] Figure 1AAn example browser interface (UI) 100 according to the implementation described herein is shown. In some examples, the UI 100 is generated and rendered by a browser application executed in an operating system of a computing device. The example UI 100 includes a tab bar 102 associated with the browser application. In this example, the UI 100 includes a browser tag 104, which is only used for discussion and illustration purposes. The UI 100 may include any number of browser tags and / or tag groups that can be opened by the browser application. Content 110 (e.g., web content, web pages, online resources, web applications, etc.) may be presented in a corresponding browser content window 120 of the browser application. In some examples, the content 110 that may be generated or provided at least in part by and / or from an external resource may be rendered by the browser application, and the external resource may be, for example, a server, including, for example, a server hosting a website, a web application, or other external resources. In some examples, content as used herein may include a resource associated with a resource identifier, and the resource identifier may include, for example, a URL (Uniform Resource Locator), a URI (Uniform Resource Identifier), etc. Content may refer to the web page (resource) currently being rendered in the browser content window 120, as well as other data (e.g., metadata) used in rendering (e.g., tags and scripts that are not actually displayed), models generated for presenting the content 110 (e.g., document object model, accessibility model, etc.).

[0045] The example UI 100 includes an address bar area 106. The address of the web page displayed in the browser content window 120 can be shown in the address bar area 106. The resource identifier for the desired content can be entered (e.g., by a user) into the address input area 108 in the address bar area 106. Other controls, icons, etc. can be included in the address bar area 106. For example, a user icon 122 can be displayed in the address bar area 306 to provide an indication of a user account or user profile associated with the browser session. The user icon 122 can be an image, text, or some other representation of the user account. Other controls and / or icons can include, for example, a forward control, a back control, a refresh control, a home page control, an extension control, a save location control, a utility menu control icon 124, and other such controls and / or icons. In some examples, a scroll control icon 129 can be provided along one or more peripheral portions of the browser content window 120. User manipulation of the scroll control icon 129 can cause the content 110 presented in the browser content window 120 to scroll in a direction corresponding to the user manipulation of the scroll control icon 129. In some examples, a reading assistant control icon 126 may be included in the address bar area 106. In some examples, the reading assistant control icon 126 may be present within the address input area 108 or within other portions of the address bar area 106. The reading assistant control icon 126 may launch a reading assistant tool when selected by a user. The reading assistant control icon 126 may be just one way in which a reading assistant tool may be launched. In some examples, a side panel control icon 128 may be included in the address bar area 106. The side panel control icon 128 may cause a side panel or second viewing pane to be presented in the browser content window 120 when selected by a user, such as Figure 1B shown.

[0046] like Figure 1BAs shown, in some examples, the selection of the reading assistant control icon 126 can start the reading assistant tool and cause the first pane 130 and the second pane 140 to be displayed in the browser content window 120. In some examples, the original, unmodified content 110 can be displayed in the first pane 130. In some examples, the modified content 115 generated by the reading assistant tool based on the original, unmodified content 110 can be displayed in the second pane 140. In some examples, the modified content 115 can be generated based on user settings and / or preferences. The user settings and / or preferences can be based on various factors, such as the user's learning style, reading / comprehension ability, language skills, etc. As described above, modifying the original content 110 to generate the modified content 115 for presentation in the second pane 140 can include tidying up, such as removing images, etc., so that the remaining text content is presented in the modified content 115. The modification of the remaining text content may include modification of font size and / or style and / or color, spacing, spacing, background color, etc. In some examples, the scrolling of the modified content 115 in the second pane 140 may be synchronized with the scrolling of the original content 110 in the first pane 130. The positioning of the modified content 115 in the second pane 140 alongside the original content 110 in the first pane 130 and the synchronized scrolling of the modified content 115 and the original content 110 may allow the user to maintain context with the original content 110 while consuming / reading the modified content 115, which has been modified to accommodate the user's particular capabilities.

[0047] In some examples, selection of the side panel control icon 128 may cause the first pane 130 and the second pane 140 to be displayed in the browser content window 120, such as Figure 1B In some examples, the reading assistant control icon 126 may be available to the user in one or both of the first pane 130 or the second pane 140. The user may launch the reading assistant tool by selecting the reading assistant control icon 126 from, for example, the second pane 140. In this case, the original, unmodified content 110 may be presented in the first pane 130, and the modified content generated by the reading assistant tool as described above may be presented in the second pane 140.

[0048] Figure 22 is a block diagram of an example system 200 for providing a reading assistant tool in a browser application according to one aspect. The system 200 includes a computing device 210, which includes an operating system 220. The computing device 210 may also be referred to as a client computing device or a client device. In some examples, the computing device 210 may be, for example, a personal desktop computing device, a personal laptop computing device, a tablet computing device, a smart phone, a wearable computing device (e.g., smart glasses, smart watches, etc.) and other such computing devices including image output capabilities. The computing device 210 may include one or more output devices 212 that enable a user to view visual content and / or receive audio content. The example output device 212 may include, for example, a display device (e.g., a monitor, a touch screen, etc.) that enables a user to view displayed content and / or interact with displayed content, an audio output device (e.g., a speaker, etc.). The computing device 210 may include one or more input devices 214 that allow a user to, for example, interact with content output by the computing device 210, input commands for execution by the computing device 210, etc. Example input devices 214 may include, for example, a keyboard, a mouse, a touch-sensitive display, a trackpad, a trackball, etc. In some examples, input device 214 may include an audio input device (e.g., a microphone) that detects audio signals for processing as user input by computing device 210. In some examples, one or more sensing devices of sensing system 216 may be used as input device 214. For example, one or more image sensors may detect user gesture input for processing as user input by computing device 210. One or more position sensors may detect user gesture input (i.e., changes in position and / or orientation of computing device 210) for processing as user input by computing device 210.

[0049] The computing device 210 may include one or more processors 218. The one or more processors 218 may be implemented as a CPU and / or GPU formed in a substrate, which is configured to execute one or more machine executable instructions or software segments, firmware, or a combination thereof. The one or more processors 218 may be semiconductor-based processors containing semiconductor materials that can execute digital logic. The computing system may include one or more communication modules 219 that provide communication between the computing device 210 and other external systems and resources. The computing device 210 may include one or more memory devices 215. The memory device 215 may include a main memory that stores information in a format that can be read and / or executed by one or more processors 218. The memory device 215 may store applications or modules, such as an operating system 220, one or more applications 230 including a browser application 232, and the like, which perform certain operations when executed by one or more processors 218.

[0050] The operating system 220 may be implemented in system-level software that manages computing system hardware and software resources, and provides common services for computing programs executed by the computing device 210. In some examples, the operating system 220 may be operable to run on personal computing devices such as laptop computing devices, desktop computing devices, convertible computing devices, etc. In some examples, the operating system 220 may be operable to run on mobile computing devices such as tablet computing devices, handheld computing devices (such as smart phones), wearable computing devices (such as smart glasses, smart watches), etc. The operating system 220 may include multiple modules that may provide common services and manage resources of the computing device 210.

[0051] As described above, computing device 210 may include one or more applications 230 representing software that is programmed to perform different functions. One of applications 230 may be a browser application 232. Browser application 232 may display web pages, execute web applications, etc. In some examples, browser application 232 may include additional functionality in the form of extensions. In some examples, browser application 232 may also be operating system 220 of computing device 210. The browser application may generate and output a user interface, such as Figure 1A and Figure 1B The example UI 100 is shown. The browser application 232 may include one or more modules that may facilitate the functionality of the browser application 232. In the example system 200 including the reading assistant tool Figure 2In the example shown, the browser application 232 may include, for example, a reading assistant module 233, a text extraction module 235, a formatting module 237, a synchronization module 239, and other such modules that may facilitate the operation of the reading assistant tool via the browser application 232. Incorporating the reading assistant module 233, the text extraction module 235, the formatting module 237, the synchronization module 239, etc. into the browser application reduces or substantially eliminates reliance on third-party applications and improves security compared to third-party applications or programs because the content accessed by the user and the user's interaction with the content are isolated within the system.

[0052] In response to starting the reading assistant module 233, the reading assistant module 233 and the text extraction module 235 can be accessed to analyze the content output by the computing device 210. The text extraction module 235 may include one or more models, such as machine learning models, which are trained and configured to analyze the content (e.g., web pages, documents, or other such content) accessed by the computing device 210 and output by one of the output devices 212. In some examples, the text extraction module 235 can be configured to analyze the content to detect attributes associated with the text content, such as titles, paragraphs, and other such attributes associated with the text content, and extract the text content in the content output by the computing device 210. The formatting module 237 can organize the text content extracted by the text extraction module into a modified form that is easier for the user to consume. For example, the formatting module 237 can apply user settings and / or preferences to the extracted text content so that the text content can be more easily consumed by the user. The user settings and / or preferences may include, for example, the selected font type and / or size and / or color, spacing, and interval to be applied to generate the modified text content to be output by the computing device 210. In some examples, a user may choose to save formatting preferences for modifications to text content to a user profile or account accessible to formatting module 237 to eliminate selection of user preferences each time text content is to be extracted and modified.

[0053] As described above, the modified text content may be output by computing device 210 alongside the original content, such as Figure 1B As in the example shown, a first pane 130 and a second pane 140 are included. The synchronization module 239 can control the scrolling of the original, unmodified content and the modified content presented side by side so that the scrolling is synchronized between the original, unmodified content and the modified content. This can allow the user to maintain the connection between the original, unmodified content and the modified content, and retain the context with the original, unmodified content, for example, as intended by the publisher. Retaining the context in this way can improve the user's understanding of the content.

[0054] In some examples, content may be provided to computing device 210 from additional resources 250 accessible to computing device 210. In some examples, computing device 210 may access additional resources 250 via network 240. In some examples, additional resources are available locally on computing device 210. In some examples, some of additional resources 250 are available to computing device 210 via network 240, and some of additional resources are available locally on computing device 210. In some examples, resources associated with one or more of reading assistant module 233, text extraction module 235, formatting module 237, and / or synchronization module 239 may be available in additional resources 250. In some examples, additional resources 250 may include, for example, one or more server computing systems 252, one or more processors 254, one or more memory devices 256, and one or more databases 258. The one or more server computing systems 252 may include one or more computing devices, including, for example, a standard server, a group of such servers, a rack server system, and the like. In some examples, computing device 210 may access content from, for example, one or more server computing systems 252 available via additional resources 250. The content may include, for example, web-based content, including content available via a website or the like.

[0055] Figure 3A An example browser interface (UI) 300 is illustrated according to implementations described herein. The example UI 300 may be accessed by a computer (e.g., the computer described above with respect to Figure 2 The operating system of the computing device 210 or other computing device described above (e.g., Figure 2 220 described above or other such operating systems) executed in a browser application (e.g., Figure 2 The browser application 232 described above or other such browser application) generates and renders. In the following, for the purpose of discussion and illustration only, reference will be made to the above-mentioned Figure 2 The systems and methods are described using the system 200 of the example computing device 210 described above. The systems and methods described herein may be implemented by other computing systems having similar components as described above with respect to Figure 2 The components described may be arranged similarly or differently.

[0056] In this example, UI 300 includes a browser tab 304, which is used for discussion and illustration purposes only. In this example, browser content 310 in the form of an article provided in a web page is presented in a browser content window 320 associated with browser tab 104. In some examples, browser content 310 can be provided by, for example, Figure 2The depicted additional resources 250 (eg, a server hosting a website or web application) are provided and rendered by a browser application for output by a computing device. Figure 3A The illustrated example UI 300 includes an address bar area 306 in which the address (ie, URL) of a web page displayed in a browser content window 320 is shown in an address input area 308 of the address bar area 306. Figure 3A In the example shown, a user icon 322 is displayed in the address bar area 306. The user icon 322 can identify a user account or user profile associated with the current browser session. The utility menu control icon 324 can provide access to auxiliary actions and tools. In some examples, a scroll control icon 329 can be provided along one or more peripheral portions of the browser content window 320. User manipulation of the scroll control icon 329 can cause the content 310 presented in the browser content window 320 to scroll in a direction corresponding to the user manipulation of the scroll control icon 329.

[0057] In some examples, reading assistant control icon 326 may be included in address bar area 306, for example, included in address input area 308 or in other parts of address bar area 306. Selection of reading assistant control icon 326 is a way to start reading assistant tool. In some examples, side panel control icon 328 may be included in address bar area 306. Selection of side panel control icon 328 may cause side panel or second viewing pane to be presented in browser content window 320. In some examples, reading assistant tool may be started via selection of side panel control icon 328. That is, in some examples, reading assistant control icon 326 may be included in the side panel presented in response to the selection of side panel control icon 328, so that reading assistant tool may be started from the side panel.

[0058] exist Figure 3A In the example shown, example browser content 310 in the form of an article presented in a web page is presented in a browser content window 320. The example content 310 includes text content 312, image content 315, and table content 318 arranged in the browser content window 320. In this example, the image content 315 includes an image presented within the content 310 and located adjacent to the text content 312 to which it is related. The table content 318 includes menu items or links that provide access to additional information related to the content 310 presented in the browser content window 320. Users, particularly users who experience reading and / or comprehension and / or learning and / or language difficulties, may find the content 310 difficult to understand in the way it is presented. Figure 3AThe text content 312 is presented in the form presented in the example browser content window 320 shown. The text content 312 is arranged in sections or paragraphs, where headings mark the beginning of these sections. The sections of the text preceded by the headings can be extracted by a text extraction module such as the above Figure 2 The depicted text extraction module 235 detects and thereby identifies at least the portion of the content 310 as text content 312. This may allow the text extraction module 235 to relatively easily separate the text content 312 from image content 315, table content 318, icons representing browser controls, links, text entry fields, and any other content that may be presented in the browser content window 320 that is not clearly / easily extractable text content.

[0059] In general, browser content can include text content, image content, table content, links, input fields, browser controls, and other types of content. In some examples, text-intensive parts of web-based content can be identified as main content intended for user consumption. Text extraction module 235 can identify main content based on text characters arranged in titles, paragraphs, sentences, etc., for example. The text content extracted from these text-intensive parts (main content) by text extraction module 235 can be processed by a reading assistant tool to produce modified or simplified text content that is easier for users to consume. Web pages that mainly include text-intensive content, such as web pages where most of the content of the page is text-intensive content, can be considered to be refinable pages. In contrast, web pages that include content such as table content 318, icons, browser controls, text input fields, etc. can be considered to be non-refinable pages. Even if some of these content items may include some text (e.g., characters, words, phrases, etc. that identify content items), extracting the text and presenting it may not produce useful results for users. In some implementations, a machine learning model can be used to determine whether a web page is considered to be a refinable page. A machine learning model can be trained, for example using supervised or semi-supervised training, to take as input an accessibility tree (e.g., nodes of the tree) and provide as output an indication of which nodes, if any, can be considered primary content and / or a prediction of whether a web page is a distillable page.

[0060] As explained above, a user may wish to consume content in a form that is adjusted or customized to the user's preferences and viewing, reading, and understanding capabilities, such as Figure 3AThe example content 310 shown makes the content 310 easier to consume and understand. Therefore, the user can choose to launch the reading assistant tool to refine the content 310. In some examples, the page type associated with the web page content presented in the example web browser UI 300 can be determined to determine that the example content 310 includes refinable content. Based on the identification of the portion of the text content 312, the web page presented in the example browser UI 300 can be determined as a refinable page. The detection can be determined by a machine learning model. The user can launch the reading assistant tool from within the browser in a variety of different ways. In some examples, the reading assistant tool can be launched in response to a user selection of the reading assistant control icon 326. As Figure 3B As shown, selection of the reading assistant control icon 326 may launch a reading assistant tool in a side panel located beside / alongside the main panel displaying the content 310. In other words, the original content 310 may be displayed in a first content pane 330 presented in the browser content window 320, and the side panel may represent a second pane in which other content may be presented, including modified content generated based on the original content 310. In some examples, the reading assistant tool may be launched in response to a user selection of the side panel control icon 328 and a user selection of the reading assistant control icon 326 from a menu 342 available in the resulting side panel, such as Figure 3C In some examples, the reading assistant tool may be launched in response to a user selection of a reading assistant tool from a menu 344 displayed in response to input from an input device (i.e., a right click on a mouse, a tap on a trackpad, an extended hold on a touch screen or touchpad, etc.), such as Figure 3D In some examples, the user may wish to consume only a portion of the text content 312 using the reading assistant tool. Figure 3E As shown, the reading assistant tool may be launched in response to a user selection (e.g., highlighting) of a portion of the text content 312 and a selection of the reading assistant tool from a menu 346 (e.g., a browser application menu) displayed in response to input from an input device. In some examples, the reading assistant tool may be launched in response to a user selection of the reading assistant tool from a browser application menu in the form of a drop-down menu of a utility menu control icon 324. In some examples, the reading assistant tool may be launched in response to a user selection manipulation of a reading assistant icon or key provided on an input device such as a keyboard or a touch screen interface.

[0061] In some examples, the reading assistant can be started according to the user setting or the user preference of setting of the user stored previously.For example, the reading assistant tool can be started automatically, that is, without the need for extra user action, or without the need for extra user intervention.For example, in some implementations, the reading assistant tool can be started automatically in response to navigating to some specified websites, for any / all websites including the content that can be refined by the reading assistant tool, in response to detecting the content of some types, etc. This type of preference can be pre-set, stored and associated with the user account or user profile accessible via user icon 322. In some examples, this type of setting can include the instruction of starting the reading assistant tool for the specific website specified by the user (for example, by URL, by name, etc.), for all websites with the content that can be refined by the reading assistant tool, for the content of some types specified by the user, for any obviously refinable content presented in the browser content window 320, etc. In some examples, the user can select the preference of how to customize or modify the content via the reading assistant toolbar 355 presented together with the content to be modified after starting the reading assistant tool. In some examples, preferences regarding the manner in which content is modified or customized may be pre-set and stored by a user and saved, for example, in association with a user account or user profile accessible via user icon 322 .

[0062] FIG. 3F to FIG. 3H An example reader view generated by a reading assistant tool of an example computing device 210 is illustrated. For example, the reader view may be generated by a reading assistant module 233 of a browser application 232 and output or displayed by one of the output devices 212 of the computing device 210. Figure 3F As shown, in some examples, launching the reading assistant tool can cause a first content pane 330 and a second content pane 340 to be displayed in the browser content window 320. In some examples, the original content 310 can be displayed in the first content pane 330. In some examples, the modified content 350 can be displayed in the second content pane 340. The modified content 350 can be a customized or modified presentation of the original content 310. As described above, the modified content 350 can be generated according to user settings and / or preferences. The user settings and / or preferences can be based on various factors, such as the user's learning style, the user's reading and / or comprehension ability, language skills, etc. Figure 3F and Figure 3G In the example shown, when generating the modified content 350, the original content 310 has been sorted to remove the image content 315, so that the modified content 350 presented in the second content pane 340 only includes the modified text content 352. Figure 3F and Figure 3GIn the example shown, image content 315 (e.g., first image content item 315A, second image content item 315B, and third image content item 315C) and table content 318 have been eliminated when generating the modified content 350, so that the modified content 350 only includes modified text content 352 presented in a format that has been customized based on the user's preferences.

[0063] As described above, in some examples, modification of the original text content 312 to generate modified text content 352 can be done according to previously set and stored user preferences. The previously stored user preferences may include, for example, previously set selections for font type and / or size and / or color, background color, line spacing, etc. In some examples, the user can select how to customize or modify preferences for the original, unmodified content 310 via, for example, a reading assistant toolbar 355 included in the second content pane 340 after launching the reading assistant tool. Figure 3F and Figure 3G As shown, the example reading assistant toolbar 355 includes various icons that provide adjustments and selections of various features to be applied in the modification of the original text content 312. In some examples, the icons may include, for example, a font type selector 355A, font size selectors 355B and 355C, a font color and / or background color selector 355D, a line spacing selector 355E, a spacing selector 355F, etc. In some examples, the second content pane 340 includes an adjustment bar that includes an adjustment selector 357. The adjustment selector 357 may provide user adjustments to the size of the second content pane 340. For example, the user may manipulate the adjustment selector 357 (via, for example, an input device such as a mouse, a touch on a touch screen or a touch pad, etc.) to adjust the width of the second content pane 340 (and the corresponding width of the first content pane 330) as the input device is moved.

[0064] In some examples, a reading assistant tool (e.g., through operation of a reading assistant module of the browser application 232) can detect and extract text content from the original content 310 (e.g., through operation of a text extraction module 235 of the browser application 232). The extracted text can be processed according to the established accessibility tree to generate modified text content 352. In some examples, the accessibility tree contains semantic role and layout information for each tree node. Therefore, the accessibility tree can form the basis of a data model for extracting text content from the original content 310. Although the document object model (DOM) contains some semantic information, such as the text content of the document object model, the text content of the document object model can be extracted from the original content 310. The accessibility tree includes the identifier of the paragraph node defined by the tag, etc., but the accessibility tree contains more extensive semantic information, thereby providing more effective extraction and refinement of text content. Examples of semantic information included in the accessibility tree include accessible rich Internet applications (ARIA) landmarks, such as headers (called ARIA banners), footers (called ARIA content information) and main text. In addition, DOM does not contain layout information that defines the boundaries of each element located on the web page. These types of elements that can be provided by the accessibility tree are useful in machine learning model extraction to notify the positions of nodes related to each other. Modified text content 352 can be generated and presented in the second content pane 340 of the browser content window 320 according to user preferences (for example, according to selected user preferences for font type / size / color, background color / interval, spacing, etc. implemented via the formatting module 237 of the browser application 232).

[0065] exist Figure 3F and Figure 3G In the example shown, the modified content 350 presented in the second content pane 340 is positioned adjacent to or side-by-side with the original, unmodified content 310 in the first content pane 330. This side-by-side positioning of the first content pane 330 and the second content pane 340 may help the user maintain context between the modified content 350 and the original, unmodified content 310. As described above, in some examples, the scrolling of the modified content 350 in the second content pane 340 may be synchronized with the scrolling of the original, unmodified content 310 in the first content pane 330. The synchronization or anchoring of the modified content 350 with the original, unmodified content 310 may further enable the user to maintain context with the original, unmodified content 310 while scrolling and consuming the modified content 350, even when scrolling through the content 310, 350.

[0066] In some examples, the synchronization of the scrolling of the modified content 350 in the second content pane 340 with the original content 310 in the first content pane 330 can be implemented via, for example, the synchronization module 239 of the browser application 232. In some examples, the bounding box is assigned to the text element. The text element can include, for example, text characters, words, and parts of the content of the characters used mainly in the language of the content. The text element can include parts of the content, including individual characters, words, paragraphs, etc. In some examples, for example, as part of the rendering process, the bounding box is assigned to or associated with the text characters and / or words and / or parts of the text of the original text content 312 of the original content 310 presented in the first content pane 330. The bounding box can be represented as a node in a document object model (DOM) tree or an accessibility tree. The text of these bounding boxes can be assigned to or associated with the text characters and / or words and / or parts of the text of the modified text content 352 presented in the second content pane 340. Thus, the modified text content 352 may be anchored to (assigned to, associated with) one or more bounding boxes that are associated with corresponding portions of the original text content 312. With the text content 312 of the original content 310 and the modified text content 352 anchored in this manner, user manipulation of the scroll control icon 329 causes the modified text content 352 to be scrolled, and the original content 310 to be synchronously scrolled accordingly. The synchronous scrolling of the original content 310 includes scrolling the text content 312, and scrolling the image content 315 / image content items 315A, 315B, 315C, respectively. In some examples, scrolling may be achieved by user manipulation of other types of input devices, such as a keyboard's directional keys, a touch and drag input on a touchpad or touch screen, manipulation of a mouse's scroll wheel, and the like. Similarly, synchronous scrolling of content may be initiated in the original content 310 presented in the first content pane 330, and the modified text content 351 presented in the second content pane 340 may be scrolled together with the original content 310 in the first content pane 330.

[0067] exist Figure 3H In the example arrangement shown, the user has moved the scroll control icon 329 from Figure 3E and Figure 3F The first position shown is manipulated to Figure 3H The user manipulation of the scroll control icon 329 in this manner has caused the modified text content 352 presented in the second content pane 340 to be scrolled, and the original content 310 in the first content pane 330 to be correspondingly synchronously scrolled. Figure 3H In the scrolling arrangement shown, both the text content 312 and the image content 315 of the original content 310 have been scrolled in coordination with the scrolling of the modified text content 352, such that the first image content item 315A is no longer visible in the first content pane 330. In this example, the promoted content item 313 is now visible in the original content 310 presented in the first content pane 330 along with the additional portion of the original text content 312. The additional portion of the original text content 312 has been modified, reformatted, and included in the modified text content 352 presented in the second content pane 340; however, the promoted content item 313 is not included in the modified content 350 presented in the second content pane 340, consistent with the previously selected and stored user preferences.

[0068] In the above about 3A to 3H In the described examples, a web page includes content in the form of primary text content, which can be identified and analyzed, for example, using an accessibility tree as described above. For example, web pages presenting content such as articles, papers, blogs, etc. can be identified by analyzing accessibility tree information. These types of refinable web pages presenting this type of content can be refined by a reading assistant tool and presented in a reader view as described above. Some web pages do not include primary text content of this type. For example, some web pages include content that can be identified as table content or table information alone or together with other types of content that are not identified as primary text content (e.g., email interfaces, spreadsheet applications, calendar or scheduling pages, etc.). These types of non-refinable web pages can be processed differently by a reading assistant tool so that the results presented to the user in the reader view meet the needs of the user and allow the user to consume content more easily.

[0069] Figure 4A An example browser interface (UI) 400 representing a web page of an email application is illustrated. As with the example browser UI 300 described above, the example UI 400 can be accessed by a computer (e.g., the computer described above with respect to Figure 2 The operating system of the computing device 210 or other computing device described above (e.g., Figure 2 220 described above or other such operating systems) executed in a browser application (e.g., Figure 2 The example UI 400 includes a browser tab 404, wherein browser content 410 in the form of an email inbox is presented in a browser content window 420 associated with the browser tab 404, which is merely for ease of discussion and illustration. The browser content 410 can be generated by, for example, the browser application 232 described above with respect to Figure 2 The depicted additional resources 250 (eg, a server hosting a website or web application) are provided and rendered by a browser application for output by a computing device. Figure 4A The illustrated example UI 400 includes an address bar area 406, wherein the address of a web page (i.e., URL) is displayed in an address input area 408 of the address bar area 406. A user icon 422 is displayed in the address bar area 406 to identify a user account or user profile associated with the current browser session. A utility menu control icon 424 may provide access to auxiliary actions and tools. Scroll control icons 429 may be provided along one or more peripheral portions of the browser content window 420 for a user to scroll through the content 410 presented in the browser content window 420. In some examples, a reading assistant control icon 426 may be included in the address bar area 406. In some examples, in response to detecting non-refinable content output in the browser content window 420, the reading assistant control icon 426 may be omitted / not displayed. In some examples, a side panel control icon 428 may be selected to cause a side panel or a second viewing pane to be presented in the browser content window 420. The functions of the icons / controls provided in association with the example UI 400 may be similar to those described above with respect to Figures 3A to 3G The functions described herein are not described herein, and thus a repeated detailed description will be omitted.

[0070] exist Figure 4A In the example shown, example content 410 represents an email interface, including a list of email exchanges in a user's inbox. In some examples, the page type associated with the webpage content presented in the example web browser UI 400 can be determined to include non-refinable content to determine that example content 410 includes non-refinable content. The webpage presented in the example browser UI 400 can be determined as a non-refinable page based on a text-intensive part that can be refined into simplified content for user consumption that lacks text content. Existing reading mode may be unavailable, or cannot refine the content of this type, and therefore simplified, more easily consumed content presentation will not be provided to the user. According to implementation described herein, the reading assistant tool can still provide the user with the option of refining the content included in the webpage that is considered to be non-refinable page.

[0071] In response to selection of one of the content items within the content presented in the browser content window 420 (e.g., a link to a first email item in a list of email items in an inbox), example content 412 in the form of content of the selected email item is displayed in the browser content window 420, such as Figure 4B The accessibility tree is used by, for example, the text extraction module 235 of the browser application 232 to Figure 4A The example content 410 and Figure 4B By analyzing the example content 412 shown, it is possible to determine the content 410 of the web page (i.e., the email interface) and the content 412 including the selected email item and Figure 4B The various other links, input fields, controls, etc. shown are not easily resolvable. For example, due to the tabular format of the text, the list of links, input fields, controls, etc., Figure 4A The content 410 and / or Figure 4B Standard distillation of the content 412 shown is unlikely to produce modified text that is easily consumed by the user. In some examples, in response to determining that the web page displayed in the browser content window 420 is not easily distillable, the reading assistant control icon 426 can be omitted / not displayed in the address bar area 406 of the browser tab 404. Omitting the reading assistant control icon 426 in the address bar area 406 of the browser tab 404 can provide an indication to the user that the web page and associated content presented in the browser content window 420 are not distillable. Figure 4A The reading assistant control icon 426 is included for discussion and illustration purposes only.

[0072] As discussed above, users, particularly those who experience reading and / or comprehension and / or learning and / or language difficulties, may find that Figure 4A The content 410 and / or Figure 4B The illustrated content 412 is difficult to consume in the form in which it is presented in the example browser content window 420. The user may benefit from the ability to consume content 410 and / or content 412 in a form that is adjusted or customized to the user's preferences and abilities so that content 410 and / or content 412 may be more easily consumed and understood. In situations where a reading assistant tool cannot easily separate and / or extract text and / or easily distill the content into a form that is easily consumed by the user, the user may still apply the reading assistant tool to the relevant portions of the content. For example, Figure 4C As shown, the reading assistant tool can be launched in response to a user selection (e.g., highlighting) of a portion of content 412, such as a portion of content 412 that the user is particularly interested in. The user can select the reading assistant tool from a menu 446, such as a browser application menu, displayed in response to input from an input device (i.e., a right click on a mouse, a tap on a trackpad, an extended hold on a touch screen or trackpad, input of a shortcut key on a keyboard, etc.). In this example, in response to launching the reading assistant tool, a menu such as the following may be presented in the browser content window 420. Figure 4D The reader view shown. The reader view can be generated by the reading assistant module 233 of the browser application 232 and output or displayed by one of the output devices 212 of the computing device 210. In some examples, launching the reading assistant tool and generating the reader view can cause the first content pane 430 and the second content pane 440 to be presented side by side in the browser content window 420. The original, unmodified content 412 can be presented in the first content pane 430, and the customized or modified content 414 can be presented in the second content pane 440. In this example, the modified content 414 represents a customized or modified version of the example portion of the original content 412 selected by the user, such as Figure 4C shown.

[0073] As described above, in some examples, the user can select preferences for how to customize or modify the original content 412 to produce the modified content 414 after launching the reading assistant tool, for example, via the reading assistant toolbar 455 presented together with the content to be modified in the second content pane 440. In some examples, preferences related to the manner in which the content is modified or customized can be pre-set by the user. Such preferences can be saved, for example, in association with a user account or user profile accessible via the user icon 422. User preferences can be based on various factors, such as the user's learning style, the user's reading and / or comprehension ability, language skills, etc. Previously stored user preferences can include, for example, previously stored selections for font type and / or size and / or color, background color, line spacing, etc. In Figure 4D In the example shown, when generating modified content 414, the selected portion of original text content 312 is presented in a format that has been customized based on the user's preferences, and without additional input fields, control icons, links, etc. that would otherwise cause distraction and adversely affect the user's consumption of the content. In some examples, second content pane 440 includes an adjustment bar that includes an adjustment selector 457 for user adjustment of the size of second content pane 440. For example, the user can manipulate adjustment selector 457 (via, for example, an input device such as a mouse, a touch on a touch screen or touchpad, etc.) to adjust the width of second content pane 440 (and the corresponding width of first content pane 430) as the input device is moved.

[0074] Figure 5 5 is a flow chart of an example method 500 according to the implementation described herein. The method 500 may be performed by a browser application, such as the one described above with respect to Figure 2 The method 500 may be implemented, for example, by using the browser application 232 described above. Figure 2 The display device of the computing device of the computing device 210 described in the description provides the startup of the reading assistant tool and the output of the content in the reader view presented in the browser user interface. The reader view can allow the original, unmodified content and the extracted text content to be presented in a side-by-side reading pane, so that the user can keep the context of the original intention when consuming the extracted text content. The appearance of the extracted text content can be modified according to the user's preferences, so that the user can read and understand the extracted text content. The scrolling of the extracted text content can be synchronized or anchored with the scrolling of the original content in its original form, so as to keep the context of the original intention even when the content is scrolled. Extracting text content from the original content and modifying the appearance of the extracted text content can reduce visual distraction, and allow the user to view the extracted text content in a form that adapts to the user's specific abilities. The original, unmodified content and the extracted text content are presented side by side in a form that has been modified based on the user's preferences, and the original, unmodified content and the extracted / reformatted text content are synchronously scrolled, which can promote the user's reading and understanding of the text, while also maintaining enhanced understanding through the connection with the original, unmodified content.

[0075] The browser application can render content (frame 510) for outputting to the user via a display device such as a computing device. The content can be a presentation of a web page, including text content, image content, links, input fields, control devices, etc. The rendered content can be output in a browser content window of a browser user interface generated by the browser application. The browser application may include a reading assistant tool implemented by a reading assistant module of the browser application. The reading assistant tool can be started in a variety of different ways. In some examples, for any web page output by the browser application for a specific user, or for a specified web page, or for a specified type of content, the reading assistant tool can be actively started without a specific user input or intervention. In some examples, the user can initiate the start of the reading assistant tool, for example, by selecting a reading assistant control icon available in the browser user interface and / or in a side panel application in the browser content window and / or from an input device control menu and a browser control menu.

[0076] In response to launching the reading assistant tool (box 520), the content output by the browser application in the browser content window can be analyzed and text content can be extracted from the content (box 530). In addition to text content, the content can also include, for example, image content, links, input fields, control devices, and other such items. In some examples, the content can be analyzed using, for example, an accessibility tree to determine the type of web page presented in the browser user interface and the degree to which the associated content can be refined in its current form. The text content can be extracted by, for example, a text extraction module, such as the one described above with respect to Figure 2 The text extraction module 235 of the browser application 232 is described.

[0077] The extracted text can be customized or modified according to the preferences of a particular user to suit the user's reading and / or understanding and / or learning and / or language ability. The extracted text can be formatted by, for example, a formatting module such as described above with respect to Figure 2 The example formatting module 237 of the described browser application is formatted to incorporate preferences such as font type and / or size and / or color, background color, spacing, spacing, etc., which will make the extracted text easier for the user to consume. In some examples, these preferences can be pre-stored and associated with a user account or user profile.

[0078] The formatting module 237 can generate the modified extracted text content (frame 550) in response to detecting the previously stored modified preferences (frame 540) for the extracted text content. The modified extracted text content can be displayed in a reader view within the browser content window together with the original, unmodified content (frame 565). Specifically, the original, unmodified content can be displayed in a first content pane, and the modified extracted text content can be displayed in a second content pane presented within the browser content window. The second content pane can be positioned next to the first content pane in the browser content window so that the modified extracted text content can be viewed next to or with the original content.

[0079] If user preferences for formatting / modifying the extracted text content have not been previously stored (block 540), the extracted text content may be displayed in a reader view within the browser content window along with the original content (block 545). The original content may be displayed in a first content pane, and the extracted text content may be displayed in a second content pane positioned next to the first content pane in a reader view presented within the browser content window. In some examples, a reading assistant toolbar may be provided by the browser application, such as described above with respect to Figures 3A to 3G The reading assistant toolbar 355 described above and / or FIG. 4A to FIG. 4D The reading assistant toolbar 455 described above may allow the extracted text content to be formatted or reformatted to suit the needs of a particular user. In response to the modification input detected via the reading assistant toolbar (box 555), the formatting module 237 may generate modified extracted text content (box 560). The modified extracted text content may be displayed in the first content pane, and the extracted text content may be displayed in a second content pane positioned next to the first content pane in a reader view presented within the browser content window (box 565), so that the modified extracted text content may be viewed next to or with the original content.

[0080] In response to a scroll input that causes the (modified or unmodified) extracted text content displayed in the second pane to scroll (block 570), the system can anchor the (modified or unmodified) text content in the second content pane to the original, unmodified content in the first content pane (block 580). Similarly, if the scroll input causes the original, unmodified content in the first content pane to scroll, the system can anchor the unmodified content in the first content pane to the (modified or unmodified) text content in the second content pane. In some examples, the synchronization module, such as described above with respect to Figure 2 The synchronization module 239 of the described browser application 232 can anchor the text characters and / or words and / or text portions assigned to or associated with the original, unmodified content in the first pane to the bounding boxes associated with the corresponding text characters and / or words and / or text portions of the (modified or unmodified) extracted text content in the second pane of the reader view. For the original, unmodified content and the (modified or unmodified) extracted text content anchored in this manner, upon user manipulation of a scrolling device (e.g., a scroll control icon displayed in the browser user interface, a keyboard arrow key, a mouse wheel, a drag input on a touch pad or touch screen, etc.), the system performs synchronized scrolling of the (modified or unmodified) extracted text content with the original, unmodified content (block 590).

[0081] Therefore, a reading assistant tool is implemented in a browser application and facilitates presenting text content in a manner and format that satisfies the reading and / or comprehension abilities of individual users while maintaining the context of the original content. The reading assistant tool causes a reader view pane to be output next to a main content pane. The original content is presented in the main content pane, and simplified / reformatted content corresponding to text content extracted from the original content is presented in the reader view pane. The scrolling of the extracted text content in the reader view pane is synchronized with the scrolling of the original content in the main content pane, so that the context is maintained as the user browses the content. The concurrent presentation of the original content and the extracted text content allows the user to more easily consume the extracted text content while also maintaining the context as originally intended.

[0082] In one general aspect, a computer-implemented method includes: extracting, by a text extraction module of a browser application executed on a computing device, text content included in browser content displayed by the browser application; generating modified text content based on the extracted text content; and rendering, by the browser application, a reader view to display the modified text content with the browser content. The rendering may include: rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content. The computer-implemented method may also include: providing a reader view for display on a display device of the computing device, the reader view including the second content pane next to the first content pane.

[0083] In some implementations, generating the modified text content includes at least one of: modifying the format of the extracted text content according to previously stored user settings related to displaying the modified text content in the second content pane; or modifying the format of the extracted text content in response to user input detected at a reading assistant toolbar of the browser application. Modifying the format of the extracted text content may include modifying at least one of: font type, font size, font color, background color, spacing, or line spacing of the extracted text content.

[0084] In some implementations, the computer-implemented method includes: rendering browser content by a browser application; analyzing the browser content by a reading assistant module of the browser application; and determining a page type associated with the browser content based on the analysis, including determining that the browser content is associated with a refinable page in response to detecting extractable text content in the browser content. In some implementations, the computer-implemented method includes launching a reading assistant tool, including at least one of the following: launching the reading assistant tool in response to determining that the browser content is associated with a refinable page; or launching the reading assistant tool in response to user input selecting the reading assistant tool from a browser application menu. In some implementations, the computer-implemented method includes: determining that the browser content is associated with a non-refinable page; detecting user input selecting a portion of the text content from the browser content; and launching the reading assistant tool in response to user input selecting the reading assistant tool from a browser application menu. In some implementations, generating the modified text content includes: generating the modified text content based on the portion of the text content selected by the user input.

[0085] In some implementations, the computer-implemented method includes: detecting a scroll input for scrolling modified text content displayed in a second content pane; scrolling the modified text content displayed in the second content pane in response to the scroll input; and scrolling browser content displayed in the first content pane in response to the scroll input.

[0086] In some implementations, the computer-implemented method includes: detecting a scroll input for scrolling browser content displayed in a first content pane; scrolling the browser content displayed in the first content pane in response to the scroll input; and scrolling modified text content displayed in a second content pane in response to the scroll input. Scrolling the browser content displayed in the first content pane may include: synchronizing scrolling of the browser content displayed in the first content pane with scrolling of the modified text content displayed in the second content pane. Synchronizing scrolling of the browser content displayed in the first content pane with scrolling of the modified text content displayed in the second content pane may include: associating a first bounding box assigned to a text element in the browser content displayed in the first content pane with a second bounding box assigned to a corresponding text element in the modified text content displayed in the second content pane; and performing synchronization of scrolling based on the association of the first bounding box with the second bounding box.

[0087] In another general aspect, a non-transitory computer-readable medium stores instructions that, when executed by at least one processor of a computing device, cause the at least one processor to: extract, by a text extraction module of a browser application executed on the computing device, text content included in browser content displayed by the browser application; generate modified text content based on the extracted text content; and render, by the browser application, a reader view to display the browser content together with the modified text content, including: rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content. The instructions may also cause the at least one processor to: provide a reader view including the first content pane and the second content pane for display on a display device of the computing device.

[0088] In some implementations, the instructions cause the at least one processor to: detect previously stored user settings related to displaying modified text content in the second content pane; modify the format of the extracted text content according to the previously stored user settings; detect user input at a reading assistant toolbar of the browser application; and modify the format of the extracted text content in response to the user input detected at the reading assistant toolbar. In some implementations, the instructions cause the at least one processor to: render browser content by the browser application; analyze the browser content by a reading assistant module of the browser application; and determine a page type associated with the browser content, including: determining that the browser content is associated with a refinable page in response to detecting extractable text content in the browser content.

[0089] In some implementations, the instructions cause the at least one processor to launch a reading assistant tool, including at least one of: launching the reading assistant tool in response to determining that the browser content is associated with a refinable page; or launching the reading assistant tool in response to user input selecting the reading assistant tool from a browser application menu. In some implementations, the instructions cause the at least one processor to: determine that the browser content is associated with a non-refinable page; detect user input selecting a portion of the text content from the browser content; and launch the reading assistant tool in response to user input selecting the reading assistant tool from the browser application menu. In some implementations, the instructions cause the at least one processor to: generate modified text content based on the portion of the text content selected by the user input.

[0090] In some implementations, the instructions cause the at least one processor to: detect a scroll input for scrolling modified text content displayed in the second content pane; scroll the modified text content displayed in the second content pane in response to the scroll input; and scroll the browser content displayed in the first content pane in response to the scroll input. In some implementations, the instructions cause the at least one processor to: synchronize scrolling of the browser content displayed in the first content pane with scrolling of the modified text content displayed in the second content pane, including: associating a first bounding box assigned to a text element in the browser content displayed in the first content pane with a second bounding box assigned to a corresponding text element in the modified text content displayed in the second content pane; and synchronizing the scrolling based on the association of the first bounding box with the second bounding box.

[0091] In some implementations, the browser content includes text content, and at least one of image content, table content, links, input fields, or browser controls, and wherein the instructions cause at least one processor to modify at least one of font type, font size, font color, background color, spacing, or line spacing of the extracted text content.

[0092] In another general aspect, a computing system includes: a processor; a display; and a memory storing instructions that, when executed by the processor, cause a browser application executing on the computing system to perform operations. The operations may include: extracting, by a text extraction module of the browser application, text content included in browser content displayed on the display; generating modified text content based on the extracted text content; rendering a reader view to display the browser content together with the modified text content, including: rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content; and providing a reader view including the first content pane and the second content pane for display on the display.

[0093] In addition to the above description, controls may be provided to the user that allow the user to make choices about whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (such as to a city, zip code, or state level) when location information is obtained so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0094] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0095] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0096] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, verbal, or tactile input.

[0097] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0098] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server is created by computer programs running on the respective computers and placing each other in a client-server relationship.

[0099] In this specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Further, conjunctions such as "and", "or", and "and / or" are inclusive unless the context clearly dictates otherwise. For example, "A and / or B" includes only A, only B, and both A and B. Further, the connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and / or physical or logical couplings between the various elements. In an actual device, there may be many alternative or additional functional relationships, physical connections, or logical connections. In addition, unless an element is specifically described as "essential" or "critical", no item or component is essential to the practice of the implementations disclosed herein.

[0100] Terms such as, but not limited to, about, substantially, approximately, etc. are used herein to indicate that exact values ​​or ranges thereof are not required and need not be specified. As used herein, the terms discussed above have their ready and immediate meanings to one of ordinary skill in the art.

[0101] Furthermore, the use of terms such as upper, lower, top, bottom, side, end, front, rear, etc. herein are all used with reference to the orientation currently considered or shown. If these terms are considered with respect to another orientation, it should be understood that such terms must be modified accordingly.

[0102] Although certain example computer-implemented methods, devices, and articles have been described herein, the scope of coverage of this patent is not limited thereto. It should be understood that the terminology employed herein is for the purpose of describing particular aspects and is not intended to be limiting. On the contrary, this patent covers all methods, devices, and articles that fairly fall within the scope of the claims of this patent.

[0103] In one aspect, a non-transitory computer-readable medium stores instructions that, when executed by a processor on a receiving computing device, cause the receiving computing device to perform any of the methods disclosed herein.

[0104] In one aspect, a computing device may be configured with at least one processor and a memory storing instructions that, when executed by the at least one processor, perform any of the methods disclosed herein.

Claims

1. A computer-implemented method comprising: extracting, by a text extraction module of a browser application executed on a computing device, text content included in browser content displayed by the browser application; generating modified text content based on the extracted text content; Rendering a reader view by the browser application to display the modified text content together with the browser content includes: Rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content; and The reader view is provided for display on a display device of the computing device, the reader view including the second content pane alongside the first content pane.

2. The computer-implemented method of claim 1 , wherein generating the modified text content comprises at least one of: modifying a format of the extracted text content according to previously stored user settings associated with displaying the modified text content in the second content pane; or In response to user input detected at a reading assistant toolbar of the browser application, the format of the extracted text content is modified.

3. A computer-implemented method as described in claim 2, wherein modifying the format of the extracted text content includes modifying at least one of the following: font type, font size, font color, background color, spacing, or line spacing of the extracted text content.

4. The computer-implemented method of any one of claims 1 to 3, further comprising: Rendering the browser content by the browser application; Analyzing the browser content by a reading assistant module of the browser application; as well as In response to detecting extractable text content in the browser content, it is determined that the browser content is associated with a refinable page.

5. The computer-implemented method of claim 4, further comprising initiating a reading assistant tool, including at least one of: In response to determining that the browser content is associated with the refinable page, launching the reading assistant tool; or In response to user input selecting the reading assistant tool from a browser application menu, the reading assistant tool is launched.

6. The computer-implemented method of claim 4, further comprising: determining that the browser content is associated with a non-refinable page; detecting user input selecting a portion of text content from the browser content; as well as In response to user input selecting a reading assistant tool from a browser application menu, the reading assistant tool is launched.

7. The computer-implemented method of claim 6, wherein generating the modified text content comprises: The modified text content is generated based on the portion of the text content selected by the user input.

8. The computer-implemented method of any preceding claim, further comprising: detecting a scroll input for scrolling the modified text content displayed in the second content pane; In response to the scroll input, scrolling the modified text content displayed in the second content pane; as well as In response to the scroll input, the browser content displayed in the first content pane is scrolled.

9. The computer-implemented method of any preceding claim, further comprising: detecting a scroll input for scrolling the browser content displayed in the first content pane; In response to the scroll input, scrolling the browser content displayed in the first content pane; as well as In response to the scroll input, the modified text content displayed in the second content pane is scrolled.

10. The computer-implemented method of claim 8, scrolling the browser content displayed in the first content pane comprising: The scrolling of the browser content displayed in the first content pane is synchronized with the scrolling of the modified text content displayed in the second content pane.

11. The computer-implemented method of claim 10, wherein synchronizing the scrolling of the browser content displayed in the first content pane with the scrolling of the modified text content displayed in the second content pane comprises: associating a first bounding box assigned to a text element in the browser content displayed in the first content pane with a second bounding box assigned to a corresponding text element in the modified text content displayed in the second content pane; as well as The synchronization of the scrolling is performed based on the association of the first bounding box with the second bounding box.

12. The computer-implemented method of any preceding claim, wherein generating the modified text content comprises: Extracting the text content from the browser content; as well as At least one of image content, table content, links, input fields, or browser controls included in the browser content is omitted from the modified text content.

13. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a computing device, cause the at least one processor to: extracting, by a text extraction module of a browser application executed on the computing device, text content included in browser content displayed by the browser application; generating modified text content based on the extracted text content; Rendering a reader view by the browser application to display the browser content together with the modified text content includes: Rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content; and The reader view including the first content pane and the second content pane is provided for display on a display device of the computing device.

14. The non-transitory computer readable medium of claim 13, wherein the instructions cause the at least one processor to: detecting a previously stored user setting related to displaying the modified text content in the second content pane; modifying the format of the extracted text content according to the previously stored user settings; Detecting user input at a reading assistant toolbar of the browser application; as well as In response to the user input detected at the reading assistant toolbar, the format of the extracted text content is modified.

15. The non-transitory computer readable medium of claim 13 or claim 14, wherein the instructions cause the at least one processor to: Rendering the browser content by the browser application; Analyzing the browser content by a reading assistant module of the browser application; and In response to detecting extractable text content in the browser content, it is determined that the browser content is associated with a refinable page.

16. The non-transitory computer-readable medium of claim 15, wherein the instructions cause the at least one processor to launch a reading assistant tool, including at least one of: In response to determining that the browser content is associated with the refinable page, launching the reading assistant tool; or In response to user input selecting the reading assistant tool from a browser application menu, the reading assistant tool is launched.

17. The non-transitory computer readable medium of claim 15, wherein the instructions cause the at least one processor to: determining that the browser content is associated with a non-refinable page; detecting user input selecting a portion of text content from the browser content; and In response to user input selecting a reading assistant tool from a browser application menu, the reading assistant tool is launched.

18. The non-transitory computer-readable medium of claim 17, wherein the instructions cause the at least one processor to generate the modified text content based on the portion of the text content selected by the user input.

19. The non-transitory computer readable medium of any one of claims 13 to 18, wherein the instructions cause the at least one processor to: detecting a scroll input for scrolling the modified text content displayed in the second content pane; In response to the scroll input, scrolling the modified text content displayed in the second content pane; and In response to the scroll input, the browser content displayed in the first content pane is scrolled.

20. The non-transitory computer readable medium of claim 19, wherein the instructions cause the at least one processor to: Synchronizing the scrolling of the browser content displayed in the first content pane with the scrolling of the modified text content displayed in the second content pane, comprising: associating a first bounding box assigned to a text element in the browser content displayed in the first content pane with a second bounding box assigned to a corresponding text element in the modified text content displayed in the second content pane; as well as The scrolling is synchronized based on the association of the first bounding box with the second bounding box.

21. A non-transitory computer-readable medium as described in any one of claims 13 to claim 20, wherein the browser content includes the text content, and at least one of image content, table content, links, input fields, or browser controls, and wherein the instructions cause the at least one processor to modify at least one of the font type, font size, font color, background color, spacing, or line spacing of the extracted text content.

22. The non-transitory computer-readable medium of any one of claims 13 to 21, wherein the instructions cause the at least one processor to generate the modified text content, comprising: Extracting the text content from the browser content; as well as At least one of image content, table content, links, input fields, or browser controls included in the browser content is omitted from the modified text content.

23. A computing system comprising: processor; monitor; and a memory storing instructions that, when executed by the processor, cause a browser application executing on the computing system to perform operations comprising: extracting, by a text extraction module of the browser application, text content included in the browser content displayed on the display; generating modified text content based on the extracted text content; Rendering a reader view to display the browser content together with the modified text content, comprising: Rendering a first content pane displaying the browser content; and rendering a second content pane next to the first content pane, the second content pane displaying the modified text content; and The reader view including the first content pane and the second content pane is provided for display on the display.

24. The computing system of claim 23, wherein the instructions executed by the processor cause the browser application to generate the modified text content, comprising: Extracting the text content from the browser content; as well as At least one of image content, table content, links, input fields, or browser controls included in the browser content is omitted from the modified text content.

Images