A low vision speech interactive reading system, method of use and apparatus

Abstract

This invention relates to the field of human-computer interaction technology, and in particular to a low-vision speech interaction reading system, method of use, and device. It includes: capturing the user's finger touch events on the screen in real time, obtaining the precise coordinates of the touch point, and locating the UI element corresponding to the precise coordinates using a traversal algorithm based on the currently active view hierarchy; reading the corresponding semantic attributes of the UI element, calling an OCR engine to perform image recognition on the semantic attributes, and generating standard reading text; inputting the standard reading text into a TTS engine with a loading and caching mechanism, and outputting the reading voice through a speaker. This invention solves the pain point of low-vision users being unable to see the device screen content clearly and unable to hear the TTS reading in a timely manner by using touch event-driven, TTS voice feedback, OCR enhanced recognition, and a floating icon keep-alive mechanism.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a low-vision speech interactive reading system, its usage method, and its device. Background Technology

[0002] Among existing accessibility technologies, solutions for visually impaired users mainly suffer from the following bottlenecks: Existing technologies primarily utilize system accessibility services to access the system's touch browsing interface, enabling quick gesture operations. During these operations, TTS (Text-to-Speech) provides real-time feedback on the process and results. This solution is mainly suitable for the operational habits of the totally blind. For those with some vision, this solution exhibits extremely low operational efficiency and a significant learning curve. Because the characteristics of low-vision individuals are that they can see the screen image and even the screen content, but cannot discern the details displayed, TTS is needed as an auxiliary reading tool, eliminating the need for quick gesture operations in touch browsing mode.

[0003] Users need TTS (Text-to-Speech) to read aloud wherever they tap, without any additional gestures. However, this is impossible with existing touch browsing technology. Touch browsing is designed for the visually impaired, and this technology introduces a delay in response to user gestures. Specifically, there's a 300-millisecond delay after each screen touch before TTS reads the content. Furthermore, two taps are required to execute the tap. This design prevents visually impaired users from clearly hearing the touches, and the double taps prevent accidental misoperations due to rapid response. This technology is provided by the system's underlying layer and cannot be customized. Temporarily pausing touch browsing allows for short-term instant reading, but due to power-saving strategies and background operation limitations, the application freezes in the background after a pause, preventing TTS from reading and ultimately rendering the software unusable. Besides the limitations of TTS technology, the timeliness of device operation is also a concern. Under touch browsing technology, each click requires two taps on the screen, with a noticeable delay between them. This significantly reduces the need for efficient, fast, and timely device operation and responsiveness among visually impaired individuals. Furthermore, touch browsing technology cannot achieve real-time operation of the point touched by the finger. This is because the system automatically sets accessibility focus during clicks. Clicks can only be performed on locations within the reach of this accessibility focus, greatly reducing software operability, especially in games. Touch browsing technology also presents significant inconvenience for visually impaired users in scenarios such as screen content recognition, instant message reading, and game content recognition. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides a low-vision voice interactive reading system, method of use, and device, thereby addressing the pain point of low-vision individuals being unable to clearly see the content displayed on the device screen and unable to hear TTS readings in a timely manner.

[0005] This invention provides a low-vision speech interactive reading system, comprising the following modules: Touch detection and coordinate localization module: captures the user's finger touch events on the screen in real time, obtains the precise coordinates of the touch point, and combines the currently active view hierarchy structure to locate the UI element corresponding to the precise coordinates through a traversal algorithm; Content extraction and processing module: Based on the UI element, read the corresponding semantic attributes, call the OCR engine to perform image recognition on the semantic attributes, and generate standard reading text; Speech synthesis and output module: Inputs the standard text into the TTS engine with a loading and caching mechanism, and outputs the text through a speaker.

[0006] According to the low-vision speech interactive reading system provided by the present invention, the working process of the touch detection and coordinate positioning module is as follows: S11: Utilizes the Android system's event mechanism to capture the user's finger touch events on the screen in real time and obtain the coordinates of the touch point; S12: Based on the coordinates of the touch point and combined with the currently active view hierarchy, the Accessibility Node Info traversal algorithm is used to locate the UI element corresponding to the coordinates of the touch point. The UI element includes buttons, text boxes, image labels, and text.

[0007] According to the low-vision interactive voice reading system provided by the present invention, the workflow of the content extraction and processing module is as follows: S21: Based on the UI element, read the corresponding semantic attributes, determine whether there is valid text information, and if there is valid text information, extract the visible text and execute step S23; if there is no valid text information, execute step S22. S22: Use the built-in OCR engine to perform image recognition on a local screen area and extract visible text; S23: Identify the visible text and perform text cleaning through the natural language processing module to generate standard reading text.

[0008] According to the low-vision speech interactive reading system provided by the present invention, step S22 is as follows: S221: Using the touch point as the center, extract a local screen image of a preset size as the region of interest; S222: Perform grayscale conversion, binarization, and noise reduction preprocessing on the local screen image in sequence; input the preprocessed image into the optical character recognition engine to extract the initial text; S223: Input the initial text into the semantic processing module based on natural language processing for semantic disambiguation and expansion to generate visible text.

[0009] According to the low-vision speech interactive reading system provided by the present invention, the method for determining the region of interest is to calculate the probability of click intent. : in, For the Sigmoid function, finger dwell time The average sliding speed of the finger. The maximum acceleration of the finger. The angle representing the change in curvature of the finger trajectory. The number of times the fingers tremble slightly. As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient. It is the fourth weighting coefficient. As the fifth weighting coefficient, when Regions exceeding the interest threshold are considered regions of interest.

[0010] According to the present invention, a low-vision speech interactive reading system is provided, wherein the speech synthesis and output module pre-constructs a high-frequency vocabulary library. When standard reading text is input to a TTS engine with a loading and caching mechanism, the system pre-synthesizes speech segments corresponding to the high-frequency vocabulary library and caches them locally. When performing speech playback, the system first traverses the local cache. If a match is found, the corresponding speech segment is directly extracted and played. If no match is found, the system requests the speech synthesis engine to perform real-time synthesis.

[0011] The low-vision voice interaction reading system provided by the present invention also includes a floating icon module. This module displays a small, customizable floating icon at the top of the screen, implemented using lightweight floating window technology. It has a higher priority than ordinary application interfaces, ensuring it is displayed above all applications. It supports transparency adjustment, size scaling, and hide / expand switching to avoid obscuring key content. It has a built-in long-press trigger mechanism: after the user long-presses the icon for a certain period, a shortcut function menu pops up. It also has background persistence capabilities, preventing system reclaiming and ensuring continuous service operation through a dual mechanism of binding to system accessibility services and foreground services.

[0012] The low-vision speech interactive reading system provided by the present invention further includes a function integration and interaction logic module, wherein the function integration and interaction logic is used for the following functions: Text tap-to-read: Users can tap anywhere to immediately read the selected content aloud without switching modes; Automatic screen content reading: When enabled, the system automatically scans and reads the full-screen content from top to bottom and from left to right, which is suitable for reading long texts; Copy function: Double-click the floating icon or select "Copy" in the shortcut menu, and the system will save the read-aloud content to the clipboard; Word explanation: Long press the target word or select "Explanation" in the menu to call up the built-in dictionary or online API to return the definition and read it aloud; OCR Enhanced Recognition: For non-text controls, OCR is automatically enabled for supplementary recognition.

[0013] The present invention also provides a method of using a low-vision speech reading system, comprising the following steps: S1: Long press the part to be identified. The touch detection and coordinate positioning module captures the user's finger touch event on the screen in real time, obtains the precise coordinates of the touch point, and combines the currently active view hierarchy structure to locate the UI element corresponding to the precise coordinates through a traversal algorithm. S2: The content extraction and processing module reads the corresponding semantic attributes based on the UI elements, calls the OCR engine to perform image recognition on the semantic attributes, and generates standard reading text; S3: The speech synthesis and output module inputs the standard text into the TTS engine with a loading and caching mechanism, and outputs the text through the speaker.

[0014] The present invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor is used to execute a computer program to implement the steps of the method of using a low-vision speech interactive reading system as described above.

[0015] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: This invention provides a low-vision voice interaction reading system, method of use, and device. By triggering floating icons, listening to touch events, locating accessible nodes, providing low-latency TTS feedback, integrating OCR with NLP semantic enhancement recognition and multimodal functions, a complete, efficient, and low-latency auxiliary interaction system is constructed to solve the technical problem of low-vision users being able to "see but not clearly see" screen content.

[0016] This invention abandons the traditional focus navigation mechanism. By directly acquiring touch coordinates and mapping them to accessibility nodes in real time, it fundamentally solves the latency problem of asynchronous operation and feedback, providing a millisecond-level positioning foundation for subsequent voice feedback.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1This is a structural block diagram of a low-vision voice interactive reading system provided by the present invention.

[0020] Figure 2 This is a flowchart illustrating the usage method of a low-vision voice interactive reading system provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0022] Figure label: 101. Touch detection and coordinate positioning module; 102. Content extraction and processing module; 103. Speech synthesis and output module; 104. Floating icon module; 105. Function integration and interaction logic module; 810. Processor; 820. Communication interface; 830. Memory; 840. Communication bus. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0024] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0025] The following is combined Figures 1 to 3 This invention is described.

[0026] like Figure 1 As shown, Figure 1 A structural block diagram of a low-vision speech interactive reading system provided by the present invention includes the following modules: Touch detection and coordinate positioning module 101: captures the user's finger touch event on the screen in real time, obtains the precise coordinates of the touch point, and combines the currently active view hierarchy structure to locate the UI element corresponding to the precise coordinates through a traversal algorithm; Content extraction and processing module 102: Based on the UI element, read the corresponding semantic attributes, call the OCR engine to perform image recognition on the semantic attributes, and generate standard reading text; Speech synthesis and output module 103: Inputs the standard text to a TTS engine with a loading and caching mechanism, and outputs the text through a speaker.

[0027] Specifically, the workflow of the touch detection and coordinate positioning module 101 is as follows: S11: Utilizes the Android system's event mechanism to capture the user's finger touch events on the screen in real time and obtain the coordinates of the touch point; S12: Based on the coordinates of the touch point and combined with the currently active view hierarchy, the Accessibility Node Info traversal algorithm is used to locate the UI element corresponding to the coordinates of the touch point. The UI element includes buttons, text boxes, image labels, and text.

[0028] Specifically, the touch detection and coordinate positioning module 101 enables a low-vision assistive interaction service. This service inherits from the Android system's AccessibilityService and completes accessibility event registration, listening for three core system events: window state changes, control clicks, and control focus acquisition. During actual development, we found significant differences in the lifecycle management of the accessibility service across different Android versions. Android 10 and later versions impose stricter restrictions on background services, and simply relying on the onServiceConnected callback cannot guarantee continuous service operation. Therefore, we introduced a foreground service binding mechanism as a supplement.

[0029] The touch detection and coordinate positioning module 101 requests system floating window permission (SYSTEM_ALERT_WINDOW) to create top-level floating windows for applications, such as of type TYPE_APPLICATION_OVERLAY, ensuring that the floating icon is displayed on top of all application interfaces. Here, it's important to note a practical issue: some domestic ROMs disable floating window permission by default. Even if the user grants permission during installation, the permission status may be reset after a system update. Therefore, the module includes logic for permission status detection and guiding the user to re-authorize.

[0030] The touch detection and coordinate positioning module 101 elevates the auxiliary service to a foreground service by starting it as a foreground service, and simultaneously binds it to a notification channel to circumvent Android's restrictions on background service operation, ensuring continuous and stable service operation without being reclaimed by the system. During testing, we found that using the foreground service alone on some devices would still result in it being killed by manufacturer-customized power-saving strategies. Therefore, we added JobScheduler polling as a second layer of protection. With both methods combined, the keep-alive success rate increased from approximately 91% to over 98%.

[0031] The touch detection and coordinate positioning module 101 uses WindowManager.LayoutParams to create a floating control, sets its type to LayoutType.TYPE_APPLICATION_OVERLAY, and its flag to FLAG_NOT_FOCUSABLE or FLAG_LAYOUT_IN_SCREEN, to ensure that it does not intercept underlying touch events; The floating icon of the touch detection and coordinate positioning module 101 uses vector Drawable resources and supports dynamic coloring and scaling. The touch detection and coordinate positioning module 101 registers an OnTouchListener to listen for ACTION_DOWN, ACTION_MOVE, and ACTION_UP events. When the touch detection and coordinate positioning module 101 detects a long press event lasting ≥1.5s, it triggers GestureDetector recognition and pops up a shortcut function menu in the form of a PopupWindow.

[0032] The reason for choosing 1.5s as the threshold is that it was determined after multiple rounds of user testing. When it is less than 1.2s, the false trigger rate is too high, and when it exceeds 2.0s, the user perceives a significant delay. 1.5s is a compromise value that takes into account both false triggers and response.

[0033] The embodiments of this invention employ lightweight floating window technology, which has a higher priority than ordinary application interfaces, ensuring that it is displayed above all applications; it supports transparency adjustment, size scaling, and hide / expand switching to avoid obscuring key content.

[0034] This invention also includes a built-in long-press trigger mechanism: when a user long-presses an icon for more than 1.5 seconds, a quick function menu pops up, including: turn reading aloud on / off, OCR recognition, automatic reading aloud, copy, word explanation, etc.; it has background persistence capability, through a dual mechanism of binding to the system's accessibility service and foreground service, preventing it from being reclaimed by the system and ensuring continuous service operation. In actual use, the timing of the menu pop-up and the interaction method have been repeatedly adjusted. The initial version used a long list menu, but it was difficult for visually impaired users to locate with their fingers. Later, it was changed to a circular menu layout, with options arranged radially around the touch point, reducing the distance the finger needs to move.

[0035] Specifically, this invention addresses the problem that fixed trigger thresholds cannot accommodate the varying reaction speeds of users with different vision levels, nor can they meet the real-time requirements of different application scenarios. This invention employs an exponentially weighted moving average to predict the user's current response capability: in, For current response capabilities, The attenuation factor (with a value range of 0.7 to 0.9, and experimental results show that 0.85 provides the most stable prediction performance). This is the average response time of the last 5 operations. This refers to the response capability at the previous moment.

[0036] This invention also performs dynamic threshold adjustment to trigger the threshold. Based on the user's vision level V l and application scenario level Make dynamic adjustments In particular, to prevent the trigger threshold from being too high or too low, this invention limits... At the minimum trigger threshold and the maximum trigger threshold between: in, .

[0037] Specifically, the workflow of the content extraction and processing module 102 is as follows: S21: Based on the UI element, read the corresponding semantic attributes and determine whether it contains valid text information. If valid text information exists, extract the visible text and execute step S23; if no valid text information exists, execute step S22. S22: Use the built-in OCR engine to perform image recognition on a local screen area and extract visible text; S23: Identify the visible text and perform text cleaning through the natural language processing module to generate standard reading text.

[0038] Specifically, step S22 is as follows: S221: Using the touch point as the center, extract a local screen image of a preset size as the region of interest; Specifically, embodiments of the present invention calculate the probability of a click intent based on finger trajectory features. : in, For the Sigmoid function, finger dwell time The average sliding speed, For maximum acceleration, The angle representing the change in trajectory curvature. The number of micro-vibrations. As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient. It is the fourth weighting coefficient. It is the fifth weighting coefficient.

[0039] when When the value is greater than the interest determination threshold, which is set to 0.95 in this embodiment of the invention, it is considered as a region of interest.

[0040] S222: Perform grayscale conversion, binarization, and noise reduction preprocessing on the local screen image in sequence; input the preprocessed image into the optical character recognition engine to extract the initial text; S223: Input the initial text into the natural language processing module based on natural language processing for semantic disambiguation and expansion to generate visible text.

[0041] This step mainly addresses common typos and sentence breaks after OCR recognition. For example, OCR sometimes recognizes "login" as "login" or "settings" as "settings". The NLP module will reassemble these fragments based on the context. Targeted word segmentation processing has also been performed for application interfaces with mixed Chinese and English text.

[0042] Specifically, the speech synthesis and output module 103 pre-builds a high-frequency vocabulary library. When standard reading text is input to a TTS engine with a loading and caching mechanism, the module pre-synthesizes speech segments corresponding to the high-frequency vocabulary library and caches them locally. When performing speech playback, the module prioritizes traversing the local cache. If a match is found, the corresponding speech segment is directly extracted and played. If no match is found, the module requests the speech synthesis engine to perform real-time synthesis.

[0043] The high-frequency vocabulary database employs a pre-loading caching mechanism, establishing a speech cache pool for frequently used words and interface elements to reduce synthesis latency during repeated readings, keeping the average response time below 300ms. In actual testing, for high-frequency interface words such as "button," "back," and "close," the latency after a cache hit can be reduced to around 80ms. However, unfamiliar words encountered for the first time still require going through the complete synthesis chain, taking approximately 250 to 350ms, resulting in an overall average response time below 300ms.

[0044] Specifically, the floating icon module 104 displays a small floating icon at a fixed position at the top of the screen, implemented using lightweight floating window technology. This icon has higher priority than ordinary application interfaces, ensuring it appears above all applications. It supports transparency adjustment, size scaling, and hide / expand switching to avoid obscuring key content. It has a built-in long-press trigger mechanism: after the user long-presses the icon for a certain period, a shortcut menu pops up. It also has background persistence capabilities, using a dual mechanism of binding to system accessibility services and foreground services to prevent it from being reclaimed by the system and ensure continuous service operation. The floating icon module 104 reads and processes the touch events captured by the touch detection and coordinate positioning module 101, assisting the touch detection and coordinate positioning module 101 in its operation.

[0045] Specifically, the present invention also includes a function integration and interaction logic module 105, which is used for the following purposes: Text tap-to-read: Users can tap anywhere to immediately read the selected content aloud without switching modes; Automatic screen content reading: When enabled, the system automatically scans and reads the full-screen content from top to bottom and from left to right, which is suitable for reading long texts; Copy function: Users can double-click the floating icon or select "Copy" in the shortcut menu to save the read-aloud content to the clipboard; Word explanation: Users can long-press on the target word or select "Explanation" in the menu to call up the built-in dictionary or online API to return the definition and read it aloud; OCR Enhanced Recognition: For non-text controls, OCR is automatically enabled for supplementary recognition.

[0046] The functional integration and interaction logic module 105 reads and processes the touch events captured by the touch detection and coordinate positioning module 101 when the user's finger touches the screen, and assists the touch detection and coordinate positioning module 101 in its operation.

[0047] Table 1 Comparison of Experimental Data

[0048] As shown in Table 1, the method of the present invention can effectively reduce the operation time and accidental touch rate, where devices A-J are all common mobile phone models.

[0049] like Figure 2 As shown, Figure 2 A flowchart illustrating the usage method of a low-vision speech interactive reading system provided by the present invention includes the following steps: S1: Long press the part to be identified. The touch detection and coordinate positioning module captures the user's finger touch event on the screen in real time, obtains the precise coordinates of the touch point, and combines the currently active view hierarchy structure to locate the UI element corresponding to the precise coordinates through a traversal algorithm. S2: The content extraction and processing module reads the corresponding semantic attributes based on the UI elements, calls the OCR engine to perform image recognition on the semantic attributes, and generates standard reading text; S3: The speech synthesis and output module inputs the standard text into the TTS engine with a loading and caching mechanism, and outputs the text through the speaker.

[0050] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a method of using a low-vision speech interactive reading system, including the following steps: S1: Long press the part to be identified. The touch detection and coordinate positioning module captures the user's finger touch event on the screen in real time, obtains the precise coordinates of the touch point, and combines the currently active view hierarchy structure to locate the UI element corresponding to the precise coordinates through a traversal algorithm. S2: The content extraction and processing module reads the corresponding semantic attributes based on the UI elements, calls the OCR engine to perform image recognition on the semantic attributes, and generates standard reading text; S3: The speech synthesis and output module inputs the standard text into the TTS engine with a loading and caching mechanism, and outputs the text through the speaker.

[0051] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0053] It should be noted that the embodiments of this disclosure can be implemented using hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a programmable memory or a data carrier such as an optical or electronic signal carrier.

[0054] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0055] While this disclosure has been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the specific embodiments disclosed. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A low-vision speech-interactive reading system, characterized in that, Includes the following modules: Touch detection and coordinate localization module: captures the user's finger touch events on the screen in real time, obtains the precise coordinates of the touch point, and combines the currently active view hierarchy structure to locate the UI element corresponding to the precise coordinates through a traversal algorithm; Content extraction and processing module: Based on the UI elements, read the corresponding semantic attributes, call the OCR engine to perform image recognition on the semantic attributes, and generate standard text for reading aloud; the workflow of the content extraction and processing module is as follows: S21: Based on the UI element, read the corresponding semantic attributes, determine whether there is valid text information, and if there is valid text information, extract the visible text and execute step S23; if there is no valid text information, execute step S22. S22: Call the built-in OCR engine to perform image recognition on a local screen area and extract visible text; the process of step S22 is as follows: S221: Using the touch point as the center, extract a local screen image of a preset size as the region of interest; the method for determining the region of interest is to calculate the probability of click intent. : in, For the Sigmoid function, The duration of finger stay. The average sliding speed of the finger. The maximum acceleration of the finger. The angle representing the change in curvature of the finger trajectory. The number of times the fingers tremble slightly. As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient. It is the fourth weighting coefficient. As the fifth weighting coefficient, when Regions exceeding the interest threshold are considered regions of interest. S222: Perform grayscale conversion, binarization, and noise reduction preprocessing on the local screen image in sequence; input the preprocessed image into the optical character recognition engine to extract the initial text; S223: Input the initial text into the semantic processing module based on natural language processing for semantic disambiguation and expansion to generate visible text; S23: Identify the visible text and perform text cleaning through the natural language processing module to generate standard reading text; Speech synthesis and output module: Inputs the standard text into the TTS engine with a loading and caching mechanism, and outputs the text through a speaker.

2. The low-vision speech interactive reading system according to claim 1, characterized in that, The workflow of the touch detection and coordinate positioning module is as follows: S11: Utilizes the Android system's event mechanism to capture the user's finger touch events on the screen in real time and obtain the coordinates of the touch point; S12: Based on the coordinates of the touch point and combined with the currently active view hierarchy, the AccessibilityNode Info traversal algorithm is used to locate the UI element corresponding to the coordinates of the touch point. The UI element includes buttons, text boxes, image labels, and text.

3. The low-vision speech interactive reading system according to claim 1, characterized in that, The speech synthesis and output module pre-builds a high-frequency vocabulary library. When standard reading text is input to a TTS engine with a loading and caching mechanism, the module pre-synthesizes speech segments corresponding to the high-frequency vocabulary library and caches them locally. When performing speech playback, the module first traverses the local cache. If a match is found, the corresponding speech segment is directly extracted and played. If no match is found, the module requests the speech synthesis engine to perform real-time synthesis.

4. The low-vision speech interactive reading system according to claim 1, characterized in that, It also includes a floating icon module, which displays a small floating icon at a custom position at the top of the screen. This is achieved using lightweight floating window technology, which has higher priority than ordinary application interfaces, ensuring that it is displayed above all applications. It supports transparency adjustment, size scaling, and toggling between hiding and expanding to avoid obscuring key content. It has a built-in long-press trigger mechanism: after the user long-presses the icon for more than a set time threshold, a quick function menu pops up. It also has background keep-alive capabilities, which prevent it from being recycled by the system through a dual mechanism of binding to the system's accessibility service and foreground service, ensuring the continuous operation of the service.

5. A low-vision speech interactive reading system according to claim 1, characterized in that, It also includes a function integration and interaction logic module, which has the following functions: Text tap-to-read: Users can tap anywhere to immediately read the selected content aloud without switching modes; Automatic screen content reading: When enabled, the system automatically scans and reads the full-screen content from top to bottom and from left to right, which is suitable for reading long texts; Copy function: Double-click the floating icon or select "Copy" in the menu, and the system will save the text to the clipboard; Word explanation: Long press the target word or select "Explanation" in the menu to call up the built-in dictionary or online API to return the definition and read it aloud; OCR Enhanced Recognition: For non-text controls, OCR is automatically enabled for supplementary recognition.

6. A method of using a low-vision interactive speech reading system, for executing the low-vision interactive speech reading system according to any one of claims 1 to 5, comprising the following steps: S1: Long press the part to be identified. The touch detection and coordinate positioning module captures the user's finger touch event on the screen in real time, obtains the precise coordinates of the touch point, and combines the currently active view hierarchy structure to locate the UI element corresponding to the precise coordinates through a traversal algorithm. S2: The content extraction and processing module reads the corresponding semantic attributes based on the UI elements, calls the OCR engine to perform image recognition on the semantic attributes, and generates standard reading text; S3: The speech synthesis and output module inputs the standard text into the TTS engine with a loading and caching mechanism, and outputs the text through the speaker.

7. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, characterized in that, When the processor executes a computer program, it implements the steps of the method of using a low-vision speech interactive reading system as described in claim 6.

Images