Automatic reading method and device based on terminal device

Abstract

The embodiment of the present application provides an automatic reading method and device based on a terminal device, which is applied to the field of terminal technology. The method includes: obtaining the node data of the node of the target window in the current interface of the terminal device in response to a trigger indication; filtering the obtained node according to the node data of the obtained node to obtain the node to be processed; wherein the node data of the node to be processed includes text content; reading aloud the text content of the node to be processed. In this way, the terminal device can filter the node data of the node to obtain a node including text content, the terminal device can read aloud the text content of the filtered node, and the user can listen to the text content read aloud by the terminal device, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an automatic reading method and device based on a terminal device. Background Art

[0002] With the development of terminal technology, people are widely using terminal devices in their daily lives and work. Among them, some users use terminal devices equipped with touch screens. When using a terminal device equipped with a touch screen, the user can use the corresponding functions of the terminal device based on the text content on the touch screen. For example, the user can view the text content of a chat message sent by others on the terminal device screen. For another example, the user can read the text content of a novel on the terminal device screen.

[0003] However, in some situations, users may need to access text content on the screen but may find it difficult to view it on their terminal device. For example, users may find it difficult to view the screen while washing, feeling tired, or navigating. Another example is that some users with dyslexia may not be able to properly use the services provided by the terminal device, affecting their user experience. Summary of the Invention

[0004] The embodiments of the present application provide a terminal device-based automatic reading method and device, which are applied to the field of terminal technology and can automatically read text content on the screen through the terminal device, simplify user operations, and improve user experience.

[0005] In the first aspect, the embodiment of the present application proposes an automatic reading method and device based on a terminal device. The method includes: in response to a trigger indication, obtaining the node data of the node of the target window in the current interface of the terminal device; wherein the trigger indication is used to indicate the reading of the information on the target window in the current interface displayed by the terminal device; the node represents a piece of data in the target window; according to the node data of the obtained node, the obtained node is filtered to obtain the node to be processed; wherein the node data of the node to be processed includes text content; and the text content of the node to be processed is read aloud. In this way, the terminal device can filter the node data to obtain a node including text content, the terminal device can read aloud the text content of the filtered node, and the user can listen to the text content read aloud by the terminal device, thereby improving the user experience.

[0006] In one possible implementation, the node data includes one or more of the following: node attributes, component attributes, text attributes, coordinate attributes, and visibility attributes; wherein the node attribute represents whether the node has child nodes, the component attribute represents the component name of the component to which the node belongs, the text attribute is used to indicate the information content in the node, the coordinate attribute is used to indicate the location information of the node, and the visibility attribute is used to indicate whether the node is visible.

[0007] In one possible implementation, the acquired nodes are filtered according to the node data of the acquired nodes to obtain the nodes to be processed, including: according to the node attributes of the acquired nodes, removing non-leaf nodes in the acquired nodes to obtain a first node combination; wherein, the first node combination includes at least one node, and the nodes in the first node combination are all leaf nodes; according to the component attributes of the nodes in the first node combination, removing nodes whose component names do not meet the preset conditions to obtain a second node combination; wherein, the second node combination includes at least one node; according to the text attributes of the nodes in the second node combination, removing nodes whose information content is not text content to obtain a third node combination; wherein, the third node combination includes at least one node; according to the coordinate attributes of the nodes in the third node combination, removing nodes whose positions do not belong to the reading area to obtain a fourth node combination; wherein, the fourth node combination includes at least one node; according to the visibility attributes of the nodes in the fourth node combination, removing invisible nodes to obtain the nodes to be processed. In this way, the terminal device can filter the nodes based on the node data to obtain nodes including text content. The terminal device can read aloud the text content of the filtered nodes, and the user can listen to the text content read aloud by the terminal device, thereby improving the user experience.

[0008] In one possible implementation, based on the node attributes of the acquired nodes, non-leaf nodes are removed from the acquired nodes to obtain a first node combination. This includes: if the node attributes of the acquired nodes indicate that the node has child nodes, determining that the node is a non-leaf node and removing the node to obtain the first node combination. In this way, the terminal device can obtain leaf nodes that can be loaded with information content, so that the terminal device can subsequently perform further filtering on the first node combination to obtain nodes to be processed.

[0009] In one possible implementation, before obtaining the first node combination, the method further includes removing nodes whose component name is "Page View." Thus, when the component name of a leaf node is "WebView," the terminal device cannot obtain the text content displayed on the current interface based on this node. Therefore, the terminal device removes this node to improve the accuracy of the automatic reading method based on the terminal device.

[0010] In one possible implementation, the component name represented by the preset condition includes: a first name and / or a second name; wherein the first name is a component name associated with text, and the second name is a component name with a non-Android prefix. In this way, the terminal device can obtain a node with the first name and / or the second name as the component name, and the node may include text content.

[0011] In one possible implementation, based on the text attributes of the nodes in the second node combination, nodes whose information content is not text are removed to obtain a third node combination, including: if it is determined that the component name of the node in the second node combination is not a preset name, the first member variable under the text attribute of the node in the second node combination is retrieved; if it is determined that the first member variable under the text attribute of the node in the second node combination contains text content, the node is determined to be placed in the third node combination. In this way, the terminal device can retain nodes with valid text content based on the filtering process of the text attribute to obtain the third node combination, so that the terminal device can subsequently perform further filtering on the third node combination to obtain the node to be processed.

[0012] In one possible implementation, the method further includes: if it is determined that the component name of the node in the second node combination is a preset name, then calling the first member variable under the text attribute of the node in the second node combination; if it is determined that the first member variable under the text attribute of the node in the second node combination does not have text content, then calling the second member variable under the text attribute of the node in the second node combination; if it is determined that the second member variable under the text attribute of the node in the second node combination has text content, then determining that the node is placed in the third node combination. In this way, the terminal device can retain nodes with valid text content based on the filtering process of the text attribute to obtain the third node combination, so that the terminal device can subsequently perform further filtering processing on the third node combination to obtain the node to be processed.

[0013] In one possible implementation, the method further includes: if it is determined that the text content of a node in the third node combination is invisible control text, removing the node from the third node combination. In this way, the terminal device can remove the invisible control text from the text content, thereby achieving the effect of not reading the invisible control text aloud by the terminal device, thereby improving the user experience.

[0014] In a possible implementation, the coordinate attributes include one or more of the following: the left edge of the node, the right edge of the node, the width of the node, and the height of the node.

[0015] In one possible implementation, nodes that do not fall within the reading area are removed based on the coordinate attributes of the nodes in the third node combination to obtain a fourth node combination, including: if it is determined that the left edge of a node in the third node combination is greater than the right edge of the terminal device screen, or if it is determined that the right edge of a node in the third node combination is greater than the left edge of the terminal device screen, then the node is determined to not fall within the reading area and the node is removed from the third node combination; if it is determined that the width of a node in the third node combination is zero, or if it is determined that the height of a node in the third node combination is zero, then the node is determined to not fall within the reading area and the node is removed from the third node combination to obtain a fourth node combination. In this way, the terminal device can retain nodes with valid rectangular areas based on the filtering process using the coordinate attributes to obtain the fourth node combination, so that the terminal device can subsequently perform further filtering on the fourth node combination to obtain the node to be processed.

[0016] In one possible implementation, when the target window is a selected area specified by the user, the method further includes: if it is determined that a node in the third node combination is not located in the selected area, then determining that the location of the node does not belong to the reading area, and removing the node from the third node combination to obtain a fourth node combination. In this way, in the point-to-point reading mode, the terminal device can accurately read the text information of the node to be processed based on the selected area, thereby improving the user experience.

[0017] In one possible implementation, based on the visibility attribute of the nodes in the fourth node combination, invisible nodes are removed to obtain the nodes to be processed, including: calling the third member variable under the visibility attribute of the nodes in the fourth node combination; if the return value of the third member variable under the visibility attribute of the nodes in the fourth node combination indicates that they are visible, then determining that the node is the node to be processed. In this way, the terminal device can filter based on the visibility attribute, retaining nodes that are not obscured by other windows, to obtain the nodes to be processed, so that the terminal device can read the text content of the nodes to be processed.

[0018] In one possible implementation, the method further includes: if it is determined that the return value of the third member variable under the visible attribute of the node in the fourth node combination indicates that it is invisible, then obtaining the component category of the root node of the node and the bottom margin information of the node; if it is determined that the component category of the root node of the node is page view and the bottom margin information of the node is in the invisible area of ​​the terminal device screen, then correcting the node to be visible and determining the node as a node to be processed. In this way, the terminal device can automatically read aloud the text content in the invisible area at the bottom of the screen in the page view, simplifying the user's sliding operation and improving the user experience.

[0019] In one possible implementation, the method further includes: if it is determined that the return value of the third member variable under the visible attribute of the node in the fourth node combination indicates that it is invisible, calling a preset interface to obtain the occlusion information of the node; if the occlusion information of the node indicates that the node is covered by the system window and the height and width of the node are not scaled, correcting the node to be visible and determining the node as a node to be processed. In this way, the terminal device can read aloud the invisible nodes covered by the transparent system window, improving the accuracy of the automatic reading method based on the terminal device, and thereby improving the user experience.

[0020] In one possible implementation, reading aloud the text content of a node to be processed includes: segmenting the text content of the node to be processed to obtain multiple paragraphs to be processed; and performing speech conversion on the paragraphs to obtain and read aloud the speech corresponding to the paragraphs to be processed. This segmentation of long text content into short text content can improve the user experience when using the click-to-read mode and the playback of the previous / next paragraph function.

[0021] In one possible implementation, the method further includes: displaying a second page and generating a trigger indication in response to a user triggering a first button on the first page; wherein the first button is a button for instructing full text reading, and the target window indicated by the trigger indication is the current interface displayed by the terminal device; and the second page includes a second button, which is a button for instructing pause. Thus, in full text reading mode, the terminal device can automatically read aloud the text content of the current interface in response to the user clicking the play button.

[0022] The first page can be Figure 6 In the interface shown in a, the first button is the play button 602 in the unopened state, and the second page is Figure 6 The interface shown in b; the target window can be Figure 6 In the interface shown in b, the second button is the play button 604 in the turned-on state. The first page can also be Figure 6 In the interface shown in c, the first button is the floating ball 605 in the unopened state, and the second page is Figure 6 The target window can be Figure 6 In the interface shown in d, the second button is a floating ball 606 in the turned-on state.

[0023] In one possible implementation, the first page also includes a third button, which is a button for indicating point-to-read; the second page also includes a fourth button, which is a button for indicating point-to-read; the method further includes: displaying the third page in response to the user triggering the third button on the first page, or in response to the user triggering the fourth button on the second page; wherein the third page includes a preset icon, and the preset icon indicates that the terminal device is in point-to-read mode; in response to the user selecting the current interface of the terminal device, determining the selected area and generating a trigger indication; wherein the target window indicated by the trigger indication is the selected area. In this way, in the point-to-read mode, the terminal device can automatically read the text content of the selected area in response to the user clicking the text content in the current interface.

[0024] The first page can be Figure 7 The second page is the interface shown in a. Figure 6 The interface shown in b; the third button is Figure 7 The fourth button is the selection mode button 701 in the interface shown in a; Figure 6 The third page can be Figure 7 In the interface shown in b, the default icon is Figure 7 The suspended ball 702 in the interface shown in b.

[0025] In one possible implementation, the method further includes: setting a preset option on the fourth page to a selected state; wherein the fourth page is used to display whether the terminal device supports automatic reading, and the selected state indicates that automatic reading is enabled. In this way, the terminal device can enable the "on-demand reading" function for the preset option.

[0026] Among them, the fourth page may correspond to Figure 3 In the interface shown in b, the preset option may be a switch button 302 for reading the content; or the fourth page may correspond to Figure 4 In the interface shown in b, the preset option may be the “Read on Demand” icon 401.

[0027] In a second aspect, embodiments of the present application provide a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver functions, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0028] The terminal device includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method of the first aspect.

[0029] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is run, it enables the computer to execute the method of the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method of the first aspect.

[0032] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the structure of a terminal device 100 provided in an embodiment of the present application;

[0034] Figure 2A schematic diagram of the software structure of a terminal device 100 provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0040] Figure 8 A flowchart of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0041] Figure 9 A flowchart of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0042] Figure 10 A schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0043] Figure 11 A schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application;

[0044] Figure 12 An internal interaction flow chart of a terminal device provided in an embodiment of the present application;

[0045] Figure 13 A schematic structural diagram of an automatic reading device based on a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0047] (1) Accessibility Service: Accessibility Service can be used to help users use Android devices and applications. The terminal device can run the accessibility service in the background and receive system callbacks when accessibility events (AccessibilityEvents) are triggered. Accessibility events represent some state transitions in the user interface, for example, the focus has changed, a button has been clicked, and so on. The accessibility service can optionally request the ability to query the content of the active window. The accessibility service is a system-level API that can simulate user operations. After the accessibility service is turned on on the terminal device, the terminal device can execute the process indicated by the simulated operation based on the simulated operation.

[0048] 2) Other terms

[0049] In the embodiments of this application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not define their order. Those skilled in the art will understand that terms such as "first" and "second" do not define the quantity or execution order, and do not necessarily define differences.

[0050] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0051] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0052] When a user uses a terminal device equipped with a touch screen, the user can use corresponding functions of the terminal device based on the text content on the touch screen. For example, the user can view the text content of a chat message sent by others on the terminal device screen. For another example, the user can read the text content of a novel on the terminal device screen.

[0053] However, in some cases, users may need to access text content on the screen but may find it difficult to view it on their terminal device. For example, users may find it difficult to view the screen while washing, feeling tired, or navigating. Another example is that some users with dyslexia cannot fully utilize the services provided by the terminal device, affecting their user experience.

[0054] In view of this, an embodiment of the present application provides an automatic reading method based on a terminal device. By responding to a trigger indication, node data of a node in a target window in the current interface of the terminal device is obtained; based on the obtained node data, the obtained node is filtered to obtain a node to be processed; wherein the node data of the node to be processed includes text content; and the text content of the node to be processed is read aloud. In this way, the terminal device can filter the node data to obtain nodes including text content, and the terminal device can read aloud the text content of the filtered node. The user can listen to the text content read aloud by the terminal device, thereby improving the user experience.

[0055] In the embodiments of the present application, the terminal device 100 may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone with a touch screen, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0056] In order to better understand the embodiment of the present application, the structure of the terminal device 100 according to the embodiment of the present application is introduced below:

[0057] Figure 1 FIG1 shows a schematic diagram of the structure of the terminal device 100. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a touch screen 194, and a subscriber identification module (SIM) card interface 195.

[0058] Among them, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0059] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0060] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0061] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0062] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the instruction or data again, it can retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0063] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuits sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0064] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is for illustrative purposes only and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0065] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also provide power to the terminal device via the power management module 141.

[0066] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the touch screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0067] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0068] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. The antennas in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0069] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0070] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the touch screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0071] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0072] In some embodiments, antenna 1 of terminal device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that terminal device 100 can communicate with the network and other devices through wireless communication technology.

[0073] The terminal device 100 implements display functions through a GPU, touch screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the touch screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0074] The touch screen 194 is used to display images, videos, etc. The touch screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, the terminal device 100 may include one or N touch screens 194, where N is a positive integer greater than one.

[0075] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0076] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0077] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0078] Exemplarily, the terminal device 100 may further include one or more of a button 190 , a motor 191 , an indicator 192 , a SIM card interface 195 (eSIM card), and the like.

[0079] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture, etc. In the embodiment of the present application, the Android system with a layered architecture is used as an example to exemplify the software structure of the terminal device 100.

[0080] Figure 2 It is a software structure block diagram of the terminal device 100 according to an embodiment of the present application.

[0081] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0082] The application layer can include a series of application packages.

[0083] like Figure 2 As shown, the application layer may include settings, video, map, phone, music, camera, gallery, text to speech (TTS), and on-demand reading applications.

[0084] Among them, TTS is used to convert language text into speech audio and output the speech audio. On-demand reading can be used to enable electronic devices to read text content on the touch screen.

[0085] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0086] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a resource manager, a view system, an accessibility service, an on-demand reading service, an activity manager, and the like.

[0087] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, touch the screen, drag the screen, take screenshots, etc.

[0088] Content providers are used to store and retrieve data and make it accessible to applications. Data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0089] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0090] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0091] The AccessibilityService module is a system-level API that can simulate user operations. After the terminal device obtains accessibility service permissions, it can simulate user operations to control the user's terminal device. In this embodiment of the application, the AccessibilityService can be used to obtain the view node in the window.

[0092] On-demand reading service: This embodiment of the present application can be used to perform an automatic reading method based on a terminal device. The on-demand reading service module may include a node extractor, a node filter, and a text segmenter. The node extractor can be used to obtain node data for nodes in the target window of the terminal device; the node filter is used to filter nodes; and the text segmenter is used to segment the text content of the nodes.

[0093] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0094] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0095] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0096] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0097] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0098] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0099] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0100] A 2D graphics engine is a drawing engine for 2D drawings.

[0101] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0102] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other. For the same or similar concepts or processes, some embodiments may not be described in detail.

[0103] To facilitate the introduction of the automatic reading method based on a terminal device provided in the embodiment of the present application, the nouns in the embodiment of the present application are first explained.

[0104] Text content: can be text information displayed on the touch screen of the terminal device, which can be understood as: text content is text information that the user can see on the terminal device. In the embodiment of the present application, the terminal device can automatically read the text content on the touch screen through the "on-demand reading" function.

[0105] It should be noted that the "on-demand reading" function in the embodiments of the present application can provide two reading modes: full-text reading and click-to-read. Full-text reading can be understood as: in full-text reading mode, the terminal device can automatically read all the text content on the touch screen from top to bottom. Click-to-read mode can be understood as: in click-to-read mode, when the terminal device receives a trigger operation for a certain text content on the touch screen, the terminal device can read the text content aloud.

[0106] The target window may be all or part of the current interface displayed on the touch screen of the terminal device. For example, when the terminal device activates the full-text reading mode, the target window may be the current interface displayed on the touch screen; when the terminal device activates the point-to-point reading mode, the target window may be the area in the current interface displayed on the touch screen that the terminal device selects in response to a user's triggering operation on the touch screen.

[0107] A node can be a piece of data representing a target window. For example, a touch screen of a terminal device can display multimedia content, which can include text, audio, image, and video content. A node can be matched with the multimedia content in the target window, and the node data of the node can be used to render and display the corresponding multimedia content in the target window. It is understood that in embodiments of the present application, the terminal device can filter the nodes of the target window to obtain nodes that include text content.

[0108] The following combination Figure 3-Figure 7 The "on-demand reading" function in the embodiment of the present application and the two reading modes of full-text reading and point-to-point reading are explained.

[0109] For example, Figure 3 The following is a schematic diagram showing two interface diagrams for enabling the automatic reading method based on the terminal device provided in the embodiment of the present application. Figure 3 As shown:

[0110] In one possible implementation, the terminal device may enter a setting interface from the main interface, and the setting interface may include a setting interface for accessibility services. The terminal device may activate the "on-demand reading" function based on the accessibility services.

[0111] For example, when the terminal device enters the setting interface of the accessibility service, the terminal device may display the following Figure 3 The interface shown in a. Figure 3 The interface shown in a in FIG includes settings for hearing and vision, among which the hearing settings may include prompt texts such as screen reading, on-demand reading, text-to-speech, and mono audio, as well as corresponding setting buttons. When the terminal device receives a trigger operation for the setting button 301 for "on-demand reading", the terminal device may enter the following settings: Figure 3 The interface shown in b. Figure 3 The interface shown in b in FIG. 3 may include a switch button 302 for reading content and information about the on-demand reading function. The switch button 302 for reading content may be in an off state. When the terminal device receives a trigger operation for the switch button 302 for reading content, the terminal device may enable the "on-demand reading" function and execute the terminal device-based automatic reading method provided in an embodiment of the present application.

[0112] For example, when the terminal device receives a trigger operation of the switch button 302 for reading content, the terminal device may enter the following Figure 3 The interface shown in c. Figure 3The interface shown in c in FIG. 3 may include a floating box 303. The floating box 303 may include multiple buttons. For example, the floating box 303 may include at least one of the following: a thumbnail button 304, a selection mode button 305, a play button 307, a play previous paragraph button 306, a play next paragraph button 308, a speed-up button 309, a drag button 310, a full-text reading mode icon 311, and an estimated reading time.

[0113] In the embodiment of the present application, the thumbnail button 304 can be used to abbreviate the floating frame 303 into a floating ball. The function of the thumbnail button will be introduced later in the embodiment of the present application and will not be repeated here.

[0114] In the embodiment of the present application, the click mode button 305 can be used to switch the reading mode of "on-demand reading", switching the reading mode from the full-text reading mode to the click reading mode. In a possible implementation, when the terminal device uses the "on-demand reading" function for the first time or the first time each day, the terminal device can select the full-text reading mode by default, such as Figure 3 In the interface shown in c in FIG, an icon 311 for the full-text read-aloud mode is displayed in a floating box. When a user wishes to use the click-to-read mode, the user may click the click-to-read mode button 305. When the terminal device receives a trigger operation of the click-to-read mode button 305 by the user, the terminal device may switch to the click-to-read mode. The click-to-read mode will be described in detail later in the embodiments of this application and will not be repeated here.

[0115] In the embodiment of the present application, the play button 307 can be used to automatically read aloud the text content in the current interface. The current interface can be an interface displayed on the touch screen that does not exceed the touch screen specifications, or an interface displayed on the touch screen that does not exceed the touch screen specifications or an interface not displayed on the touch screen.

[0116] For example, a user can browse the web version of news on a terminal device. After reading the text on the current interface, the user can slide up to pull out the unread text behind the read text. In some embodiments, when the terminal device starts the "select to read aloud" function, the terminal device can automatically read aloud the text on the touch screen. After the terminal device finishes reading aloud the text on the current touch screen, the terminal device can stop reading aloud. In other embodiments, when the terminal device starts the "select to read aloud" function, the terminal device can automatically read aloud the text on the touch screen. After the terminal device finishes reading aloud the text on the current touch screen, the terminal device can continue to read aloud the text that is not displayed on the touch screen. The embodiments of the present application are not limited to this.

[0117] In an embodiment of the present application, the play previous paragraph button 306 can be used to read aloud the previous paragraph of the current text. The play next paragraph button 308 can be used to read aloud the next paragraph of the current text. It is understandable that in order to improve the user experience, the terminal device can read aloud the text in segments. For example, when a user uses the "read on demand" function to listen to the reading content of the terminal device, there may be a situation where the text content is not heard clearly. In this case, the terminal device can receive the user's trigger operation on the play previous paragraph button 306 and replay the previous paragraph of text. Or the user may not be interested in the current text and want to skip the current text. In this case, the terminal device can receive the user's trigger operation on the play next paragraph button 308 and play the next paragraph of text. The terminal device can segment the text according to paragraphs, sentences and / or fixed characters, etc. The embodiment of the present application does not limit the text segmentation rules.

[0118] In the embodiment of the present application, the speed button 309 can be used to set the speed of the terminal device reading aloud. The embodiment of the present application does not limit the speed value of the reading aloud speed.

[0119] In an embodiment of the present application, the drag button 310 can be used to adjust the display position of the floating box 303. To reduce the obstruction of the content on the touch screen by the floating box 303 or to facilitate user operation, the user can use the drag button 310 to adjust the display position of the floating box 303. In one possible implementation, when the terminal device uses the "Select to Read Aloud" function for the first time or the first time each day, the terminal device may display a prompt such as "Press and hold the top to move" above the floating box 303 to remind the user that the display position of the floating box 303 can be adjusted.

[0120] In the embodiment of the present application, the estimated reading time can be used to prompt the user the reading time of the current interface. The terminal device can calculate the estimated reading time according to the number of words in the text on the screen and the reading speed.

[0121] In the above implementation, the terminal device can enter the setting interface of the on-demand reading function based on the setting interface of the barrier-free function, and turn on the "on-demand reading" function in the setting interface of the on-demand reading function, such as Figure 3 In addition, the embodiment of the present application also provides a convenient implementation method, the terminal device can also open the "on-demand reading" function based on the drop-down menu bar, such as Figure 4 shown.

[0122] For example, the terminal device may display an interface to be read aloud. When the terminal device receives a trigger operation of sliding the screen downward in the interface to be read aloud, the terminal device may enter the following Figure 4 The interface shown in a. Figure 4The interface shown in a can be a drop-down menu bar interface in the unfolded state. The interface may include icons of some applications, such as a wireless network icon, a Bluetooth icon, a flashlight icon, etc. The icons in the interface can be understood as shortcut switches for the application. For example, the terminal device can turn on the Bluetooth connection when it receives a trigger operation for the icon of the Bluetooth application.

[0123] When the terminal device is Figure 4 When a trigger operation of sliding the screen downward is received in the interface shown in a, the terminal device may display the following Figure 4 The interface shown in b. Figure 4 The interface shown in b in the figure may be a drop-down menu bar interface in an expanded state, which may include icons of all applications for which shortcut switches have been set. The icons of applications for which shortcut switches have been set may include the "Select to Read" icon 401. Figure 4 The "Select to Read Aloud" icon 401 in the interface shown in b is in the off state. When the terminal device receives a trigger operation for the "Select to Read Aloud" icon 401, the "Select to Read Aloud" icon 401 can be switched to the on state and the "Select to Read Aloud" function is started, and the terminal device enters the state as shown in FIG. Figure 4 The interface shown in c.

[0124] It should be noted that the icons of applications that have been set up with a quick switch may not include the "Select to Read Aloud" icon 303. The terminal device can edit the quick switch and add the "Select to Read Aloud" quick switch in the drop-down menu bar. The terminal device can then quickly launch the "Select to Read Aloud" function based on the "Select to Read Aloud" icon 401 in the drop-down menu bar.

[0125] exist Figure 4 In the interface shown in c in FIG, the terminal device can return to the interface to be read aloud. At this time, a floating box 402 appears in the interface to be read aloud. The terminal device can respond to the user's triggering operation on the icon in the floating box to implement the corresponding on-demand reading function. The floating box 402 can refer to Figure 3 The relevant description of the interface shown in c in the present application will not be repeated here.

[0126] The above embodiments are combined Figure 3 and Figure 4 Two methods of entering the "Select to Read Aloud" function are introduced. It can be understood that when the terminal device turns on the "Select to Read Aloud" function, a floating box of "Select to Read Aloud" can pop up in the current interface of the terminal device. In order to improve the aesthetics of the display interface and reduce the obstruction of the floating box on the displayed content, the embodiment of the application also provides a form of floating ball, such as Figure 5 As shown:

[0127] In one possible implementation, the terminal device may display Figure 5 The interface shown in a, Figure 5 The interface shown in a in FIG includes a floating box 501, and a thumbnail button 502 is displayed in the floating box 501. When the terminal device receives a trigger operation for the thumbnail button 502, the terminal device can enter the following Figure 5 The interface shown in b. Figure 5 The interface shown in b in the figure can display a floating ball 503. At this time, the floating ball 503 is in the floating ball state in the full text reading mode. There is a difference between the icon of the floating ball in the reading mode and the icon of the floating ball 503, which will be explained later. The floating ball 503 can be understood as the floating box 502 in the retracted state.

[0128] exist Figure 5 In the interface shown in b in FIG, if the terminal device does not receive a trigger operation for reading text content within a period of time, the terminal device may Figure 5 Switch to the interface shown in b Figure 5 In the interface shown in c in the figure, the floating ball can be floating ball 504. It is understood that both floating ball 503 and floating ball 504 are in the floating ball state in the full-text reading mode. When the user turns on the "Select to Read" function but does not use the "Select to Read" function to read text content for a period of time, the terminal device can switch the floating ball 503 to the floating ball 504 to reduce the obstruction of the current interface by the floating ball and improve the user experience.

[0129] In another possible implementation, the terminal device may display Figure 5 If the terminal device does not receive a trigger operation in the floating box 501 within the preset time, the terminal device enters the interface shown in a. Figure 5 In the interface shown in b, the floating box 501 is automatically abbreviated to a floating ball 503. Among them, the preset time can be set to 5 seconds. In the embodiment of the present application, the preset time can be customized, and the embodiment of the present application does not limit this.

[0130] The following combination Figure 6 The use scenarios of the floating frame and the floating ball in the embodiments of this application are described respectively. It should be noted that, Figure 6 Taking the full text reading mode as an example, the automatic reading method based on the terminal device provided in the embodiment of the present application is described. Figure 6 The following is a schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application. Figure 6 As shown:

[0131] In one possible implementation, after the terminal device starts the "Select to Read" function, the terminal device displays a floating box, such as Figure 6 The a interface in Figure 6 As shown in interface b.

[0132] When the terminal device starts the "Select to Read" function, the terminal device can display the following Figure 6 The interface shown in a in may be the interface to be read aloud. Figure 6 The interface shown in a in FIG includes the text content to be read aloud and a floating box 601, wherein the floating box 601 includes a play button 602, wherein the icon of the play button 602 can be displayed in an unactivated state (the unactivated state can be understood as a state in which the terminal device does not read aloud the text content). When the terminal device receives a trigger operation for the play button 602, the terminal device can enter the following state: Figure 6 The interface shown in b. Figure 6 The interface shown in b in FIG may include text content to be read aloud and a floating box 603. The floating box 603 may include a play button 604, wherein the icon of the play button 604 may be displayed as an on state (the on state may be understood as the state in which the terminal device is reading aloud the text content). The terminal device may read aloud the text content in the current interface. When the terminal device finishes reading aloud, the icon of the play button 604 returns to an off state (as shown in FIG. Figure 6 ).

[0133] In some embodiments, when the terminal device is reading aloud the text content, the terminal device may display the estimated reading time. In one possible implementation, the terminal device may calculate the estimated time for reading aloud all the text content in the current interface, and display the estimated reading time in a floating box. When the terminal device is reading aloud the text content in the current interface, the prompt text of the estimated reading time in the floating box does not change. In another possible implementation, the terminal device may calculate the estimated time for reading aloud the text content that has not been read aloud in the current interface, and display the estimated reading time in a floating box. When the terminal device is reading aloud the text content in the current interface, the prompt text of the estimated reading time in the floating box changes, and the estimated reading time starts counting down from the start of reading until the terminal device finishes reading the text content in the current interface.

[0134] In another possible implementation, after the terminal device starts the "Select to Read Aloud" function, the terminal device displays a floating ball, such as Figure 6 The c interface in Figure 6 As shown in the d interface.

[0135] When the terminal device starts the "Select to Read" function, the terminal device can display the following Figure 5 The interface shown in c in may be the interface to be read aloud. Figure 6 The interface shown in c includes the text content to be read aloud and the floating ball 605, wherein the icon of the floating ball 605 can be displayed as an unopened state. When the terminal device receives a trigger operation for the floating ball 605, the terminal device can enter the following Figure 6 The interface shown in d. Figure 6 The interface shown in d in FIG may include the text content to be read aloud and the floating ball 606, and the icon of the floating ball 606 may be displayed as an open state. The terminal device reads the text content on the screen. When the terminal device finishes reading aloud, the icon of the floating ball 606 returns to the unopened state (such as Figure 6 (the icon of the floating ball 605 in the interface shown in c).

[0136] In some embodiments, while a terminal device is reading aloud text content, the terminal device may display the location of the text currently being read aloud. For example, the terminal device may highlight the text being read aloud. For example, the terminal device may highlight the text being read aloud by changing the size, font, color, or adding a border. In an embodiment of the present application, the terminal device may also not process the text being read aloud and not highlight the text being read aloud. The embodiment of the present application does not limit the method for highlighting the text being read aloud.

[0137] In some embodiments, the terminal device may display a floating box or a floating ball, and when the icon of the play button in the floating box shows an on state or the icon of the floating ball shows an on state, the terminal device reads aloud the text content on the screen. In one possible implementation, when the terminal device reads aloud the text content on the screen, the terminal device may maintain the interface shown on the screen until the terminal device finishes reading aloud all the text content in the interface. It can be understood that when the terminal device is in the reading state, the user cannot set the area outside the floating box or floating ball on the screen, and the text content displayed on the terminal device will not change. In another possible implementation, when the terminal device is in the reading state, the terminal device can switch the interface shown on the screen, and the text content in the interface may change. The embodiments of the present application are not limited to this.

[0138] The above embodiment describes the full text reading mode of the "on-demand reading" function in the embodiment of the present application. Figure 7 The click-to-read mode of the "on-demand reading" function in the embodiment of the present application is described. Figure 7 The following is a schematic diagram of an interface of an automatic reading method based on a terminal device provided in an embodiment of the present application. Figure 7 As shown:

[0139] Understandably, in some cases, users may wish to hear a portion of the text on the screen. For example, a user may encounter an unfamiliar character while reading a paragraph. Another example is a user wanting to learn and follow the pronunciation of a paragraph. In such cases, the terminal device can utilize the "Select to Read" feature in the embodiments of this application. In this mode, the terminal device can read aloud the text content selected by the user.

[0140] For example, after the "Select to Read" function is enabled, the terminal device may display the following Figure 7 The interface shown in a. Figure 7 The interface shown in a in FIG may include a click mode button 701. When the terminal device receives a trigger operation for clicking the mode button 701, the terminal device may enter Figure 7 The interface shown in b. Figure 7 The interface shown in b in FIG. 1 includes text content to be read aloud and a floating ball 702 in the click-to-read mode, and the terminal device turns on the click-to-read mode.

[0141] It should be noted that the form of the floating ball 702 in the click reading mode is different from that in the full text reading mode. The floating ball in the full text reading mode can be referred to Figure 5 The floating ball 503 in the interface shown in b is not described here in detail.

[0142] In the click-to-read mode, the terminal device may receive a trigger operation from the user to select an area in the current interface, and in response to the operation, the terminal device may read aloud the text content in the selected area. Figure 7 In the interface shown in b, the terminal device can receive a user's click operation on the text content of "We crossed the river and entered the station" on the screen. In response to the click operation, the terminal device can enter Figure 7 The interface shown in c in FIG, and the terminal device reads the text content "We crossed the river and entered the station".

[0143] In one possible implementation, when the terminal device reads aloud the text content of the selected area, the touch screen of the terminal device can highlight the selected area to prompt the user where the text content currently being read is located. For example, the terminal device can highlight the selected area in the following manner: Figure 7 As shown in the c interface. Figure 7 The c interface may display the text “We crossed the river and entered the station” and a border around the text content (selected area 704 ).

[0144] In some embodiments, after the terminal device finishes reading the text content in the selected area, it can automatically exit the click-to-read mode, and clicking the floating ball 702 in the read-aloud mode can switch to the floating ball 703 in the full-text read-aloud mode, such as Figure 7 The b interface and Figure 7 In some other embodiments, after the terminal device finishes reading aloud the text content in the selected area, it can maintain the click-to-read mode. When the terminal device receives a trigger operation from the user to select an area in the current interface again, the terminal device can continue to read aloud the text content in the selected area. This embodiment of the present application does not impose any restrictions on this.

[0145] It should be understood that the interface provided in the embodiment of the present application is merely an example and does not constitute a limitation on the embodiment of the present application.

[0146] The above embodiments describe the use scenarios of the automatic reading method based on the terminal device provided by the embodiments of the present application. Figure 8 The process of the automatic reading method based on the terminal device provided in the embodiment of the present application is described. Figure 8 FIG. 1 shows a flow chart of an automatic reading method based on a terminal device provided in an embodiment of the present application. Figure 8 As shown:

[0147] S801. In response to a trigger instruction, obtain node data of a node in a target window in a current interface of a terminal device.

[0148] The target window may be the current interface displayed on the touch screen of the terminal device. For example, in the full text reading mode, the target window may correspond to Figure 6 The current interface shown in a in the target window is displayed. Figure 6 The target window can also be a part of the current interface displayed on the touch screen of the terminal device. For example, in the point-to-point reading mode, the target window can correspond to Figure 7 In the selected area 704 in the interface shown in c, the target window displays the text content in the selected area 704.

[0149] It can be understood that the target window is related to the window currently displayed by the terminal device. When the current interface displayed by the terminal device changes, the target window also changes, and the nodes in the target window also change accordingly. The target window in the embodiment of the present application is not limited to a fixed window.

[0150] The terminal device can select any of the two target windows based on the trigger indication. The trigger indication is used to indicate the reading of the information on the target window in the current interface displayed by the terminal device, for example: Figure 6 In the interface shown in a, when the terminal device receives a trigger operation for the play button 602, the terminal device displays Figure 6 The interface shown in b in FIG, and the terminal device generates a trigger instruction. At this time, the target window indicated by the trigger instruction is the current interface displayed by the terminal device, such as Figure 6 The interface shown in b. For example: Figure 7 In the interface shown in b, when the terminal device receives the user's selection operation on the current interface, the terminal device determines the selected area 704 and generates a trigger indication. At this time, the target window indicated by the trigger indication is the selected area, such as Figure 7 The selected area 704 in the interface shown in c.

[0151] A node may represent a piece of data in a target window, and the data may include but is not limited to: data corresponding to text content, data corresponding to image content, data corresponding to video content, and data corresponding to audio content.

[0152] When a terminal device receives a trigger indication, the terminal device may obtain node data for the nodes in the target window in the current interface of the terminal device in response to the trigger indication. In one possible implementation, the terminal device uses an accessibility service to obtain node data for all nodes in the target window. For example, the terminal device may call the AccessibilityService interface to obtain the target window, and the AccessibilityService will return the target window and all nodes under the target window.

[0153] S802 : Filter the acquired nodes according to the acquired node data to obtain nodes to be processed; wherein the node data of the nodes to be processed include text content.

[0154] Node data includes one or more of the following: node attributes, component attributes, text attributes, coordinate attributes, and visibility attributes. Node attributes indicate whether a node has child nodes, component attributes indicate the component name of the component to which the node belongs, text attributes indicate the information content of the node, coordinate attributes indicate the location of the node, and visibility attributes indicate whether the node is visible. Node data will be further described in the following embodiments of this application and will not be repeated here.

[0155] The terminal device may filter the nodes based on the node data to obtain nodes to be processed, wherein the node data of the nodes to be processed includes text content. It is understood that after filtering, the terminal device may obtain nodes in the target window that match the text content, and the nodes to be processed may be one or more nodes.

[0156] S803: Read aloud the text content of the node to be processed.

[0157] The node to be processed may represent data corresponding to the text content in the target window. Accordingly, the terminal device may obtain the text content corresponding to the node to be processed based on the node to be processed, and the terminal device may read the text content aloud.

[0158] In one possible implementation, the terminal device may call a text-to-speech (TTS) application at the application layer to convert the text content into voice information and play the voice information. The terminal device of the embodiment of the present application may also use other methods to read the text content aloud, and the embodiment of the present application does not limit this.

[0159] The present application provides an automatic reading method based on a terminal device. In response to a trigger indication, the method obtains node data of a node in a target window in the current interface of the terminal device; filters the obtained node based on the obtained node data to obtain a node to be processed; wherein the node data of the node to be processed includes text content; and reads aloud the text content of the node to be processed. In this way, the terminal device can filter the node data to obtain nodes including text content, and the terminal device can read aloud the text content of the filtered node. The user can listen to the text content read aloud by the terminal device, thereby improving the user experience.

[0160] The following combination Figure 9 The process of node filtering in step S802 is described as follows. Figure 9 As shown:

[0161] S901: Remove non-leaf nodes from the acquired nodes based on the acquired node data to obtain a first node combination, wherein the first node combination includes at least one node, and all nodes in the first node combination are leaf nodes.

[0162] S902: According to the component attributes of the nodes in the first node combination, remove the nodes whose component names do not meet the preset conditions to obtain a second node combination, wherein the second node combination includes at least one node.

[0163] S903: Remove nodes whose information content is not text content according to the text attributes of the nodes in the second node combination to obtain a third node combination, wherein the third node combination includes at least one node.

[0164] S904: Remove nodes that are not located in the reading area according to the coordinate attributes of the nodes in the third node combination to obtain a fourth node combination, wherein the fourth node combination includes at least one node.

[0165] S905 : Remove invisible nodes according to the visibility attributes of the nodes in the fourth node combination to obtain nodes to be processed.

[0166] In an embodiment of the present application, the terminal device can filter all nodes in the target window based on the order of steps S901-S905 to obtain the nodes to be processed corresponding to the valid text content in the target interface, so that the terminal device can read aloud the text content of the nodes to be processed.

[0167] The filtering process of steps S901 to S905 is further described below.

[0168] Exemplarily, S901 , according to the acquired node data of the nodes, remove non-leaf nodes from the acquired nodes to obtain a first node combination.

[0169] There are direct or indirect connections between nodes in the target window. For example, the target window may include one or more root nodes, any root node may be connected to a number of first child nodes, any first child node may be connected to a number of second child nodes, and so on. The terminal device may determine whether the node has child nodes in the connection relationship based on the node attributes of the node. For example, the node may be a root node, a first child node, a second child node, etc.

[0170] If the node attribute of the node acquired by the terminal device indicates that the node has child nodes, the node is determined to be a non-leaf node, and the node is removed to obtain a first node combination.

[0171] It can be understood that the node attribute of a node can represent whether the node has child nodes. If there are no child nodes under the node, the terminal device can determine that the node is a leaf node; otherwise, the node is a non-leaf node.

[0172] For example, the terminal device can use the AccessibilityService interface to obtain all nodes in the target window. In the embodiment of the present application, the connection relationship can exist in the form of a node list, which can include node data and node attributes of each node in the target window.

[0173] The following describes the leaf nodes in conjunction with the node list. For example, the node list may be as follows:

[0174] RelativeLayout[123,123][456,456]

[0175] LinearLayout[123,123][400,400]

[0176] TextView: Weather[123,123][400,400]

[0177] Among them, the leaf node can be TextView:Weather[123,123][400,400], and the non-leaf node can be RelativeLayout[123,123][456,456], LinearLayout[123,123][400,400].

[0178] It should be noted that in the embodiments of the present application, leaf nodes can carry text content; non-leaf nodes can be understood as containers for leaf nodes. Non-leaf nodes generally do not carry text content, and when non-leaf nodes do contain text content, the terminal device displays the text content of the non-leaf nodes as invisible text. Based on this, the terminal device can filter out non-leaf nodes from the node data, retaining leaf nodes that can carry text content, and obtain a first node combination.

[0179] Optionally, the terminal device may remove the node whose component name is the page view WebView.

[0180] The node data includes component attributes, and the component name of the node can represent the component attributes of the node. It should be noted that when the interface displayed by the target window is a web page, some nodes in the target window may be nodes with a component name of WebView. In some embodiments, the node with a component name of WebView may be a leaf node, and the node includes text content, but the terminal device will not display the text content in the node during rendering. Therefore, when it is detected that the node with a component name of WebView is a leaf node, the terminal device corrects the node to a non-leaf node, and the terminal device filters out the non-leaf node based on step S901; when it is detected that the node with a component name of WebView is a non-leaf node, the terminal device can directly filter out the non-leaf node through step S901, that is, the terminal device can remove the node with a component name of WebView.

[0181] It is understandable that after obtaining the node of the target window, the terminal device can filter the node to obtain the node to be processed. In the embodiment of the present application, the terminal device filters out a node or removes a node, which can be understood as the terminal device screening out the node when performing the filtering process; the node does not participate in the subsequent filtering process, but the node still exists in the connection relationship and is not deleted. The terminal device can still trace back to the filtered node later, but this embodiment of the application does not explain this.

[0182] The automatic reading method based on the terminal device provided in the embodiment of the present application obtains a first node combination by removing non-leaf nodes from the obtained nodes according to the node data of the obtained nodes; in this way, the terminal device can obtain leaf nodes that can be loaded with information content, so that the terminal device can subsequently further filter and process the first node combination to obtain the node to be processed.

[0183] Step S902 is described below. Exemplarily, in step S902, based on the component attributes of the nodes in the first node combination, nodes whose component names do not meet the preset conditions are removed to obtain a second node combination. The second node combination includes at least one node.

[0184] The component name can represent the component properties of the node, and the component properties can be properties related to text, pictures, videos, or audio. The component name can be the package name of the node. For example, take the node in the first node combination as TextView: Weather [123, 123][400, 400] in the above node list as an example, where the component name of the node can be TextView, the text content is Weather, and [123, 123][400, 400] is the coordinate of the node in the current interface.

[0185] The terminal device can set a preset condition. When the component name of a node does not meet the preset condition, the terminal device can remove the node. The component name represented by the preset condition includes: a first name and / or a second name; wherein the first name is a component name related to the text, and the second name is a component name with a non-Android prefix.

[0186] For example, the component with the first name may include an android.widget component and an android.view.View view component for representing text content; wherein the android.widget component may include but is not limited to the following components:

[0187] android.widget.TextView; text view component.

[0188] android.widget.EditText; text editing box component.

[0189] android.widget.AutoCompleteTextView; automatically completes the text box component.

[0190] android.widget.MultiAutoCompleteTextView;Multi-auto-complete text box component;

[0191] android.widget.Button; button component.

[0192] android.widget.CheckBox; multiple selection box.

[0193] android.widget.CheckedTextView; a selectable text box component.

[0194] android.widget.RadioButton; radio button component.

[0195] android.widget.Switch; switch component.

[0196] In an embodiment of the present application, the components of the first name may include the above-mentioned components, and the terminal device may determine whether the node is the node of the first name based on the component name of the node. For example, when the component name of the node does not include the above-mentioned component name, the terminal device may filter out the node.

[0197] For example, the second name can be a component name with a non-Android prefix. A component name with a non-Android prefix can be a custom View node. For a custom View node, the terminal device cannot determine whether text content exists in the node based on the component name. The terminal device cannot directly remove the node with a non-Android prefix. In this case, the terminal device may retain the node and subsequently perform other filtering operations to determine the node.

[0198] It should be noted that, in the embodiment of the present application, the terminal device may filter the nodes in the first node group based on the first name; the terminal may also filter the nodes in the first node group based on the first name and the second name. The embodiment of the present application does not impose any restrictions on this.

[0199] The terminal device-based automatic reading method provided in an embodiment of the present application obtains a second node combination by removing nodes whose component names do not meet preset conditions based on the component attributes of the nodes in the first node combination. In this way, the terminal device can filter based on the component attributes, retaining nodes whose component names are relevant to the text content, to obtain the second node combination, so that the terminal device can subsequently perform further filtering on the second node combination to obtain the node to be processed.

[0200] It is understandable that in the embodiment of the present application, the nodes in the second node combination may carry visible text content, but in actual operation, the second node may carry empty text. The terminal device can filter the nodes carrying empty text through step S903.

[0201] Exemplarily, S903: Based on the text attributes of the nodes in the second node combination, remove the nodes whose information content is not text content to obtain a third node combination. The third node combination includes at least one node, including:

[0202] The text attribute can indicate whether the node contains text content. When the terminal device detects that the text attribute of the node indicates that the text content of the node is empty, the terminal device removes the node. The process of the terminal device determining whether the node contains text content can be shown in steps S1001-1003:

[0203] S1001. The terminal device obtains the component name of the node in the second node combination, and determines whether the component name of the node in the second node combination is a preset name.

[0204] For example, the preset name may be android.view.View. When the component name of the node in the second node combination is android.view.View, the terminal device may execute steps S1004-S006; when the component name of the node in the second node combination is not android.view.View, the terminal device may execute steps S1002-S1003.

[0205] S1002: If it is determined that the component name of the node in the second node combination is not a preset name, then call the first member variable under the text attribute of the node in the second node combination.

[0206] The text attribute may include a first member variable and a second member variable, the first member variable may be node.getText, and the second member variable may be node.getContentDescription; node.getText is used to obtain the text content of the Text in the node, and node.getContentDescription is used to obtain the text content of the ContentDescription in the node, wherein ContentDescription can be used to describe the text content of the accessibility service.

[0207] When the component name of the node in the second node combination is not android.view.View, the terminal device calls node.getText of the node.

[0208] S1003: If it is determined that the first member variable under the text attribute of the node in the second node combination has text content, then determine to put the node into the third node combination.

[0209] When the component name of the node in the second node combination is not android.view.View, the terminal device can obtain the text content of the node based on node.getText; if the terminal device determines that the node.getText under the text attribute of the node of the second node combination has text content, the terminal device can retain the node and obtain the third node combination.

[0210] Alternatively, S1004: if it is determined that the component name of the node in the second node combination is a preset name, then the first member variable under the text attribute of the node in the second node combination is retrieved.

[0211] android.view.View can be the base class of components and can be used to depict block-like views. For a node named android.view.View, the text content of the node can be obtained using node.getText or node.getContentDescription.

[0212] If the terminal device determines that the component name of the node in the second node combination is android.view.View, the node.getText under the text attribute of the node in the second node combination is called.

[0213] S1005: If it is determined that the first member variable under the text attribute of the node in the second node combination does not have text content, then call the second member variable under the text attribute of the node in the second node combination.

[0214] If the terminal device determines that node.getText under the text attribute of the node in the second node combination does not have text content, node.getContentDescription under the text attribute of the node in the second node combination is called.

[0215] S1006: If it is determined that the second member variable under the text attribute of the node in the second node combination has text content, then determine to put the node into the third node combination.

[0216] If the terminal device determines that the node in the second node combination has text content in the node.getContentDescription under the text attribute, the node is retained to obtain a third node combination.

[0217] It is understood that in this embodiment of the present application, when the component name of the node in the second node group is android.view.View, the terminal device may preferentially access the first member variable under the text attribute of the node in the second node group; the terminal device may also preferentially access the second member variable under the text attribute of the node in the second node group. This embodiment of the present application does not impose any restrictions on this.

[0218] After the terminal device executes steps S1001-S1006, it can obtain nodes with text content to obtain a third node combination. Subsequently, the terminal device can execute step S1007.

[0219] S1007: If it is determined that the text content of the node in the third node combination is invisible control text, remove the node from the third node combination.

[0220] The nodes in the third node combination have text content, wherein the text content may include invisible control text. Invisible control text can be understood as the node carrying text content, but when the terminal device renders and displays the node, the text content cannot be displayed on the touch screen. Exemplarily, the invisible control text can be an invisible control symbol, and the invisible control symbol can be Cc, Cf, Co, Cn, etc. To enhance the user's listening experience, the terminal device can remove the invisible control text from the text content to achieve the effect of the terminal device not reading the invisible control text aloud.

[0221] In some embodiments, the text content of the nodes in the third node combination may all be invisible control text. After the terminal device removes the invisible control text from the text content, the node no longer has text content. In this case, the terminal device can remove the node from the third node combination and update the third node combination without the node to a new third node combination.

[0222] It is understood that in the embodiment of the present application, the terminal device may perform filtering processing in the order of S1001-S1007 to obtain the third node combination. The terminal device may also first perform S1007 to obtain nodes that do not include invisible control text, and then the terminal device performs S1002-S1006 to obtain the third node combination. The embodiment of the present application is not limited to this.

[0223] The terminal device-based automatic reading method provided in an embodiment of the present application removes nodes whose information content is not textual content based on the text attributes of the nodes in the second node combination, thereby obtaining a third node combination. In this way, the terminal device can filter the nodes based on the text attributes and retain nodes with valid textual content to obtain the third node combination, so that the terminal device can subsequently perform further filtering on the third node combination to obtain the node to be processed.

[0224] Step S904 is described below in conjunction with steps S1101-S1104. Exemplarily, in step S904, based on the coordinate attributes of the nodes in the third node combination, nodes whose positions do not belong to the reading area are removed to obtain a fourth node combination, including:

[0225] S1101. The terminal device obtains the coordinate attributes of the nodes in the third node combination.

[0226] The coordinate attribute is used to represent the position coordinates of the nodes in the third node combination in the current interface. The coordinate attribute includes one or more of the following: the left edge of the node, the right edge of the node, the width of the node, the height of the node, etc.

[0227] It is understandable that the information content that the user can intuitively see on the touch screen can be a rectangular area with an effective area, that is, the node can represent the information content of a rectangular area. For example, taking the node in the third node combination as TextView: Weather [123, 123] [400, 400] as an example, [123, 123] and [400, 400] can be represented as the vertex at the upper left corner and the vertex at the lower right corner of the rectangular area of ​​the node, respectively. The terminal device can obtain the above coordinate attributes based on the member variables under the coordinate attribute.

[0228] The reading area may correspond to a target window, and the terminal device may read aloud text content corresponding to the node belonging to the reading area. The terminal device may determine whether the node belongs to the reading area based on steps S1102-S1104.

[0229] S1102. If it is determined that the left edge of the node in the third node combination is larger than the right edge of the screen of the terminal device, or if it is determined that the right edge of the node in the third node combination is larger than the left edge of the screen of the terminal device, it is determined that the position of the node does not belong to the reading area, and the node is removed from the third node combination.

[0230] It is understandable that the current interface of the terminal device may be an interface whose size exceeds the size of the touch screen, such as Figure 10 Interface 1001 in the illustrated interface can be either a finite or infinite canvas. Interface 1001 displays interface 1002, which is the same size as the screen, nodes 1003, 1004, 1005, 1006, and 1007. When the terminal device activates full-text reading mode, the reading area can be interface 1002.

[0231] In some embodiments, the terminal device's reading area may include nodes 1004, 1005, and 1006. If a portion of the rectangular area of ​​node 1004 is located within interface 1002, the terminal device may determine that node 1004 belongs to the reading area. When the rectangular area of ​​a node does not intersect with the reading area, the terminal device may determine that the node does not belong to the reading area, such as nodes 1003 and 1007. In this way, when partial text content is displayed on the terminal device, the terminal device can automatically read the entire text content of the node, thereby improving the user experience.

[0232] Based on this, the terminal device can determine whether a node in the third node combination extends beyond the screen in the horizontal direction according to the positional relationship between the left and right edges of the node in the third node combination and the screen. Exemplarily, if the terminal device determines that the left edge of the node in the third node combination is greater than the right edge of the terminal device's screen, it determines that the position of the node does not belong to the reading area and removes the node from the third node combination. For example, the node may correspond to Figure 10 node 1007 in the interface shown. Or, if the terminal device determines that the right edge of the node in the third node combination is greater than the left edge of the terminal device's screen, it determines that the position of the node does not belong to the reading area and removes the node from the third node combination. For example, the node may correspond to Figure 10 node 1003 in the interface shown.

[0233] In a possible implementation, the terminal device can call the member variable node.left to obtain the left edge of the node in the third node combination, and call the member variable node.right to obtain the right edge of the node in the third node combination; and the terminal device can call the member variable window.left to obtain the left edge of the terminal device's screen, and call the member variable window.right to obtain the right edge of the terminal device's screen. When node.left > window.right or node.right < window.left, the terminal device determines that the position of the node does not belong to the reading area and removes the node from the third node combination.

[0234] S1103. If it is determined that the width of the node in the third node combination is zero, or if it is determined that the height of the node in the third node combination is zero, it is determined that the position of the node does not belong to the reading area and the node is removed from the third node combination to obtain a fourth node combination.

[0235] In a possible implementation, the terminal device can obtain the width width and height height of the node in the third node combination. If width = 0 and / or height = 0, the terminal device determines that the position of the node does not belong to the reading area and removes the node from the third node combination to obtain a fourth node combination. The node may correspond to Figure 10 node 1005 in the interface shown.

[0236] It can be understood that the terminal device can execute steps S1101 - S110 sequentially; the terminal device can also execute steps S1101, S1103, and S1102 sequentially. The embodiments of this application do not limit this.

[0237] The above embodiment is a process for the terminal device to obtain the fourth node combination based on the coordinate attributes in the full-text reading mode; the following describes the process for the terminal device to obtain the fourth node combination based on the coordinate attributes using the point-to-point reading mode as an example.

[0238] S1104: If it is determined that the node in the third node combination is not located in the selected area, determine that the position of the node does not belong to the reading area, and remove the node from the third node combination to obtain a fourth node combination.

[0239] For example, in the point-to-point reading mode, the target window is the selected area specified by the user, such as Figure 10 1008 in the selected area. It is understandable that if the terminal device determines that a node in the third node combination is not located in the selected area, then the location of the node is determined not to belong to the reading area; for example, if nodes 1003, 1004, 1005, and 1007 are not located in the selected area 1008, the terminal device may remove these nodes to obtain a fourth node combination, and the node of the fourth node combination may be node 1006.

[0240] The terminal device-based automatic reading method provided in the embodiments of the present application removes nodes whose locations do not fall within the reading area based on the coordinate attributes of the nodes in the third node group, thereby obtaining a fourth node group. In this way, the terminal device can filter the nodes with valid rectangular areas based on the coordinate attributes to obtain the fourth node group, so that the terminal device can subsequently perform further filtering on the fourth node group to obtain the nodes to be processed.

[0241] Step S905 is described below in conjunction with steps S1201-S1204. Exemplarily, S905, based on the visibility attributes of the nodes in the fourth node combination, removes invisible nodes to obtain nodes to be processed, including:

[0242] S1201: The terminal device retrieves the third member variable under the visible attribute of the node in the fourth node combination.

[0243] The third member variable under the visible attribute may be node.isVisibleToUser. The terminal device may call node.isVisibleToUser under the visible attribute of the node in the fourth node combination.

[0244] It is understandable that the current interface of a terminal device may include multiple windows, among which there may be obstructions between windows. Visible nodes can be understood as if there are no other windows above the window, and the node attributes in the window are visible. Conversely, the node attributes in the window are invisible. For example: when a terminal device displays text content, there may be a video or image pop-up window on the screen. In this case, the video or image may cover part of the text content. The covered part of the text content may be text with a visible attribute of invisible.

[0245] S1202: If it is determined that the return value of the third member variable under the visible attribute of the node in the fourth node combination indicates visible, then the node is determined to be a node to be processed.

[0246] The third member variable may represent the visibility of the node based on the return value. If the terminal device determines that the return value of node.isVisibleToUser under the visible attribute of the node in the fourth node combination represents visible, then the node is a visible node, and the terminal device determines that the node is a node to be processed.

[0247] However, in some cases, when the return value of the third member variable under the visible attribute of the node in the fourth node combination indicates that it is invisible, the text content of the node may still be displayed on the touch screen of the terminal device. This situation is explained below.

[0248] In a possible implementation, the nodes of the web page text content may include visible nodes and invisible nodes. For example, the web page text content may include: Figure 11 The interface shown in a in FIG. 1 includes nodes 901 and 902. Node 901 may be a visible node; node 902 may be an invisible node, obscured by the function key window on the webpage. To enhance the user experience, the terminal device may continue reading the text content of node 902 after completing reading the text content of node 901. This simplifies the user's downward swipe operation and allows the user to directly listen to text content not displayed on the touch screen, thereby enhancing the user experience.

[0249] Exemplarily, in S1203 , if it is determined that the return value of the third member variable under the visible attribute of the node in the fourth node combination is represented as invisible, then the component category and bottom margin information of the root node of the node are obtained.

[0250] The terminal device can display Figure 11In the interface shown in a, the terminal device can call node.isVisibleToUser to obtain the visible properties of node 901 and node 902 respectively, wherein the return value of node.isVisibleToUser of node 901 indicates visible, and the return value of node.isVisibleToUser of node 902 indicates invisible. For the invisible node 902, the terminal device can obtain the component category of the root node of the node and the bottom margin information of the node. Among them, the terminal device can obtain the connection relationship of the node according to the node attribute of the node, and trace the root node of the node; and the terminal device can call the member variable node.bottom to obtain the bottom margin information of the node.

[0251] S1204: If it is determined that the component category of the root node of the node is page view, and the bottom margin information of the node is in the invisible area of ​​the screen of the terminal device, the node is corrected to be visible, and the node is determined to be a node to be processed.

[0252] The terminal device obtains the component category of the root node of the node. When the component category is the page view android.webkit.WebView, and the bottom margin information obtained by node.bottom of the node is in the invisible area of ​​the terminal device screen, the terminal device can set the visible attribute of the node to visible and determine that the node is the node to be processed. The invisible area can be understood as Figure 11 The function key area at the bottom of the screen of the terminal device in the interface shown in a.

[0253] In another possible implementation, the terminal device can provide a full-screen gesture. When the terminal device receives a sliding operation of the system window by the user, the terminal device can display other interfaces. For example, when the user slides inward on the left or right side of the screen, the terminal device responds to the operation and returns to the previous interface. For example, the terminal device can display Figure 11 The interface shown in b in Figure 1 includes the node's text content and a system window, which can be a gesture navigation window 903. Gesture navigation window 903 floats above the text content. The return value of node.isVisibleToUser on the terminal device indicates that the node in the area covered by gesture navigation window 903 is invisible, but in fact, the transparent gesture navigation window 903 does not obscure the text content. In this case, the terminal device can set the node to be visible.

[0254] Exemplarily, in S1205 , if it is determined that the return value of the third member variable under the visible attribute of the node in the fourth node combination indicates invisible, a preset interface is called to obtain occlusion information of the node.

[0255] The preset interface may be used to represent the occlusion information of a node. For example, the preset interface may be tap_system, and the occlusion information may be a system window.

[0256] S1206: If the occlusion information of the node indicates that the node is covered by the system window and the height and width of the node are not scaled, correct the node to be visible and determine the node as a node to be processed.

[0257] If the occlusion information of the node obtained by the preset interface indicates that the node is covered by the system window and the height and width of the node are not scaled, the terminal device sets the node to be visible to obtain the node to be processed.

[0258] The terminal device-based automatic reading method provided in the embodiments of the present application removes invisible nodes based on the visibility attributes of the nodes in the fourth node combination to obtain the nodes to be processed. In this way, the terminal device can filter based on the visibility attributes to retain nodes that are not obscured by other windows to obtain the nodes to be processed, so that the terminal device can read the text content of the nodes to be processed.

[0259] The automatic reading method based on the terminal device provided in the embodiment of the present application also provides the function of selecting the reading mode and playing the previous / next paragraph of text. In order to facilitate the user to use the above functions, in the embodiment of the present application, the terminal device can read the visible text in sections. For example:

[0260] The terminal device can segment the text content of the node to be processed to obtain multiple paragraphs to be processed; the terminal device performs voice conversion processing on the paragraphs to be processed to obtain and read aloud the voice corresponding to the paragraphs to be processed.

[0261] In one possible implementation, the terminal device may segment the text content according to the nodes. It is understood that the target window may include multiple text contents, each of which may correspond to a node. When the terminal device detects a node, the terminal device may set the text content of the node as a paragraph to be processed.

[0262] In another possible implementation, the terminal device may segment the text content based on characters to obtain multiple paragraphs to be processed. For example, the terminal device may segment the text content based on special symbols. For example, the special symbols may be line breaks, paragraph marks, etc. For another example, the special symbols may be punctuation marks such as ".", "?", and "...".

[0263] In another possible implementation, there may be no special symbols in the text content, and the terminal device may also segment the text content according to a preset word length.

[0264] In the embodiment of the present application, the terminal device may also segment the text content according to other segmentation rules. The embodiment of the present application does not limit the segmentation method of the text content.

[0265] For example, Figure 12 The internal interaction flow chart of the automatic reading method based on the terminal device provided in the embodiment of the present application is as follows: Figure 12 As shown:

[0266] The application layer of the terminal device may include on-demand reading and TTS; the framework layer may include an accessibility service module and an on-demand reading service module. The on-demand reading service module may include a node extractor, a node filter, and a text segmenter. The node filter includes a leaf node filter, a component filter, a text filter, a coordinate filter, and a visibility filter. The node filter can be understood as a manager that filters based on the corresponding node data.

[0267] In one possible implementation, the "select-to-read" function of the terminal device is in a full-text reading mode.

[0268] The terminal device starts the "Select to Read" function of the application layer. The terminal device can pull the current interface through the accessibility service of the framework layer and call the node extractor of the Select to Read service. The node extractor can obtain the node data of the node of the target window in the current interface of the terminal device; the target window can correspond to Figure 6 The node extractor reports the node data of the node to the node filter, and the node filter filters the nodes according to the node data of the node.

[0269] Exemplarily, the leaf node filter obtains nodes in the target window, removes non-leaf nodes from the obtained nodes based on the node data of the obtained nodes, and obtains a first node combination. The leaf node filter reports the nodes of the first node combination to the component filter.

[0270] The component filter removes nodes whose component names do not meet a preset condition based on the component attributes of the nodes in the first node combination, thereby obtaining a second node combination, wherein the component names of the nodes in the second node combination are the first name or the second name. The component filter reports the nodes in the second node combination to the text filter.

[0271] The text filter removes nodes whose information content is not textual content based on the text attributes of the nodes in the second node combination, thereby obtaining a third node combination. The terminal device may execute steps S1001-1007 to obtain the third node combination. The text filter reports the nodes in the third node combination to the coordinate filter.

[0272] The coordinate filter removes nodes that are not located within the reading area based on the coordinate attributes of the nodes in the third node combination, thereby obtaining a fourth node combination. The terminal device may execute steps S1101-S1103 to obtain the fourth node combination. The coordinate filter reports the nodes in the fourth node combination to the visibility filter.

[0273] The visibility filter removes invisible nodes according to the visibility attributes of the nodes in the fourth node combination to obtain nodes to be processed, wherein the terminal device can execute steps S1201-S1204 to obtain the nodes to be processed. The visibility filter reports the nodes to be processed to the text segmenter.

[0274] The text segmenter segments the text content of the node to be processed to obtain multiple paragraphs to be processed. After the TTS application obtains the paragraphs to be processed, it can perform speech conversion on the paragraphs to obtain and read aloud the speech corresponding to the paragraphs to be processed.

[0275] It can be understood that in some embodiments, before the node extractor obtains the node data of the node of the target window in the current interface of the terminal device, it may include: in response to the user triggering the first button in the first page, displaying the second page, and generating a trigger indication; wherein the first button is a button for indicating full-text reading, and the target window indicated by the trigger indication is the current interface displayed by the terminal device; the second page includes a second button, and the second button is a button for indicating pause.

[0276] The first page can be Figure 6 In the interface shown in a, the first button is the play button 602 in the unopened state, and the second page is Figure 6 The interface shown in b; the target window can be Figure 6 In the interface shown in b, the second button is the play button 604 in the turned-on state. The first page can also be Figure 6 In the interface shown in c, the first button is the floating ball 605 in the unopened state, and the second page is Figure 6 The target window can be Figure 6 In the interface shown in d, the second button is a floating ball 606 in the turned-on state.

[0277] When the terminal device receives the user's Figure 6 When the play button 602 is triggered in the interface shown in a, the node extractor can obtain Figure 6 Alternatively, when the terminal device receives the user's Figure 6 When the trigger operation for the floating ball 605 in the interface shown in c is executed, the node extractor can obtain Figure 6 The node in the interface shown in c.

[0278] The terminal device then performs filtering and segmentation processing on the node until the terminal device obtains the paragraph to be processed, and the TTS of the terminal device reads the text of the paragraph to be processed.

[0279] In other embodiments, the terminal device may receive a trigger operation from the user to enable the "select to read aloud" function (for example, Figure 4 In the interface shown in b in FIG, for the triggering operation of the "select and read aloud" function 401, the node extractor can obtain the node data of the node of the target window in the current interface of the terminal device (the current interface can be Figure 4 The terminal device performs filtering and segmentation processing on the node to obtain the paragraph to be processed. When the terminal device receives a trigger operation for playing the text content of the node to be processed (in Figure 6 In the interface shown in a, for the triggering operation of the play button 602, the terminal device reads aloud the text of the paragraph to be processed.

[0280] In another possible implementation, the "on-demand reading" function of the terminal device is in a click-to-read mode.

[0281] The terminal device starts the "Select to Read" function of the application layer. The terminal device can pull the current interface through the accessibility service of the framework layer and call the node extractor of the Select to Read service. The node extractor can obtain the node data of the node of the target window in the current interface of the terminal device; the target window can correspond to Figure 7 The selected area 704 in the interface shown in c in FIG. The node extractor reports the node data of the node to the node filter, and the node filter filters the node according to the node data of the node.

[0282] Exemplarily, the leaf node filter obtains nodes in the target window, removes non-leaf nodes from the obtained nodes based on the node data of the obtained nodes, and obtains a first node combination. The leaf node filter reports the nodes of the first node combination to the component filter.

[0283] The component filter removes nodes whose component names do not meet a preset condition based on the component attributes of the nodes in the first node combination, thereby obtaining a second node combination, wherein the component names of the nodes in the second node combination are either the first name or the second name. The component filter reports the nodes in the second node combination to the text filter.

[0284] The text filter removes nodes whose information content is not textual content based on the text attributes of the nodes in the second node combination, thereby obtaining a third node combination. The terminal device may execute steps S1001-1007 to obtain the third node combination. The text filter reports the nodes in the third node combination to the coordinate filter.

[0285] The coordinate filter removes nodes not located within the reading area based on the coordinate attributes of the nodes in the third node combination, thereby obtaining a fourth node combination. The terminal device may execute steps S1101, S1104, and S1103 to obtain the fourth node combination. The coordinate filter reports the nodes in the fourth node combination to the visibility filter.

[0286] The visibility filter removes invisible nodes based on the visibility attributes of the nodes in the fourth node combination to obtain nodes to be processed. The terminal device may execute steps S1201, S1202, S1205, and S1206 to obtain the nodes to be processed. The visibility filter reports the nodes to be processed to the text segmenter.

[0287] The text segmenter segments the text content of the node to be processed to obtain multiple paragraphs to be processed. After the TTS application obtains the paragraphs to be processed, it can perform speech conversion on the paragraphs to obtain and read aloud the speech corresponding to the paragraphs to be processed.

[0288] It is understandable that in some embodiments, before the node extractor obtains the node data of the node of the target window in the current interface of the terminal device, it may include:

[0289] The first page also includes a third button for instructing point reading. The second page also includes a fourth button for instructing point reading. In response to a user triggering the third button on the first page, or in response to a user triggering the fourth button on the second page, the third page is displayed. The third page includes a preset icon indicating that the terminal device is in point reading mode. In response to a user selecting the current interface of the terminal device, a selected area is determined and a trigger indication is generated. The target window indicated by the trigger indication is the selected area.

[0290] The first page can be Figure 7 The second page is the interface shown in a. Figure 6 The interface shown in b; the third button is Figure 7 The fourth button is the selection mode button 701 in the interface shown in a; Figure 6 The third page can be Figure 7 In the interface shown in b, the default icon is Figure 7 The suspended ball 702 in the interface shown in b.

[0291] exist Figure 7 In the interface shown in a, when the terminal device receives a trigger operation for clicking the mode button 701, the terminal device enters Figure 7 The interface shown in b, Figure 7The interface shown in b in FIG includes a floating ball 702. When the terminal device receives a Figure 7 When the operation is selected in the interface shown in b, the terminal device enters Figure 7 In the interface shown in c, Figure 7 The interface shown in c includes a selected area 704, and the terminal device generates a trigger indication. The terminal device node extractor can obtain Figure 7 In the interface shown in c, the node in area 704 is selected.

[0292] The terminal device performs filtering and segmentation processing until the terminal device obtains a paragraph to be processed, and the TTS of the terminal device reads the text of the paragraph to be processed.

[0293] Optionally, a preset option is set to a selected state on the fourth page; wherein the fourth page is used to display whether the terminal device supports automatic reading aloud, and the selected state indicates that automatic reading aloud is turned on.

[0294] Among them, the fourth page may correspond to Figure 3 In the interface shown in b, the preset option may be a switch button 302 for reading the content; or the fourth page may correspond to Figure 4 In the interface shown in b, the preset option may be the “Read on Demand” icon 401.

[0295] The automatic reading method based on the terminal device in the embodiment of the present application has been described above. The device for executing the above method provided by the embodiment of the present application is described below. Figure 13 As shown, Figure 13 A schematic structural diagram of an automatic reading device based on a terminal device provided in an embodiment of the present application. The automatic reading device based on a terminal device can be the terminal device in the embodiment of the present application, or it can be a chip or chip system in the terminal device.

[0296] like Figure 13 As shown, an automatic reading aloud device 1300 based on a terminal device can be used in a communication device, circuit, hardware component, or chip. The automatic reading aloud device based on a terminal device includes a display unit 1301 and a processing unit 1302. The display unit 1301 is used to support the display step performed by the automatic reading aloud device 1300 based on the terminal device; the processing unit 1302 is used to support the information processing step performed by the automatic reading aloud device 1300 based on the terminal device.

[0297] In a possible implementation, the terminal-based automatic reading device 1300 may also include a communication unit 1303. Specifically, the communication unit is used to support the terminal-based automatic reading device 1300 in executing the steps of sending and receiving data. The communication unit 1303 may be an input or output interface, pin, or circuit.

[0298] In a possible embodiment, the terminal-based automatic reading device may further include a storage unit 1304. The processing unit 1302 and the storage unit 1304 are connected via a circuit. The storage unit 1304 may include one or more memories, which may be devices in one or more devices or circuits used to store programs or data. The storage unit 1304 may exist independently and be connected to the processing unit 1302 of the terminal-based automatic reading device via a communication line. Alternatively, the storage unit 1304 may be integrated with the processing unit 1302.

[0299] The storage unit 1304 can store computer-executable instructions for the method in the terminal device, so that the processing unit 1302 executes the method in the above embodiment. The storage unit 1304 can be a register, a cache, or a RAM, etc. The storage unit 1304 can be integrated with the processing unit 1302. The storage unit 1304 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions. The storage unit 1304 can be independent of the processing unit 1302.

[0300] An embodiment of the present application provides a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0301] The terminal device includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.

[0302] The present application embodiment provides a terminal device, the structure of which is shown in FIG. Figure 3 The memory of the terminal device can be used to store at least one program instruction, and the processor is used to execute at least one program instruction to implement the technical solution of the above method embodiment. Its implementation principle and technical effects are similar to those of the above method related embodiments and will not be repeated here.

[0303] The present embodiment provides a chip. The chip includes a processor configured to invoke a computer program stored in a memory to execute the technical solution of the above embodiment. The implementation principles and technical effects are similar to those of the above-mentioned related embodiments and will not be further described here.

[0304] The present application provides a computer program product that, when executed on an electronic device, enables the terminal device to execute the technical solution in the above embodiment. The implementation principle and technical effects are similar to those of the above-mentioned related embodiments and will not be described in detail here.

[0305] An embodiment of the present application provides a computer-readable storage medium having program instructions stored thereon. When the program instructions are executed by a terminal device, the terminal device executes the technical solution of the above-mentioned embodiment. Its implementation principle and technical effects are similar to those of the above-mentioned related embodiments and will not be repeated here. The method described in the above-mentioned embodiment can be implemented in whole or in part by software, hardware, firmware or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0306] Computer-readable media may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium designed to carry or store the desired program code in the form of instructions or data structures and accessible by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technologies such as infrared, radio and microwave are used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and optical disk, as used herein, include optical disk, laser disk, optical disk, digital versatile disk (DVD), floppy disk and Blu-ray disk, where disks generally reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0307] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0308] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An automatic reading method based on a terminal device, characterized in that: The method is applied to a terminal device, and the method includes: In response to a trigger indication, node data of a node of a target window in a current interface of the terminal device is obtained; wherein the trigger indication is used to instruct reading aloud information on the target window in the current interface displayed by the terminal device; the node represents a piece of data in the target window; wherein the node data includes one or more of the following: node attributes, component attributes, text attributes, coordinate attributes, and visibility attributes; the node attribute represents whether the node has child nodes, the component attribute represents the component name of the component to which the node belongs, the text attribute is used to indicate information content in the node, the coordinate attribute is used to indicate location information of the node, and the visibility attribute is used to indicate whether the node is visible; According to the node data of the obtained nodes, the obtained nodes are filtered to obtain nodes to be processed; wherein the node data of the nodes to be processed include text content; the filtering process is as follows: for the obtained nodes, the following steps are performed in sequence: removing non-leaf nodes in the nodes according to the node attributes of the nodes, removing nodes whose component names do not meet preset conditions according to the component attributes of the nodes, removing nodes whose information content is not text content according to the text attributes of the nodes, removing nodes whose positions do not belong to the reading area according to the coordinate attributes of the nodes, and removing invisible nodes according to the visibility attributes of the nodes; Read aloud the text content of the node to be processed.

2. The method according to claim 1, characterized in that According to the node data of the obtained nodes, the obtained nodes are filtered to obtain the nodes to be processed, including: According to the node attributes of the obtained nodes, non-leaf nodes in the obtained nodes are removed to obtain a first node combination; wherein the first node combination includes at least one node, and all nodes in the first node combination are leaf nodes; According to the component attributes of the nodes in the first node combination, nodes whose component names do not meet the preset conditions are removed to obtain a second node combination; wherein the second node combination includes at least one node; According to the text attributes of the nodes in the second node combination, nodes whose information content is not text content are removed to obtain a third node combination; wherein the third node combination includes at least one node; According to the coordinate attributes of the nodes in the third node combination, nodes whose positions do not belong to the reading area are removed to obtain a fourth node combination; wherein the fourth node combination includes at least one node; Invisible nodes are removed according to the visibility attributes of the nodes in the fourth node combination to obtain nodes to be processed.

3. The method according to claim 2, characterized in that According to the node attributes of the obtained nodes, non-leaf nodes in the obtained nodes are removed to obtain a first node combination, including: If the acquired node attribute indicates that the node has child nodes, the node is determined to be a non-leaf node, and the node is removed to obtain the first node combination.

4. The method according to claim 3, characterized in that Before obtaining the first node combination, the method further includes: Remove the node whose component name is Page View.

5. The method according to claim 2, characterized in that The component names represented by the preset conditions include: a first name, and / or a second name; The first name is a component name related to the text, and the second name is a component name with a non-Android prefix.

6. The method according to claim 2, characterized in that According to the text attributes of the nodes in the second node combination, nodes whose information content is not text content are removed to obtain a third node combination, including: If it is determined that the component name of the node in the second node combination is not a preset name, calling the first member variable under the text attribute of the node in the second node combination; If it is determined that the first member variable under the text attribute of the node in the second node combination has text content, then the node is determined to be placed in the third node combination.

7. The method according to claim 6, characterized in that The method further comprises: If it is determined that the component name of the node in the second node combination is a preset name, then calling the first member variable under the text attribute of the node in the second node combination; If it is determined that the first member variable under the text attribute of the node in the second node combination does not have text content, then calling the second member variable under the text attribute of the node in the second node combination; If it is determined that the second member variable under the text attribute of the node in the second node combination has text content, then the node is determined to be placed in the third node combination.

8. The method according to claim 7, characterized in that The method further comprises: If it is determined that the text content of a node in the third node combination is invisible control text, the node is removed from the third node combination.

9. The method according to claim 2, characterized in that The coordinate attributes include one or more of the following: the left edge of the node, the right edge of the node, the width of the node, and the height of the node.

10. The method according to claim 9, characterized in that According to the coordinate attributes of the nodes in the third node combination, nodes whose positions do not belong to the reading area are removed to obtain a fourth node combination, including: If it is determined that the left edge of a node in the third node combination is larger than the right edge of the screen of the terminal device, or if it is determined that the right edge of a node in the third node combination is larger than the left edge of the screen of the terminal device, then determining that the position of the node does not belong to the reading area, and removing the node from the third node combination; If it is determined that the width of the node in the third node combination is zero, or if it is determined that the height of the node in the third node combination is zero, it is determined that the position of the node does not belong to the reading area, and the node is removed from the third node combination to obtain the fourth node combination.

11. The method according to claim 10, characterized in that When the target window is a selected area specified by the user, the method further includes: If it is determined that the node in the third node combination is not located in the selected area, then the position of the node is determined not to belong to the reading area, and the node is removed from the third node combination to obtain the fourth node combination.

12. The method according to claim 2, characterized in that According to the visibility attributes of the nodes in the fourth node combination, invisible nodes are removed to obtain nodes to be processed, including: Retrieve the third member variable under the visible attribute of the node in the fourth node combination; If it is determined that the return value of the third member variable under the visible attribute of the node in the fourth node combination indicates that it is visible, then the node is determined to be the node to be processed.

13. The method according to claim 12, characterized in that The method further comprises: If it is determined that the return value of the third member variable under the visible attribute of the node in the fourth node combination indicates that it is invisible, then obtaining the component category of the root node of the node and the bottom margin information of the node; If it is determined that the component category of the root node of the node is page view, and the bottom margin information of the node is in the invisible area of ​​the screen of the terminal device, the node is corrected to be visible, and the node is determined to be the node to be processed.

14. The method according to claim 12 or 13, characterized in that The method further comprises: If it is determined that the return value of the third member variable under the visible attribute of the node in the fourth node combination indicates that it is invisible, calling a preset interface to obtain the occlusion information of the node; If the occlusion information of the node indicates that the node is covered by the system window and the height and width of the node are not scaled, the node is corrected to be visible and is determined to be the node to be processed.

15. The method according to any one of claims 1 to 13, characterized in that Read aloud the text content of the node to be processed, including: Segmenting the text content in the node to be processed to obtain multiple paragraphs to be processed; Perform voice conversion processing on the paragraph to be processed to obtain and read aloud the voice corresponding to the paragraph to be processed.

16. The method according to any one of claims 1 to 13, characterized in that The method further comprises: In response to the user triggering the first button in the first page, the second page is displayed and the trigger indication is generated; wherein, the first button is a button for indicating full-text reading, and the target window indicated by the trigger indication is the current interface displayed by the terminal device; the second page includes a second button, and the second button is a button for indicating pause.

17. The method according to claim 16, characterized in that The first page further includes a third button, which is used to instruct point reading; the second page further includes a fourth button, which is used to instruct point reading; and the method further includes: In response to a user triggering the third button on the first page, or in response to a user triggering the fourth button on the second page, displaying a third page; wherein the third page includes a preset icon, and the preset icon indicates that the terminal device is in the point-to-read mode; In response to a user selecting operation on the current interface of the terminal device, a selected area is determined and the trigger indication is generated; wherein the target window indicated by the trigger indication is the selected area.

18. The method according to claim 16, characterized in that The method further comprises: The preset option is set to a selected state on the fourth page; wherein the fourth page is used to display whether the terminal device supports automatic reading aloud, and the selected state indicates that automatic reading aloud is turned on.

19. A terminal device, characterized in that: include: processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the terminal device performs the method according to any one of claims 1 to 18.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 18 is implemented.

21. A computer program product, characterized in that The method comprises a computer program which, when being executed, causes a computer to execute the method according to any one of claims 1 to 18.

Images