Assistive reading methods, electronic devices and computer-readable storage media

Abstract

This application relates to the field of communication and computer technology, providing an assisted reading method, an electronic device, and a computer-readable storage medium. The method includes: an electronic device using a camera to capture an image of text, obtaining a first image, and displaying the first image on a screen; when the first image only includes a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the movement of the electronic device and / or the movement of the text; the electronic device using a camera to capture an image of the text, obtaining a second image, and displaying the second image on a screen; when the second image includes the entire text, the electronic device reads aloud the entire text. This application avoids assisted reading failures, improves assisted reading efficiency, and enhances the user experience.

Description

Technical Field

[0001] This application relates to the field of communication and computer technology, and in particular to an auxiliary reading method, electronic device, and computer-readable storage medium. Background Technology

[0002] In daily life, visually impaired people often find it difficult to read the contents of printed books, contracts, medicine instructions, and other texts due to their visual impairments. Therefore, how to assist visually impaired people in accessing textual content has become a technical problem worthy of research.

[0003] Currently, the main methods to assist visually impaired people in obtaining text content are: using the camera of an electronic device to capture images of the text content of paper books, contracts, medicine instructions, etc., and then using the electronic device to output the corresponding text content from the images via voice.

[0004] However, in the process of assisting visually impaired people to obtain text content, it is easy to only collect the corresponding parts of the text in paper books, contracts, drug instructions, etc., resulting in failure of assisted reading.

[0005] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0006] This application provides an assisted reading method, an electronic device, and a computer-readable storage medium, with the aim of avoiding assisted reading failures, improving assisted reading efficiency, and enhancing user experience.

[0007] Firstly, this application provides an assisted reading method, which includes: an electronic device using a camera to capture an image of text, obtaining a first image, and displaying the first image on a screen; when the first image only includes a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device and / or the text to be moved; the electronic device using a camera to capture an image of the text, obtaining a second image, and displaying the second image on a screen; when the second image includes all the text, the electronic device reads aloud the entire text. Thus, when assisted reading is needed but the electronic device has not fully captured the entire text, the electronic device issues a voice prompt, instructing the electronic device and / or the text to be moved. After moving the electronic device and / or the text, the electronic device continues to capture an image of the text using a camera, obtaining a second image, and displays the second image on a screen; thereby achieving the goal that when the second image includes all the text, the electronic device reads aloud the entire text, i.e., through the voice prompt, the electronic device can read aloud the entire text, thus avoiding the technical problem of assisted reading failure and low assisted reading efficiency.

[0008] In one possible implementation, the step of the electronic device using a camera to capture an image of the text includes:

[0009] When the electronic device is in auxiliary mode, it uses a camera to capture images of text.

[0010] In another possible implementation, the electronic device is determined to be in auxiliary mode if the trigger condition for auxiliary mode is detected.

[0011] In another possible implementation, the step of the electronic device using a camera to capture an image of text when the electronic device is in auxiliary mode includes:

[0012] When the electronic device is in auxiliary mode, if a trigger condition is detected to use the camera to capture images of the text, the electronic device will use the camera to capture images of the text.

[0013] The triggering conditions for using a camera to capture images of text are different from the triggering conditions for camera recording.

[0014] In another possible implementation, the camera includes at least a first camera and a second camera, wherein the field of view of the first camera is larger than that of the second camera;

[0015] The steps of the electronic device using a camera to capture images of text include:

[0016] The electronic device uses a first camera and / or a second camera to capture images of the text.

[0017] In another possible implementation, the step of the electronic device acquiring images of the text using a first camera and / or a second camera includes:

[0018] Electronic devices use the first and second cameras simultaneously or sequentially to capture images of the text.

[0019] Before the step of the electronic device issuing a voice prompt when the first image only includes a portion of the text, the following steps are included:

[0020] When the electronic device fails to recognize the content of the text in the image captured by the first camera or the second camera, the electronic device determines whether the first image only includes a portion of the text based on the image captured by the first camera and the image captured by the second camera.

[0021] In another possible implementation, before the step of determining whether the first image contains only a portion of the text content based on the images captured by the first camera and the second camera when the electronic device fails to successfully identify the content of the text in the images captured by the first camera or the second camera, the electronic device includes:

[0022] The electronic device identifies the image captured by the first camera or the second camera. If it is determined that the image captured by the first camera or the second camera includes the text content, then edge detection is performed on the text in the captured image.

[0023] If it is determined that the edges of the text in the captured image are not within the preview stream area of ​​the corresponding camera, the electronic device determines that it has failed to successfully recognize the content of the text in the captured image.

[0024] In another possible implementation, the step of the electronic device determining whether the first image contains only a portion of the text based on the image captured by the first camera and the image captured by the second camera includes:

[0025] The electronic device detects whether the image captured by the first camera includes the text content. When the electronic device detects that the image captured by the first camera includes the text content, it performs edge detection on the text in the image captured by the second camera.

[0026] If the edge of the text in the image captured by the second camera is not within the preview stream area of ​​the second camera, then the electronic device determines whether the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera.

[0027] If the electronic device determines that the edge of the text in the image captured by the first camera is not within the preview stream area of ​​the first camera, then it determines that the first image only includes a portion of the text.

[0028] In another possible implementation, if the edge of the text in the image captured by the second camera is not within the preview stream area of ​​the second camera, then after the step of determining whether the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera, the electronic device includes:

[0029] If the electronic device determines that the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera, it then determines whether the sharpness of the text in the image captured by the second camera is greater than the sharpness threshold.

[0030] If the resolution is not greater than the resolution threshold, the electronic device determines that the first image only includes a portion of the text.

[0031] In another possible implementation, when the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the steps of moving the electronic device and / or moving the text, including:

[0032] When the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to move and / or the text to move, and the voice prompt also instructs the electronic device to move in a first direction and at a first target distance, or the voice prompt also instructs the electronic device to move in a second direction and at a second target distance, wherein the first direction, the first target distance, the second direction, and the second target distance are all associated with a positional relationship, which is the positional relationship between the edge of the text and the edge of the corresponding camera preview stream area.

[0033] In another possible implementation, after the step of the electronic device reading aloud the entire content of the text when the second image includes it, the method further includes:

[0034] The electronic device issues a voice instruction indicating that it is no longer necessary to capture images of the text.

[0035] In another possible implementation, when the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the steps of moving the electronic device and / or moving the text, including:

[0036] When the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to pan, rotate, or turn, or the voice prompt also instructs the electronic device to move closer to or away from the text;

[0037] And / or when the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the text to be translated, rotated, or turned, or the voice prompt also instructs the text to move closer to or away from the electronic device.

[0038] In another possible implementation, the step of the electronic device issuing a voice prompt when the first image includes only a portion of the text, the voice prompt instructing the movement of the electronic device and / or the movement of the text, includes:

[0039] When the first image includes only a portion of the text, and another portion of the text in the first image is obscured, the electronic device issues a voice prompt, which instructs the electronic device to move and / or the text to move, and also instructs the removal of the obstruction that is obscuring the text.

[0040] Secondly, this application provides an auxiliary reading device, including: a first display module, a prompting module, a second display module, and a reading aloud module;

[0041] A first display module in an electronic device is used to capture images of text using a camera, obtain a first image, and display the first image on a screen.

[0042] A prompting module in an electronic device is used to issue a voice prompt when the first image includes only a portion of the text, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved;

[0043] The second display module in the electronic device is used to capture images of the text using a camera, obtain a second image, and display the second image on a screen.

[0044] A text-to-speech module in an electronic device is used to read aloud the entire content of the text when the second image includes the entire content of the text.

[0045] Thirdly, this application provides an electronic device, comprising:

[0046] One or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the assisted reading method as described in any one of the possible implementations of the first aspect or the first / aspect above.

[0047] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the assisted reading method as described in any one of the first aspects or possible implementations thereof.

[0048] Fifthly, this application provides a chip, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the auxiliary reading method described in any one of the first aspects or possible implementations of the first aspect.

[0049] Optionally, as one possible implementation, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform the auxiliary reading method according to any one of the first aspects or possible implementations of the first aspect. Attached Figure Description

[0050] Figure 1 This is an illustrative scene diagram of an existing technology for assisting reading;

[0051] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0052] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0053] Figure 4 A flowchart illustrating an embodiment of the auxiliary reading method provided in this application;

[0054] Figure 5 A module interaction diagram of an auxiliary reading method provided in one embodiment of this application;

[0055] Figure 6 A flowchart illustrating an auxiliary reading method provided in another embodiment of this application;

[0056] Figure 7A flowchart illustrating an auxiliary reading method provided in yet another embodiment of this application;

[0057] Figure 8 A flowchart illustrating an auxiliary reading method provided in another embodiment of this application;

[0058] Figure 9 A schematic diagram of the structure of an auxiliary reading device provided in this application;

[0059] Figure 10 A module interaction diagram of an assisted reading method provided in another embodiment of this application;

[0060] Figure 11 This is a schematic diagram illustrating the target distance between each edge of the text in the electronic device and the corresponding edge of the camera preview stream area in this application. Figure 1 ;

[0061] Figure 12 This is a schematic diagram illustrating how the electronic device in this application moves based on the target distance between each edge of the text and the corresponding edge of the camera preview stream area. Figure 1 ;

[0062] Figure 13 This is a schematic diagram illustrating how the electronic device in this application moves based on the target distance between each edge of the text and the corresponding edge of the camera preview stream area. Figure 2 ;

[0063] Figure 14 This is a schematic diagram illustrating how the electronic device in this application moves based on the target distance between each edge of the text and the corresponding edge of the camera preview stream area. Figure 3 ;

[0064] Figure 15 This is a schematic diagram illustrating how the electronic device in this application moves based on the target distance between each edge of the text and the corresponding edge of the camera preview stream area. Figure 4 ;

[0065] Figure 16 This is a schematic diagram illustrating how the electronic device in this application moves based on the target distance between each edge of the text and the corresponding edge of the camera preview stream area. Figure 5 ;

[0066] Figure 17 This is a schematic diagram illustrating how the electronic device in this application moves based on the target distance between each edge of the text and the corresponding edge of the camera preview stream area. Figure 6 ;

[0067] Figure 18 This is a schematic diagram illustrating how the electronic device in this application moves based on the target distance between each edge of the text and the corresponding edge of the camera preview stream area. Figure 7 ;

[0068] Figure 19 This is a schematic diagram illustrating how the electronic device in this application moves based on the target distance between each edge of the text and the corresponding edge of the camera preview stream area. Figure 9 ;

[0069] Figure 20 This is a schematic diagram illustrating the effect of the electronic device before and after the prompt in this application. Figure 1 ;

[0070] Figure 21 A schematic diagram of a first scene for capturing text images using the first camera of the electronic device in this application;

[0071] Figure 22 A schematic diagram of a second scene for capturing text images using the second camera of the electronic device in this application;

[0072] Figure 23 This is a schematic diagram of a third scene in which the first camera of the electronic device in this application combines with the second camera to capture text images;

[0073] Figure 24 This is a schematic diagram of the first scenario of electronic device-assisted identification of drug instructions in this application;

[0074] Figure 25 This is a schematic diagram of the second scenario of electronic device-assisted recognition of planar non-forward text in this application;

[0075] Figure 26 This is a first schematic diagram of an electronic device acquiring an image of obscured text in this application;

[0076] Figure 27 A flowchart illustrating an auxiliary reading method provided in another embodiment of this application;

[0077] Figure 28 A module interaction diagram of an auxiliary reading method provided in another embodiment of this application;

[0078] Figure 29 This is a schematic diagram of a first scene provided in another embodiment of the present application, showing the second camera combined with the second camera capturing text images;

[0079] Figure 30 This is a schematic diagram of the first interface of the electronic device in this application when entering the assisted reading mode;

[0080] Figure 31 This is a schematic diagram of the second interface of the electronic device in this application when it enters the assisted reading mode. Detailed Implementation

[0081] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different texts, not to limit a specific order.

[0082] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0083] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to an implementation scheme to aid reading is first given:

[0084] Currently, mobile phones, tablets, and other electronic devices have become an indispensable part of the lives of visually impaired people. When visually impaired people use these electronic devices, the devices use cameras to capture images of text, obtaining a first image, which is then displayed on the screen. However, the first image may only include a portion of the text. When this happens, the electronic device issues a voice prompt, instructing the user to move the device and / or the text. The device then uses its camera to capture images of the text again, obtaining a second image, which is also displayed on the screen. When the second image includes the entire text, the electronic device reads the entire text aloud. Alternatively, users can first activate the electronic device's assistive mode specifically designed for visually impaired individuals, or launch relevant assistive applications (APPs) such as camera, screen reader, vision aid, and reading aid APPs when visually impaired individuals use the electronic device. Based on this, the electronic device uses its camera to capture an image of the text, obtaining a first image, and displays the first image on the screen. When the first image only includes a portion of the text, the electronic device issues a voice prompt, instructing the user to move the electronic device and / or move the text. The electronic device then uses its camera to capture an image of the text, obtaining a second image, and displays both images on the screen. When the second image includes the entire text, the electronic device reads aloud the entire text.

[0085] Figure 1 This is an illustrative scenario illustrating how, in existing technologies for assisting reading, the first image captured by the electronic device's camera only includes a portion of the text, causing the assistance to fail. In existing technologies, such as... Figure 1As shown, if an electronic device needs to assist in reading the text content on a book page, the device uses a camera to capture (photograph) the text on the book page and extracts the text content from the captured image based on technologies such as OCR. However, when the user is visually impaired, it cannot be determined whether the entire text content can be obtained (e.g., ...). Figure 1 In the left image, the camera only captures the content in the upper left corner of the book page, and in the right image, it only captures a portion of the drug instructions. This results in the electronic device only being able to obtain a portion of the text and sending that portion to the TTS (Text-to-Speech) module within the device. The TTS module then outputs only a portion of the text, not the entire text, preventing the user from accurately obtaining the information they need.

[0086] To address the problems existing in the aforementioned technical solutions, this application provides an assisted reading method. This method, during the process of assisting visually impaired individuals in reading or acquiring text content, prompts the user to move the electronic device and / or the text until the electronic device acquires and reads the entire text aloud, thereby improving assistance efficiency and enhancing user experience. The provided assisted reading method is applicable to mobile phones, tablets, desktop computers, laptops, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable electronic devices, smartwatches, and other electronic devices. The structure of the electronic device to which this assisted reading method is applicable can be as follows: Figure 2 As shown.

[0087] like Figure 2 As shown, Figure 2The following is an example diagram of the composition of an electronic device provided in this application. The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a pressure sensor 280A, a gyroscope sensor 280B, a barometric pressure sensor 280C, a magnetic sensor 280D, an accelerometer sensor 280E, a distance sensor 280F, a proximity sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, an ambient light sensor 280L, a bone conduction sensor 280M, etc.

[0088] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0089] Processor 210 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0090] The controller can be the nerve center and command center of the electronic device 200. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0091] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0092] In some embodiments, the processor 210 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0093] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 210 may include multiple I2C buses. The processor 210 can couple to the touch sensor 280K, charger, flash, camera 293, etc., through different I2C bus interfaces. For example, the processor 210 can couple to the touch sensor 280K through the I2C interface, enabling the processor 210 and the touch sensor 280K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 200.

[0094] The I2S interface can be used for audio communication. In some embodiments, the processor 210 may include multiple I2S buses. The processor 210 can be coupled to the audio module 270 via the I2S bus to enable communication between the processor 210 and the audio module 270. In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0095] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 270 and the wireless communication module 260 can be coupled via the PCM bus interface. In some embodiments, the audio module 270 can also transmit audio signals to the wireless communication module 260 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0096] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 210 and the wireless communication module 260. For example, the processor 210 communicates with the Bluetooth module in the wireless communication module 260 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 270 can transmit audio signals to the wireless communication module 260 via the UART interface to enable music playback through Bluetooth headphones.

[0097] The MIPI interface can be used to connect the processor 210 to peripheral devices such as the display screen 294 and the camera 293. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 210 and the camera 293 communicate via the CSI interface to enable the electronic device 200 to capture images. The processor 210 and the display screen 294 communicate via the DSI interface to enable the electronic device 200 to display images.

[0098] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 210 to a camera 293, a display screen 294, a wireless communication module 260, an audio module 270, a sensor module 280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0099] USB port 230 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, or USB Type-C port. USB port 230 can be used to connect a charger to charge electronic device 200, and can also be used for data transfer between electronic device 200 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0100] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a structural limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0101] The charging management module 240 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 receives charging input from the wired charger via a USB interface 230. In some wireless charging embodiments, the charging management module 240 receives wireless charging input via the wireless charging coil of the electronic device 200. While charging the battery 242, the charging management module 240 can also supply power to the electronic device 200 via the power management module 241.

[0102] The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, providing power to the processor 210, internal memory 221, display screen 294, camera 293, and wireless communication module 260, etc. The power management module 241 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 241 may also be located within the processor 210. In other embodiments, the power management module 241 and the charging management module 240 may be located in the same device.

[0103] The wireless communication function of electronic device 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor.

[0104] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0105] The mobile communication module 250 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 200. The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 250 may be housed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 and at least some modules of the processor 210 may be housed in the same device.

[0106] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 270A, receiver 270B, etc.) or displays images or videos through the display screen 294. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 210 and may be housed in the same device as the mobile communication module 250 or other functional modules.

[0107] The wireless communication module 260 can provide solutions for wireless communication applications on the electronic device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 260 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0108] In some embodiments, antenna 1 of electronic device 200 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling electronic device 200 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0109] Electronic device 200 implements display functions through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0110] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 200 may include one or N displays 294, where N is a positive integer greater than 1.

[0111] The display screen 294 of the electronic device 200 can display a series of graphical user interfaces (GUIs), which serve as the main screen of the electronic device 200. Generally, the size of the display screen 294 of the electronic device 200 is fixed, and only a limited number of controls can be displayed on the display screen 294. A control is a GUI element, a software component contained in an application, that controls all data processed by the application and interactive operations related to that data. Users can interact with controls through direct manipulation, thereby reading or editing information related to the application. Generally, controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets. For example, in this embodiment, the display screen 291 can display virtual buttons (one-click arrangement, start arrangement, scene arrangement).

[0112] Electronic device 200 can perform shooting functions through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0113] The ISP (Image Signal Processor) is used to process data fed back from the camera 293. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 293.

[0114] Camera 293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 200 may include one or N cameras 293, where N is a positive integer greater than 1.

[0115] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 200 selects a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0116] Video codecs are used to compress or decompress digital video. Electronic device 200 may support one or more video codecs. Thus, electronic device 200 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0117] An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0118] The external storage interface 220 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0119] Internal memory 221 can be used to store computer executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of electronic device 200 by running the instructions stored in internal memory 221. For example, in this embodiment, processor 210 can perform scene arrangement by executing instructions stored in internal memory 221. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of electronic device 200 (such as audio data, phone book, etc.). In addition, internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 210 executes various functional applications and data processing of electronic device 200 by running instructions stored in internal memory 221 and / or instructions stored in memory disposed in the processor.

[0120] Electronic device 200 can implement audio functions such as music playback and recording through audio module 270, speaker 270A, receiver 270B, microphone 270C, headphone jack 270D, and application processor.

[0121] The audio module 270 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 270 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 270 may be located in the processor 210, or some functional modules of the audio module 270 may be located in the processor 210.

[0122] The speaker 270A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 200 can listen to music or make hands-free calls through the speaker 270A.

[0123] The receiver 270B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 200 answers a telephone call or voice message, the receiver 270B can be brought close to the ear to listen to the voice.

[0124] Microphone 270C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 270C, inputting the sound signal into microphone 270C. Electronic device 200 may have at least one microphone 270C. In some embodiments, electronic device 200 may have two microphones 270C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 200 may also have three, four, or more microphones 270C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0125] The headphone jack 270D is used to connect wired headphones. The headphone jack 270D can be a USB 230 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0126] Pressure sensor 280A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 280A can be disposed on display screen 294. There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 280A, the capacitance between the electrodes changes. Electronic device 200 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 294, electronic device 200 detects the intensity of the touch operation based on pressure sensor 280A. Electronic device 200 can also calculate the touch position based on the detection signal from pressure sensor 280A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0127] The gyroscope sensor 280B can be used to determine the motion attitude of the electronic device 200. In some embodiments, the gyroscope sensor 280B can determine the angular velocity of the electronic device 200 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 280B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 280B detects the angle of the electronic device 200's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 200 through reverse movement, thus achieving image stabilization. The gyroscope sensor 280B can also be used in navigation and motion-sensing game scenarios.

[0128] The barometric pressure sensor 280C is used to measure air pressure. In some embodiments, the electronic device 200 calculates altitude using the air pressure value measured by the barometric pressure sensor 280C to assist in positioning and navigation.

[0129] The magnetic sensor 280D includes a Hall sensor. The electronic device 200 can use the magnetic sensor 280D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 200 is a flip phone, the electronic device 200 can detect the opening and closing of the flip cover using the magnetic sensor 280D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0130] The accelerometer 280E can detect the magnitude of acceleration of electronic device 200 in various directions (typically three axes). When electronic device 200 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic device 200, and can be applied to applications such as screen orientation switching and pedometers.

[0131] A distance sensor 280F is used to measure distance. Electronic device 200 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 200 can utilize the distance sensor 280F to measure distance for rapid focusing.

[0132] The proximity sensor 280G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 200 emits infrared light outward through the LED. The electronic device 200 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that an object is near the electronic device 200. When insufficient reflected light is detected, the electronic device 200 can determine that no object is near the electronic device 200. The electronic device 200 may use the proximity sensor 280G to detect when a user holds the electronic device 200 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 280G can also be used in holster mode and pocket mode for automatic unlocking and locking of the screen.

[0133] The ambient light sensor 280L is used to sense the brightness of ambient light. The electronic device 200 can adaptively adjust the brightness of its display screen 294 based on the sensed ambient light level. The ambient light sensor 280L can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 280L can also work in conjunction with the proximity sensor 280G to detect whether the electronic device 200 is in a pocket, preventing accidental touches.

[0134] The fingerprint sensor 280H is used to collect fingerprints. The electronic device 200 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0135] Temperature sensor 280J is used to detect temperature. In some embodiments, electronic device 200 uses the temperature detected by temperature sensor 280J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 280J exceeds a threshold, electronic device 200 reduces the performance of a processor located near temperature sensor 280J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, electronic device 200 heats battery 242 to prevent abnormal shutdown of electronic device 200 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, electronic device 200 boosts the output voltage of battery 242 to prevent abnormal shutdown due to low temperature.

[0136] Touch sensor 280K, also known as a "touch device," can be located on display screen 294. The touch sensor 280K and display screen 294 together form a touchscreen, also known as a "touchscreen." Touch sensor 280K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 294. In other embodiments, touch sensor 280K may also be located on the surface of electronic device 200, in a different position than display screen 294.

[0137] The bone conduction sensor 280M can acquire vibration signals. In some embodiments, the bone conduction sensor 280M can acquire vibration signals from vibrating bone segments in the human vocal cords. The bone conduction sensor 280M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 280M can also be incorporated into headphones to form bone conduction headphones.

[0138] The audio module 270 can analyze the vibration signal of the sound-vibrating bone block acquired by the bone conduction sensor 280M to extract the voice signal and realize the voice function. The application processor can analyze the heart rate information based on the blood pressure and heart rate signals acquired by the bone conduction sensor 280M to realize the heart rate detection function.

[0139] Buttons 290 include a power button, volume buttons, etc. Buttons 290 can be mechanical buttons or touch-sensitive buttons. Electronic device 200 can receive button input and generate key signal inputs related to user settings and function control of electronic device 200.

[0140] Motor 291 can generate vibration alerts. Motor 291 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 291 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 294. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0141] Indicator 292 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0142] The SIM card interface 295 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 295 to make contact with and separate from the electronic device 200. The electronic device 200 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 295 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 295 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 295 is also compatible with different types of SIM cards. The SIM card interface 295 is also compatible with external memory cards. The electronic device 200 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 200 and cannot be separated from the electronic device 200.

[0143] In addition, an operating system runs on top of the aforementioned components. Examples include HarmonyOS, iOS, Android (open-source operating system), and Windows. Applications can be installed and run on this operating system.

[0144] Figure 3 This is a software structure block diagram of an electronic device provided in an embodiment of this application.

[0145] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments of this application, the software architecture of the electronic device includes at least five layers, from top to bottom: the application layer, the application framework layer, the algorithm engine layer, the system library layer, and the kernel layer.

[0146] The application layer may include a series of application packages. These application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS. In this embodiment, the application layer includes a camera app that can activate different cameras, such as a wide-angle camera, a main camera, a front camera, and a rear camera. When different cameras are activated, the camera has different preview stream areas, such as a wide-angle preview stream area and a main camera preview stream area.

[0147] Understandably, the applications included in the application layer do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, compared to Figure 3 The application layer shown contains applications. Electronic devices 200 may include more or fewer applications. Different electronic devices 200 may include the same application or completely different applications.

[0148] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 3 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc. For example, in some embodiments of this application, the application framework layer may also include a text-to-speech (TTS) module. The TTS module is used to issue voice prompts and also to read aloud the content of the text.

[0149] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0150] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0151] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0152] Activity Manager is used to manage the lifecycle of an application and provides commonly used navigation and back functions.

[0153] The algorithm engine layer provides application with algorithm interfaces and frameworks. These interfaces include predefined functions and regular expressions, etc. Figure 3 As shown, the algorithm engine layer includes a text detection module, an edge detection module, and a distance calculation module. The text detection module is used to detect text content. Specifically, it performs text detection on the image captured by the camera on the electronic device, determining whether text is detected or not. The edge detection module is used to detect edges. Specifically, after the text detection module detects text, it determines whether the edges of the text are complete (whether they are completely displayed in the preview stream area of ​​the electronic device, or whether all the edges of the text are within the preview stream area of ​​the corresponding camera). The distance calculation module is used to calculate the positional relationship, such as the distance relationship (target distance), between the edges of the text detected by the edge detection module and the edges of the camera preview stream area.

[0154] The system library layer can include multiple functional modules. Examples include media libraries, graphics processing libraries (e.g., OpenGL ES), and the Android runtime.

[0155] The media library is used to handle different streaming media inputs. For example, the media library supports playback and recording of various common audio and video formats, as well as still image files. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. In some embodiments, the media library includes a MediaLibrary and a MediaProvider. The MediaLibrary provides the data query API for the gallery's Photo data interface with the function of querying the database and filtering data that meets the requirements; the MediaProvider provides the data update API for the gallery's Photo data interface with the function of initiating media scans and updating the database.

[0156] The graphics processing library is used to implement graphics drawing, image rendering, compositing, and layer processing, etc.

[0157] The Android Runtime consists of the core libraries and the virtual machine. The Android Runtime is responsible for the navigation and management of the Android system. The core libraries comprise two parts: one part contains the functionalities that Java calls, and the other part contains the core Android libraries.

[0158] The kernel layer is the layer between hardware and software. It contains at least display drivers, camera drivers, audio drivers, and sensor drivers. The hardware can include devices such as cameras, displays, microphones, processors, and memory.

[0159] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as HarmonyOS, iOS, and Windows.

[0160] For ease of understanding, the following embodiments of this application will be described using the following methods: Figure 2 and Figure 3 Taking the electronic device 200 with the structure shown as an example, the auxiliary reading method provided by the embodiments of this application will be specifically described in conjunction with the accompanying drawings and application scenarios.

[0161] Figure 4 A flowchart illustrating an embodiment of the auxiliary reading method provided in this application. Figure 5 This is a module interaction diagram of an auxiliary reading method provided in one embodiment of this application.

[0162] like Figure 4 and Figure 5 As shown, this assisted reading method is applied to electronic devices, and the assisted reading method may include:

[0163] Step S401: The electronic device uses a camera to capture an image of the text, obtains a first image, and displays the first image on the screen;

[0164] Step S402: When the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved;

[0165] Step S403: The electronic device uses a camera to capture an image of the text, obtains a second image, and displays the second image on the screen;

[0166] Step S404: When the second image includes all the content of the text, the electronic device reads aloud the entire content of the text.

[0167] As an example, the text that electronic devices are meant to assist in reading can be text and images, pictures, documents, identification codes, etc., without any specific limitations.

[0168] As an example, the identification code can be a QR code, barcode, irregular code, mini-program code, QR code, or one-dimensional code, etc.

[0169] As an example, the document can be Excel, Word, PowerPoint, Notepad, business card, etc., and there are no specific limitations.

[0170] As an example, documents can also be paper documents, brochures, business cards, slides, exam papers, invoices, cards, etc.

[0171] As an example, the card or certificate can be an ID card, bank card, driver's license, etc.

[0172] In this embodiment, the electronic device uses a camera to capture images of text such as books, contracts, drug instructions, graphics, pictures, documents, and identification codes to obtain a first image. This first image is then displayed on the screen. Subsequently, the electronic device performs text detection on the first image based on its internal text detection module. If text is detected, the edge detection module determines whether the edges of the text are within the camera's preview stream area. If not... Figure 21 Then, based on the distance calculation module of the electronic device, the target distance between the edge of the text and the corresponding edge of the preview stream area is determined (the distance between the edge of each text and the corresponding edge of the camera preview stream area is the target distance), and the positional relationship between the target distance and the corresponding preset distance is determined, so as to determine the first movement direction of the user moving the electronic device based on the positional relationship, and / or determine the second movement direction of the user moving the text. After determining the first movement direction and the second movement direction, a prompt is given based on the voice TTS module. If a voice output trigger condition is detected, a voice prompt is given based on the voice TTS module.

[0173] As an example, if prompts are made based on the TTS (Text-to-Speech) module, the distance calculation module needs to send the corresponding prompt content to the TTS module for output.

[0174] As an example, after the electronic device issues a voice prompt (the voice prompt instructs the user to move the electronic device and / or move the text), the user moves the electronic device and / or moves the text based on the voice prompt. During the user's movement of the electronic device and / or the text, the electronic device also uses corresponding acquisition rules, such as using a corresponding acquisition frequency, to capture an image of the text using a camera, obtaining a second image, and displays the second image on the screen. When the second image includes the entire content of the text, the electronic device reads aloud the entire content of the text.

[0175] As an example, the electronic device will only stop capturing images when the captured second image includes the entire content of the text.

[0176] As an example, the electronic device sends the entire content of the acquired text to the voice TTS module so that the voice TTS module can output the entire content of the text.

[0177] The specific steps are as follows;

[0178] Step S401: The electronic device uses a camera to capture an image of the text, obtains a first image, and displays the first image on the screen;

[0179] As an example, there can be one, two, or more cameras, and these cameras can be of the same or different types.

[0180] As an example, an electronic device can use multiple cameras to capture images of text, obtain multiple first images, and then display only one first image on the screen.

[0181] As an example, when an electronic device uses multiple cameras to capture images of text, one of the cameras is a wide-angle camera.

[0182] As an example, an electronic device may activate multiple cameras sequentially or simultaneously to capture images of text, thus obtaining a first image.

[0183] As an example, the electronic device first uses a wide-angle camera to capture an image of the text, and then uses the main camera to capture an image of the text.

[0184] As an example, the electronic device first uses a wide-angle camera to capture an image of the text. If the first image only includes part of the text, then the main camera is used to capture an image of the text to determine whether it includes all or part of the text.

[0185] As an example, the electronic device first uses a wide-angle camera to capture an image of the text. If the content of the text in the first image cannot be recognized, then it uses the main camera to capture an image of the text and determines whether it includes all or part of the text.

[0186] As an example, the electronic device first uses a wide-angle camera to capture an image of the text. If the clarity of the text content in the first image is less than a clarity threshold, then the main camera is used to capture an image of the text, and the result is used to determine whether the image includes all or part of the text.

[0187] As an example, the electronic device first uses the main camera to capture an image of the text. If the first image only includes part of the text, then the image obtained after capturing the text using the main camera is used to determine whether it includes all or part of the text.

[0188] As an example, an electronic device first uses the main camera to capture an image of the text. If the first image only includes part of the text, then the image obtained by simultaneously capturing images of the text using both the main camera and the wide-angle camera is used to determine whether it includes all or part of the text.

[0189] As an example, an electronic device first uses a wide-angle camera to capture an image of the text. If the first image only includes part of the text, then the image obtained by simultaneously capturing images of the text using both the main camera and the wide-angle camera is used to determine whether it includes all or part of the text.

[0190] As an example, if an electronic device simultaneously uses a wide-angle camera and a main camera to capture images of text, and the first image obtained from the wide-angle camera only includes a portion of the text, then the image obtained after simultaneously capturing images of the text using the main camera and the wide-angle camera can be used to determine whether it includes all or part of the text.

[0191] As an example, if an electronic device simultaneously uses a wide-angle camera and a main camera to capture images of text, and the first image obtained from the main camera only includes a portion of the text, then the image obtained after simultaneously capturing images of the text using both the main camera and the wide-angle camera can be used to determine whether it includes all or part of the text.

[0192] As an example, the step of the electronic device using a camera to capture images of text includes the following step A1:

[0193] Step A1: When the electronic device is in auxiliary mode, the electronic device uses the camera to capture images of the text.

[0194] As an example, assistive modes specifically include assisted reading mode, accessibility mode, visually impaired mode, screen reading (mode), etc.

[0195] As an example, the assistive mode or screen reading (mode) can be turned on or off via a mode (function) button. Specifically, the electronic device has a mode function button directly set up, and you can enter the assistive mode by clicking the function button.

[0196] As an example, Figure 30 This is a schematic diagram of the interface for entering accessibility mode. Specifically, the user clicks the settings icon on the main screen of the electronic device to enable the device to display... Figure 30 (a) Interface, where the user... Figure 30 (a) Locate the accessibility icon on the interface and click it to make the electronic device display... Figure 30 (b) Interface, where the user... Figure 30 (b) Locate the accessibility icon on the interface and click it to make the electronic device display... Figure 30 (c) Interface, where the user... Figure 30 (c) Locate the screen reading icon on the interface and click it. At this time, the electronic device is in screen reading mode. Figure 30 (d)).

[0197] As an example, Figure 31 This is a schematic diagram of the interface for entering the auxiliary mode via the camera function button. Specifically, the user can enter the capture or shooting interface by clicking the camera icon on the main interface of the electronic device, and then click the corresponding auxiliary mode or reading auxiliary mode button in the capture or shooting interface.

[0198] As an example, electronic devices can also enter assistive mode by clicking on a pre-set assistive application. For example, the pre-set assistive application could be a screen reader (application name), and the assistive mode can be entered by clicking on the screen reader application.

[0199] As an example, accessibility mode can also be accessed by clicking a keyboard shortcut.

[0200] As an example, the assistance mode can also be turned on automatically at set intervals.

[0201] As an example, after obtaining a first image, the electronic device performs recognition on the first image.

[0202] The process by which an electronic device recognizes the first image can be as follows:

[0203] The electronic device determines whether the first image contains the content of the text. If it does not contain the text, the subsequent steps are not performed. If it does contain the text, the electronic device performs edge detection on the first image.

[0204] As an example, the corresponding edge lines of the text are determined in order to detect the edges of the corresponding text.

[0205] If it is determined that the edge (line) of the first image is not within the corresponding camera preview stream area, the electronic device determines that the first image only includes a portion of the text (the text was not successfully recognized); if it is determined that the text in the first image is within the corresponding camera preview stream area, the electronic device determines the clarity of the text content in the first image. If the clarity is greater than the clarity threshold, the electronic device can extract the entire text content, and at this time, the electronic device determines that the text content in the first image has been successfully recognized. If the clarity is not greater than the clarity threshold, the electronic device cannot extract the entire text content, and at this time, the electronic device determines that the text content in the first image has not been successfully recognized.

[0206] Step S402: When the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved;

[0207] When the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved.

[0208] As an example, a voice prompt instructs the movement of the electronic device;

[0209] As an example, a voice prompt instructs the user to move the text.

[0210] As an example, a voice prompt instructs the movement of the text and the electronic device.

[0211] As an example, the prompts could include: left, right, up, down, further away, closer, and slower.

[0212] As an example, it can also indicate the specific content being moved, such as how far it is being moved.

[0213] As an example, the content of movement can be output through the prompt module or the voice TTS module; there is no specific limitation.

[0214] As an example, when outputting moving content through a voice TTS module, it is necessary to first determine the specific moving content and then send the specific moving content to the voice TTS module for output.

[0215] As an example, if the text is page content, after the first image is displayed on the screen, if it is detected that the edge of the text is within the preview stream area of ​​the corresponding camera, and a trigger condition for assisting in reading the full view of the page containing the text is detected, it is determined whether the edge of the page containing the text is within the preview stream area of ​​the corresponding camera. If it is not within the preview stream area of ​​the corresponding camera, a prompt is made to move the electronic device or the text.

[0216] As an example, after detecting that the edges of the text are within the preview stream area of ​​the corresponding camera, such as Figure 28 As shown, if a user requests to view or assist in reading the full content of the page containing the text (the user triggers the corresponding request through a triggering condition), it is further determined whether the edges of the page containing the text are fully displayed in the camera's preview stream or viewfinder. If they are not fully displayed in the camera's preview stream or viewfinder, the user is prompted to continue moving the electronic device or the text until the edges of the page containing the text are fully displayed in the camera's preview stream or viewfinder.

[0217] As an example, if the text is a card or certificate, after the first image is displayed on the screen, if the edge of the text is detected to be within the preview stream area of ​​the corresponding camera, and a trigger condition for assisting in reading the full view of the document containing the card or certificate is detected, it is determined whether the edge of the document containing the card or certificate is within the preview stream area of ​​the corresponding camera. If it is not within the preview stream area of ​​the corresponding camera, a prompt is made to move the electronic device or the document.

[0218] As an example, after the electronic device or the document is moved, the electronic device uses its camera to capture an image of the page containing the text, obtaining a second image, and displays the second image on the screen. After detecting that the edge of the text is within the preview stream area of ​​the corresponding camera, it further determines whether the edge of the page containing the text is fully presented in the preview stream or viewfinder of the camera (determining whether the edge of the page containing the text is fully scanned or captured). If it is fully presented in the preview stream or viewfinder of the camera, the electronic device reads aloud the entire content of the page containing the text.

[0219] In one possible implementation, if the triggering condition of the auxiliary mode is detected, it is determined that the electronic device is in the auxiliary mode.

[0220] As an example, Figure 30This is a schematic diagram of the interface for entering accessibility mode. Specifically, the user clicks the settings icon on the main screen of the electronic device to enable the device to display... Figure 30 (a) Interface, where the user... Figure 30 (a) Locate the accessibility icon on the interface and click it to make the electronic device display... Figure 30 (b) Interface, where the user... Figure 30 (b) Locate the accessibility icon on the interface and click it to make the electronic device display... Figure 30 (c) Interface, where the user... Figure 30 (c) The screen reading icon is found on the interface and clicked. At this time, the electronic device is in screen reading (mode) 30 (d).

[0221] As an example, Figure 31 This is a schematic diagram of the interface for entering the auxiliary mode via the camera function button. Specifically, the user can enter the image acquisition or shooting interface by clicking the camera icon on the main interface of the electronic device, and then click the corresponding auxiliary mode or reading auxiliary mode button in the image acquisition or shooting interface.

[0222] As an example, it is possible to set an electronic device to be in auxiliary mode;

[0223] As an example, accessibility mode can be set by clicking the accessibility mode shortcut key.

[0224] As an example, accessibility mode can also be triggered by actions such as clicking the application icon.

[0225] As an example, electronic devices designed for visually impaired individuals can be set to assistive mode at the factory.

[0226] As an example, the electronic device is determined to be in auxiliary mode whenever the trigger condition for auxiliary mode is detected.

[0227] As an example, the triggering conditions can be determined according to the actual situation, and all triggering conditions are within the protection scope of this application.

[0228] In one possible implementation, the step of the electronic device using a camera to capture an image of text when the electronic device is in auxiliary mode includes the following step B1:

[0229] Step B1: When the electronic device is in auxiliary mode, if a trigger condition for using the camera to capture images of text is detected, the electronic device uses the camera to capture images of the text.

[0230] The triggering conditions for using a camera to capture images of text are different from the triggering conditions for camera recording.

[0231] As an example, when the electronic device is in auxiliary mode, if a trigger condition for using the camera to capture an image of the text is detected, the electronic device uses the camera to capture an image of the text.

[0232] As an example, the triggering condition for using a camera to capture images of text differs from the triggering condition for camera recording.

[0233] As an example, one reason why the triggering condition for using a camera to capture images of text differs from the triggering condition for camera recording is to avoid confusion between camera shooting and entering assisted reading mode.

[0234] As an example, such as Figure 20 As shown, after being prompted, the user moves the electronic device in the direction of the arrow until all the text in the image captured by the camera of the electronic device is recognized.

[0235] As an example, in this embodiment, the prompt text suggesting moving the electronic device and / or moving the text can be the user or other devices.

[0236] As an example, this embodiment can be applied in a scenario where an electronic device alerts a drone or other device to move.

[0237] That is, as an example, the scenario in which this embodiment is applied can be: the text corresponding to the image captured by the electronic device is on other devices such as a robotic arm or robotic hand. In this case, if the first image only includes part of the text, the electronic device prompts the robotic arm or robotic hand or other devices to move.

[0238] Step S403: The electronic device uses a camera to capture an image of the text, obtains a second image, and displays the second image on the screen;

[0239] As an example, after being prompted, the user moves the electronic device or the text, and at the same time, the electronic device uses its camera to capture an image of the text, obtains a second image, and displays the second image on the screen;

[0240] As an example, an electronic device uses a camera to capture images of the text at a certain acquisition frequency to obtain a second image, and then displays the second image on a screen.

[0241] As an example, an electronic device uses multiple cameras to capture images of the text, obtains a second image, and displays the second image on a screen.

[0242] As an example, if the first image is captured using multiple cameras, then the second image is captured using the same camera and the same capture method.

[0243] Step S404: When the second image includes all the content of the text, the electronic device reads aloud the entire content of the text.

[0244] As an example, when the second image includes the entire content of the text, the electronic device reads aloud the entire content of the text.

[0245] As an example, the volume of the reading can be adjusted.

[0246] As an example, if the second image includes the entire content of the text, then the entire content of the text is output through the TTS (Text-to-Speech) module so that the user can obtain the corresponding content in a timely manner.

[0247] In one possible implementation, after the step of the electronic device reading aloud the entire content of the text when the second image includes it, the method further includes the following step S405:

[0248] In step S405, the electronic device issues a voice instruction indicating that it is no longer necessary to capture images of the text.

[0249] As an example, when the second image includes the entire content of the text, a prompt is made indicating that there is no need to capture the image of the text again, thus avoiding hand tremors or hand fatigue caused by prolonged use of electronic devices, thereby improving the user experience.

[0250] As an example, the electronic device issues a voice instruction, which includes: Please put down the electronic device, the data collection is complete, etc.

[0251] As an example, there is no need to capture images of the text, thus avoiding the waste of resources caused by the electronic device constantly capturing images.

[0252] As an example, it is necessary to determine that the electronic device should only read the entire text when the second image includes the entire text, rather than simultaneously, to avoid degrading the user experience (which would require continuous image acquisition by the electronic device).

[0253] In one possible implementation, when the second image includes the entire content of the text, the electronic device reads the entire content of the text aloud when a trigger condition for reading the entire text is detected.

[0254] As an example, the purpose of setting reading-aloud trigger conditions is:

[0255] This is suitable for scenarios where, under special circumstances, the text content does not need to be output via voice.

[0256] As an example, if the current scene is silent, it is not necessary to read the entire text aloud.

[0257] As an example, text can also be divided into different parts. For example, if the text is a contract with a total of 5 parts, and the user only focuses on the third part of the contract, then the electronic device will read the content of the third page of the contract aloud based on the user's selection.

[0258] This application's electronic device uses a camera to capture an image of text, obtaining a first image, and displays the first image on a screen. When the first image only includes a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device and / or the text to be moved. The electronic device then uses the camera to capture an image of the text, obtaining a second image, and displays both images on a screen. When the second image includes the entire text, the electronic device reads aloud the entire text. This application avoids the technical problem of assisted reading failure and low efficiency caused by the first image only including a portion of the text.

[0259] Figure 6 A flowchart of an auxiliary reading method provided in another embodiment of this application is shown below. Figure 6 As shown, the camera includes at least a first camera and a second camera, wherein the field of view of the first camera is greater than that of the second camera;

[0260] The above-mentioned methods for assisting reading may include:

[0261] Step S601: The electronic device uses a first camera and / or a second camera to capture images of the text, obtains a first image, and displays the first image on the screen;

[0262] Step S602: When the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved;

[0263] Step S603: The electronic device uses a first camera and / or a second camera to capture an image of the text, obtains a second image, and displays the second image on the screen;

[0264] Step S604: When the second image includes all the content of the text, the electronic device reads aloud the entire content of the text.

[0265] In this embodiment, as Figure 10 As shown, the electronic device uses a first camera and / or a second camera to capture images of text (books, contracts, drug instructions, etc.) to obtain a first image. This first image is then displayed on a screen. Simultaneously, the electronic device uses its internal text detection module to detect the content of the text in the first image. If no text content is detected, the process ends. If text content is detected, the electronic device's edge detection module determines whether the edges of the text in the first image are within the corresponding camera preview stream area. If they are within the preview stream area, the entire text is read aloud. If they are not within the preview stream area, the first image is determined to contain only a portion of the text. When the first image contains only a portion of the text, the electronic device issues a voice prompt, which instructs the user to move the electronic device and / or move the text (e.g., ...). Figure 20 As shown), the electronic device uses a first camera and / or a second camera to capture images of the text, obtains a second image, and displays the two images on the screen. Figure 22 When the second image includes all the content of the text, the electronic device reads aloud the entire content of the text.

[0266] The specific steps are as follows:

[0267] Step S601: The electronic device uses a first camera and / or a second camera to capture images of the text, obtains a first image, and displays the first image on the screen;

[0268] Specifically, (when the electronic device is in auxiliary mode), the electronic device uses a first camera and / or a second camera to capture images of the text to obtain a first image, including: (when the electronic device is in auxiliary mode), the electronic device uses the first camera to capture images of the text to obtain a first image; (when the electronic device is in auxiliary mode), the electronic device uses the second camera to capture images of the text to obtain a first image; (when the electronic device is in auxiliary mode), the electronic device uses both the first camera and the second camera to capture images of the text to obtain different first images.

[0269] As an example, the electronic device first uses a wide-angle camera to capture an image of the text, and then uses the main camera to capture an image of the text.

[0270] As an example, such as Figure 23 And such as Figure 29As shown, although the first image captured by the main camera cannot include the entire text, the first camera (ultra-wide-angle camera) has a wider field of view (larger acquisition area, which can capture an image including the entire text at once to the maximum extent). Therefore, the process can be streamlined by first starting the first camera (ultra-wide-angle camera) to acquire images, and only starting the second camera (main camera) to acquire images when the image captured by the first camera (ultra-wide-angle camera) cannot include the entire text, and / or when the image captured by the first camera (ultra-wide-angle camera) cannot include the entire text.

[0271] As an example, the electronic device first uses a wide-angle camera to capture an image of the text. If the first image only includes part of the text, then the main camera is used to capture an image of the text to determine whether it includes all or part of the text.

[0272] As an example, the electronic device first uses a wide-angle camera to capture an image of the text. If the content of the text in the first image cannot be recognized, then it uses the main camera to capture an image of the text and determines whether it includes all or part of the text.

[0273] As an example, the electronic device first uses a wide-angle camera to capture an image of the text. If the clarity of the text content in the first image is less than a clarity threshold, then the main camera is used to capture an image of the text, and the result is used to determine whether the image includes all or part of the text.

[0274] As an example, the electronic device first uses the main camera to capture an image of the text. If the first image only includes part of the text, then the image obtained after capturing the text using the main camera is used to determine whether it includes all or part of the text.

[0275] As an example, since electronic devices may prioritize using the main camera for image acquisition, in this embodiment, the electronic device first uses the main camera to acquire an image of the text. If the first image only includes part of the text, then the image obtained after acquiring the text using the main camera is used to determine whether it includes all or part of the text. This conforms to the inherent or pre-defined acquisition process of the electronic device and avoids wasting resources.

[0276] Step S602: When the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved;

[0277] As an example, when the first image includes only a portion of the text, such as only half of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved.

[0278] Prior to the step of the electronic device issuing a voice prompt when the first image only includes a portion of the text, the procedure includes:

[0279] Step D1: When the electronic device fails to recognize the content of the text in the image captured by the first camera or the second camera, the electronic device determines whether the first image only includes a portion of the text based on the image captured by the first camera and the image captured by the second camera.

[0280] As an example, an electronic device first uses the main camera to capture an image of the text. If the first image only includes part of the text, then the image obtained by simultaneously capturing images of the text using both the main camera and the wide-angle camera is used to determine whether it includes all or part of the text.

[0281] As an example, an electronic device first uses a wide-angle camera to capture an image of the text. If the first image only includes part of the text, then the image obtained by simultaneously capturing images of the text using both the main camera and the wide-angle camera is used to determine whether it includes all or part of the text.

[0282] As an example, if an electronic device simultaneously uses a wide-angle camera and a main camera to capture images of text, and the first image obtained from the wide-angle camera only includes a portion of the text, then the image obtained after simultaneously capturing images of the text using the main camera and the wide-angle camera can be used to determine whether it includes all or part of the text.

[0283] As an example, if an electronic device simultaneously uses a wide-angle camera and a main camera to capture images of text, and the first image obtained from the main camera only includes a portion of the text, then the image obtained after simultaneously capturing images of the text using both the main camera and the wide-angle camera can be used to determine whether it includes all or part of the text.

[0284] As an example, the electronic device does not perform subsequent steps when it successfully recognizes the text content in the image captured by the first camera or the second camera.

[0285] Wherein, when the electronic device fails to recognize the content of the text in the image captured by the first camera or the second camera, before the electronic device determines whether the first image only includes a portion of the text based on the images captured by the first camera and the second camera, the electronic device includes the following steps E1-E2:

[0286] Step E1: The electronic device identifies the image captured by the first camera or the second camera. If it is determined that the image captured by the first camera or the second camera includes the text content, then edge detection is performed on the text in the captured image.

[0287] As an example, the electronic device recognizes the image captured by the first camera. If it is determined that the image captured by the first camera includes the text content, then edge detection is performed on the text in the captured image.

[0288] As an example, the electronic device recognizes the image captured by the second camera. If it is determined that the image captured by the second camera includes the text content, then edge detection is performed on the text in the captured image.

[0289] Step E2: If it is determined that the edge of the text in the captured image is not within the preview stream area of ​​the corresponding camera, then the electronic device determines that it has failed to successfully recognize the content of the text in the captured image.

[0290] In this embodiment, if it is determined that the edge of the text in the captured image is within the preview stream area of ​​the corresponding camera, the electronic device determines that it has successfully recognized the content of the text in the captured image.

[0291] In this embodiment, if it is determined that the edge of the text in the captured image is not within the preview stream area of ​​the corresponding camera (if the camera capturing the image is the first camera, then the preview stream area is the preview stream area of ​​the first camera; if the camera capturing the image is the second camera, then the preview stream area is the preview stream area of ​​the second camera), then the electronic device determines that it has failed to successfully recognize the content of the text in the captured image.

[0292] The step of the electronic device determining whether the first image contains only a portion of the text based on the image captured by the first camera and the image captured by the second camera includes the following steps F1-F3:

[0293] Step F1: The electronic device detects whether the image captured by the first camera includes the text content. When the image captured by the first camera is detected to include the text content, the electronic device performs edge detection on the text in the image captured by the second camera.

[0294] In this embodiment, it is explained how the electronic device performs joint detection on the image captured by the first camera and the image captured by the second camera, and determines whether the first image only includes a portion of the text.

[0295] As an example, such as Figure 28 As shown, the electronic device first uses a text detection module to detect whether the image captured by the first camera includes the text content. When the text content is detected in the image captured by the first camera, the electronic device uses an edge detection module to perform edge detection on the text in the image captured by the second camera.

[0296] Step F2: If the edge of the text in the image captured by the second camera is not within the preview stream area of ​​the second camera, then the electronic device determines whether the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera.

[0297] As an example, such as Figure 28 As shown, if the edge of the text in the image captured by the second camera is not within the preview stream area of ​​the second camera, the electronic device determines whether the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera based on the edge detection module.

[0298] In step E3, if the electronic device determines that the edge of the text in the image captured by the first camera is not within the preview stream area of ​​the first camera, then it determines that the first image only includes a portion of the text.

[0299] As an example, if the electronic device determines that the edge of the text in the image captured by the first camera is not within the preview stream area of ​​the first camera, then it determines that the first image only includes a portion of the text.

[0300] As an example, if the electronic device determines that the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera, further judgment is required.

[0301] If the edge of the text in the image captured by the second camera is not within the preview stream area of ​​the second camera, then after the electronic device determines whether the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera, the following steps G1-G2 are included:

[0302] Step G1: If the electronic device determines that the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera, then it determines whether the clarity of the text in the image captured by the second camera is greater than the clarity threshold.

[0303] In step G2, if the resolution is not greater than the resolution threshold, the electronic device determines that the first image only includes a portion of the text.

[0304] As an example, if the electronic device determines that the edge of the text in the image captured by the first camera is within the preview stream area of ​​the first camera, it then determines whether the sharpness of the text in the image captured by the second camera is greater than a sharpness threshold. If it is not greater than the sharpness threshold, the electronic device determines that the first image only includes a portion of the text. If it is greater than the sharpness threshold, the electronic device determines that the first image includes the entire text.

[0305] Step S603: The electronic device uses a first camera and / or a second camera to capture an image of the text, obtains a second image, and displays the second image on the screen;

[0306] As an example, after being prompted, the user moves the electronic device or the text, and at the same time, the electronic device uses a first camera and / or a second camera to capture an image of the text, obtain a second image, and display the two images on the screen;

[0307] As an example, an electronic device uses a first camera and / or a second camera to capture images of the text at a certain acquisition frequency to obtain a second image, and then displays the two images on a screen.

[0308] As an example, an electronic device uses multiple cameras to capture images of the text, obtains a second image, and displays the second image on a screen.

[0309] As an example, if the first image is captured using multiple cameras, then the second image is captured using the same camera and the same capture method.

[0310] Step S604: When the second image includes all the content of the text, the electronic device reads aloud the entire content of the text.

[0311] As an example, when the second image includes the entire content of the text, the electronic device reads aloud the entire content of the text.

[0312] As an example, the volume of the reading can be adjusted.

[0313] As an example, if the second image includes the entire content of the text, then the entire content of the text is output through the TTS (Text-to-Speech) module so that the user can obtain the corresponding content in a timely manner.

[0314] As an example, Figure 27 The overall flowchart of the assisted reading method provided in another embodiment of this application (the first camera and the second camera assist in reading text together) is as follows: Specifically, the electronic device is a mobile phone, which includes multiple cameras. After the mobile phone activates the multiple cameras, the camera with the largest fovea (camera) is the first camera, and the main camera (main camera, second lens) is the second camera. The main camera and the camera work simultaneously, and the mobile phone screen can display only the content of the main camera or display the content of the main camera and the ultra-wide-angle lens at the same time (201).

[0315] As an example, the first camera (ultra-wide-angle camera, first lens) sends the captured image of the text to the text detection module (electronic device) for detection, and determines whether the text detection module has detected the text or the content of the text (202). If the text or the content of the text is detected, the subsequent process continues (the subsequent process starts at 203 or 210). If the text or the content of the text is not detected, the text detection module detects whether the image captured by the second camera includes the text or the text (209). If it includes the text, the subsequent process continues (the subsequent process starts at 203). If it does not include the text, the process exits.

[0316] As an example, the first camera (camera, first lens) sends the captured image of the text to the text detection module of the (electronic device) for detection, and determines whether the text detection module has detected the text or the content of the text (202). If the text or the content of the text is detected, and the clarity detection module of the electronic device determines that the clarity of the text or the content of the text is greater than the preset clarity threshold, then the text or all the content of the text is extracted and the text or all the content of the text is read out (204).

[0317] As an example, the edge detection module of the electronic device detects the edges of the text corresponding to the image captured by the second camera, and detects whether the text edges are complete (whether the text edges are within the preview stream area of ​​the second camera) (203). If they are complete (the text edges are within the preview stream area of ​​the second camera), the text or all the content of the text is extracted, and the text or all the content of the text is read out (204).

[0318] As an example, if the edge detection module of the electronic device detects that the text edges in the image captured by the second camera (main camera) are incomplete (the edges of the text are within the preview stream area of ​​the second camera), then it is determined whether the text edges or scanned text in the image captured by the first camera (camera) are complete (205) (each edge of the text is within the preview stream area of ​​the corresponding camera). If the content of the text in the image captured by the first camera (camera) is incomplete (the captured image only includes part of the text content), then according to the method of calculating the target distance in the above embodiment (the distance between each edge of each text and the corresponding edge of the preview stream area of ​​the corresponding camera is the target distance), the user is prompted to move the electronic device or the user is prompted to move the text (based on the comparison result between the target distance and the corresponding preset distance, the target distance and the direction of movement are determined) until the first lens (ultra-wide-angle camera) scans the entire text (207).

[0319] As an example, if the edge detection module of the electronic device detects that the text edges of the image captured by the first lens (ultra-wide-angle camera) are complete (each edge of the text is within the corresponding ultra-wide-angle camera preview stream area), then the sharpness calculation module determines whether the content of the text or the sharpness of the text in the image captured by the first lens (ultra-wide-angle camera) is greater than a certain threshold (206). If it is greater, then the text in the captured image is read and all the content is read (204).

[0320] As an example, if it is determined that the text content in the image captured by the first lens (camera) is not clear, the target distance is calculated according to the above scheme (the distance between each edge of each text and the corresponding edge of the corresponding camera preview stream area is the target distance), prompting the user to move the electronic device or move the text until the main camera scans the entire text (208).

[0321] In this application, the electronic device uses a first camera and / or a second camera to capture images of text, obtaining a first image, and displays the first image on a screen. When the first image only includes a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the movement of the electronic device and / or the movement of the text. The electronic device then uses the first camera and / or the second camera to capture images of the text, obtaining a second image, and displays the second image on a screen. When the second image includes the entire text, the electronic device reads aloud the entire text. This application avoids wasting resources.

[0322] Figure 7 A flowchart illustrating an auxiliary reading method provided in yet another embodiment of this application. For example... Figure 7 As shown, the above-mentioned reading assistance methods may include:

[0323] Step S701: The electronic device uses a camera to capture an image of the text, obtains a first image, and displays the first image on the screen;

[0324] Step S702: When the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to translate, rotate, or turn, or the voice prompt also instructs the electronic device to move closer to or away from the text;

[0325] And / or in step S703, when the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the text to be translated, rotated, or turned, or the voice prompt further instructs the text to move closer to or further away from the electronic device;

[0326] Step S704: The electronic device uses a camera to capture an image of the text, obtains a second image, and displays the second image on the screen;

[0327] Step S705: When the second image includes all the content of the text, the electronic device reads aloud the entire content of the text.

[0328] That is, in this embodiment, movement includes translation, rotation, turning, moving closer to or further away from.

[0329] As an example, movement can be: indicating translation, rotation, turning of the electronic device, indicating the electronic device to move closer to the text, or indicating the electronic device to move away from the text.

[0330] As an example, movement can be: indicating translation, rotation, turning the text, indicating the electronic device to move closer to the text, or indicating the text to move away from the electronic device.

[0331] As an example, such as Figure 25 As shown, the electronic device uses a camera to capture an image of the text, obtaining a first image, which is then displayed on the screen. When the first image only includes a portion of the text (the text is planar and oriented), a voice prompt instructs the electronic device to rotate. The voice prompt also indicates a first direction and a first distance of movement (rotation) when rotating the electronic device, or a voice prompt instructs the device to rotate the text, along with a second direction and a second distance of movement (rotation). After rotation, the electronic device uses the camera to capture an image of the text, obtaining a second image, which is then displayed on the screen. When the second image includes the entire text, the electronic device reads aloud the entire text.

[0332] As an example, such as Figure 25 As shown, the electronic device uses a camera to capture an image of the text, obtaining a first image, which is then displayed on the screen. When the first image only includes a portion of the text (the text is planar and not oriented), a voice prompt instructs the electronic device to rotate. The voice prompt also indicates a first direction and a first distance of movement (rotation) when rotating the electronic device, or a voice prompt instructs the text to rotate, along with a second direction and a second distance of movement (rotation). After rotation, the electronic device uses the camera to capture an image of the text, obtaining a second image, which is then displayed on the screen. When the second image includes the entire text, the electronic device reads aloud the entire text.

[0333] As an example, such as Figure 24As shown, the electronic device uses a camera to capture an image of text (curved text) to obtain a first image, which is then displayed on the screen. When the first image only includes a portion of the text (curved text), a voice prompt instructs the electronic device to rotate. The voice prompt also indicates a first direction and a first distance of movement (rotation) when rotating the electronic device, or a voice prompt instructs the device to rotate the text, along with a second direction and a second distance of movement (rotation). After rotation, the electronic device uses the camera to capture an image of the text to obtain a second image, which is then displayed on the screen. When the second image includes the entire text, the electronic device reads aloud the entire text.

[0334] As an example, text may be obscured by other occlusions, such as Figure 26 As shown, when the first image only includes a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the steps of moving the electronic device and / or moving the text, including:

[0335] Step N1: When the first image includes only a portion of the text, and another portion of the text in the first image is obscured, the electronic device issues a voice prompt. The voice prompt instructs the electronic device to move and / or the text to move. The voice prompt also instructs the removal of the obstruction that is obscuring the text.

[0336] As an example, the text in the captured image may be obscured by other occlusions such as dirt (ink), for example... Figure 26 As shown, at this time, the electronic device detects whether the content of the text in the first image is incomplete or worn. If the content of the text is incomplete or worn, a voice prompt instructs the electronic device to be moved and / or the text to be moved out of the way. The voice prompt also instructs the removal of any obstruction that is blocking the text.

[0337] Figure 8 This is a flowchart of an auxiliary reading method provided in another embodiment of this application. The auxiliary reading method may include:

[0338] Step S801: The electronic device uses a camera to capture an image of the text, obtains a first image, and displays the first image on the screen;

[0339] Step S802: When the first image only includes a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to move and / or the text to move, and the voice prompt also instructs the electronic device to move in a first direction and a first distance, or the voice prompt also instructs the electronic device to move in a second direction and a second distance, wherein the first direction, the first distance, the second direction, and the second distance are associated with the positional relationship between the edge of the text and the corresponding camera preview stream area;

[0340] Step S803: The electronic device uses a camera to capture an image of the text, obtains a second image, and displays the second image on the screen;

[0341] And / or in step S804, when the second image includes the entire content of the text, the electronic device reads aloud the entire content of the text.

[0342] As an example, the first movement direction, the first movement distance, the second movement direction, and the second movement distance are associated with the positional relationship between the edge of the text and the corresponding camera preview stream area.

[0343] As an example, such as Figure 11 As shown, each edge of the text can be a first edge (corresponding to the top boundary), a second edge (corresponding to the left boundary), a third edge (corresponding to the right boundary), and a fourth edge (corresponding to the bottom boundary). The target distance between the first edge and the corresponding edge of the camera preview stream area is the first target distance d1, the target distance between the second edge and the corresponding edge of the camera preview stream area is the second target distance d2, the target distance between the third edge and the corresponding edge of the camera preview stream area is the third target distance d3, and the target distance between the fourth edge and the corresponding edge of the camera preview stream area is the fourth target distance d4.

[0344] The specific directions of the first and second movements are as follows (1):

[0345] Table (1)

[0346]

[0347]

[0348] As an example, the thresholds in different directions can be the same, different, or customizable. Figures 11-19 As shown, the horizontal threshold (associated with the second and third thresholds) can be defined as 50; the vertical threshold (associated with the first and fourth thresholds) can be defined as 100; and the height and width of the preview stream area are 1280*720.

[0349] It should be noted that, in this embodiment, the voice prompt indicating the first direction and distance of movement (rotation) when rotating the electronic device, and the voice prompt indicating the second direction and distance of movement (rotation) when rotating the text, are relative.

[0350] As an example, when an electronic device is prompted to move to the left, the prompt text moves to the right.

[0351] As an example, prompts need to be given through a prompting module or a voice TTS module.

[0352] As an example, when outputting moving content through a voice TTS module, it is necessary to first determine the specific moving content and then send the specific moving content to the voice TTS module for output.

[0353] As an example, this embodiment also indicates the specific target distance, allowing users to receive more precise prompts and improving the user experience.

[0354] Figure 9 A schematic diagram of the structure of an auxiliary reading device provided in this application is shown. The auxiliary reading device 900 provided in this application includes: a first display module 901, a prompt module 902, a second display module 903, and a voice TTS module 904;

[0355] The first display module 901 in the electronic device is used to capture images of text using a camera, obtain a first image, and display the first image on a screen.

[0356] The prompting module 902 in the electronic device is used to issue a voice prompt when the first image includes only a portion of the text, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved;

[0357] The second display module 903 in the electronic device is used to capture images of the text using a camera, obtain a second image, and display the second image on the screen.

[0358] The electronic device includes a text-to-speech (TTS) module 904, which is used to read aloud the entire content of the text when the second image includes the entire content of the text.

[0359] It should be understood that the electronic equipment described here is embodied in the form of functional modules. The term "module" here can be implemented in software and / or hardware, without specific limitation. For example, a "module" can be a software program, hardware circuit, or a combination of both that implements the above-described functions. The hardware circuit may include application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors) and memory for executing one or more software or firmware programs, integrated logic circuits, and / or other suitable components that support the described functions.

[0360] This application also provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the assisted reading method as described in any one of the first aspects or possible implementations of the first aspect above.

[0361] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the assisted reading method as described in any one of the first aspects or possible implementations thereof.

[0362] This application also provides a chip, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the auxiliary reading method according to any one of the first aspect or any possible implementation of the first aspect.

[0363] Optionally, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform the assisted reading method according to any one of the first aspect or possible implementations of the first aspect.

[0364] The memory can be read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types of dynamic storage devices that can store information and instructions. It can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Alternatively, it can be any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0365] In this embodiment, "and / or" describes the relationship between related texts, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related texts are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0366] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0367] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0368] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0369] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for assisting reading, characterized in that, Applied to an electronic device, the electronic device includes a first camera and a second camera, wherein the field of view of the first camera is larger than the field of view of the second camera, and the assisted reading method includes the following steps: The electronic device captures an image of the text using the first camera to obtain a first image, displays the first image on the screen, and captures an image of the text using the second camera. Determine whether the first image contains the content of the text; If the first image includes the content of the text, then determine whether the image captured by the second camera includes all the content of the text; If the image captured by the second camera includes all the content of the text, then the electronic device extracts all the content of the text based on the image captured by the second camera, and reads aloud all the content of the text. If the image captured by the second camera only includes a portion of the text, then determine whether the first image includes the entire text. If the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved; The electronic device captures an image of the text again using the first camera to obtain a second image, and displays the two images on the screen; If the second image includes all the content of the text, then determine whether the clarity of the text in the second image is greater than the clarity threshold; If it is determined that the clarity of the text in the second image is greater than the clarity threshold, then the entire content of the text is obtained based on the second image, and the electronic device reads aloud the entire content of the text.

2. The assisted reading method as described in claim 1, characterized in that, The step of the electronic device acquiring images of text based on the first camera includes: When the electronic device is in auxiliary mode, it captures images of the text using the first camera.

3. The assisted reading method as described in claim 2, characterized in that, If the trigger condition for the auxiliary mode is detected, the electronic device is determined to be in auxiliary mode.

4. The assisted reading method as described in claim 2, characterized in that, The step of the electronic device capturing images of text based on the first camera when the electronic device is in auxiliary mode includes: When the electronic device is in auxiliary mode, if a trigger condition for image acquisition of text based on the first camera is detected, the electronic device acquires an image of the text based on the first camera. The triggering condition for capturing images of text based on the first camera is different from the triggering condition for camera recording.

5. The assisted reading method as described in claim 1, characterized in that, After determining whether the first image includes the content of the text, the method further includes: If it is determined that the first image does not contain the content of the text, then it is determined whether the image captured by the second camera contains the content of the text. If the image captured by the second camera includes the content of the text, then the step of determining whether the image captured by the second camera includes all the content of the text is performed.

6. The assisted reading method as described in claim 1, characterized in that, The step of determining whether the image captured by the second camera includes all the content of the text includes: The electronic device performs edge detection on the text in the image captured by the second camera; If the edges of the text in the image captured by the second camera are not within the preview stream area of ​​the second camera, then it is determined that the image captured by the second camera only includes a portion of the text.

7. The assisted reading method as described in claim 1, characterized in that, The step of determining whether the first image includes all the content of the text includes: The electronic device determines whether the edge of the text in the first image is within the preview stream area of ​​the first camera; If it is determined that the edges of the text in the first image are not within the preview stream area of ​​the first camera, then it is determined that the first image only includes a portion of the text.

8. The assisted reading method as described in claim 1, characterized in that, After determining whether the sharpness of the text in the second image is greater than the sharpness threshold, the method further includes: If the clarity of the text in the second image is not greater than a clarity threshold, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved; The electronic device then captures an image of the text again using the second camera. Return to the step of determining whether the image captured by the second camera includes all the content of the text.

9. The assisted reading method as described in claim 1, wherein if the first image only includes a portion of the text, the electronic device issues a voice prompt, wherein... The voice prompt instructs the steps of moving the electronic device and / or moving the text, including: If the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to move and / or the text to move, and the voice prompt also instructs the electronic device to move in a first direction and at a first target distance, or the voice prompt also instructs the electronic device to move in a second direction and at a second target distance, wherein the first direction, the first target distance, the second direction, and the second target distance are all associated with a positional relationship, which is the positional relationship between the edge of the text and the edge of the corresponding camera preview stream area.

10. The assisted reading method as described in claim 1, characterized in that, After the step of the electronic device reading aloud the entire text if it is determined that the clarity of the text in the second image is greater than the clarity threshold, the method further includes: The electronic device issues a voice instruction indicating that it is no longer necessary to capture images of the text.

11. The assisted reading method as described in claim 1, characterized in that, If the first image only includes a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the step of moving the electronic device and / or moving the text, including: If the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the electronic device to translate, rotate, or turn, or the voice prompt also instructs the electronic device to move closer to or away from the text; And / or if the first image includes only a portion of the text, the electronic device issues a voice prompt, wherein the voice prompt instructs the text to be translated, rotated, or turned, or the voice prompt also instructs the text to move closer to or further away from the electronic device.

12. The assisted reading method as described in claim 1, characterized in that, The step of the electronic device issuing a voice prompt if the first image only includes a portion of the text, wherein the voice prompt instructs the electronic device to be moved and / or the text to be moved, includes: If the first image includes only a portion of the text, and another portion of the text in the first image is obscured, the electronic device issues a voice prompt, which instructs the electronic device to move and / or the text to move, and also instructs the removal of the obstruction from the text.

13. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the electronic device, cause the electronic device to perform the assisted reading method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the assisted reading method according to any one of claims 1 to 12.

Images