Audio playing method, electronic device and system

Abstract

Provided in the present application are an audio playing method, an electronic device and a system. The method comprises: acquiring first progress information of first content; determining a first playing resource on the basis of the first progress information; playing a first audio on the basis of the first playing resource, and displaying a first picture on the basis of the first playing resource, the first audio being an audio corresponding to a first text; acquiring second progress information of the first content; determining a second playing resource on the basis of the second progress information and the first playing resource; and playing a second audio on the basis of the second playing resource, and displaying a second picture on the basis of the second playing resource, the second audio being an audio corresponding to a second text, wherein the first progress information is different from the second progress information. The present application can synchronously display illustrations related to a text that is currently being read during audio playing, so that the problem of pictures and texts being mismatched during book listening is solved, thereby improving the function usability, and improving the user experience.

Description

Audio playback method, electronic device, and system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on November 17, 2023, with application number 202311540502.9, and priority to the Chinese patent application entitled “A method, electronic device and system for audio playback”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to an audio playback method, electronic device, and system. Background Art

[0003] With the development of artificial intelligence (AI) technology, text-to-speech (TTS) technology, which is used to convert the text content of e-books into audio playback, has been widely used. In the real-time listening scenario, electronic devices can obtain the audio corresponding to the text part of the content to be read in real time and play the audio. During the audio playback process, the electronic device may display an image that does not match the audio, such as the cover image of the e-book, or may not display any image, resulting in a problem of inconsistency between the image and the text, and low functional usability.

[0004] Summary of the Invention

[0005] The present application discloses an audio playback method, electronic device and system, which can synchronously display illustrations related to the text currently being read during audio playback, solve the problem of inconsistency between pictures and text during listening to books, improve functional usability and enhance user experience.

[0006] In a first aspect, an embodiment of the present application provides an audio playback method, applied to a first device, the method comprising: obtaining first progress information of a first content, the first content including text and at least one picture; determining a first playback resource based on the first progress information, the first playback resource including a first text and a first picture in the first content, the first text and the first picture both being related to the first progress information; playing a first audio based on the first playback resource, displaying the first picture based on the first playback resource, the first audio being the audio corresponding to the first text; obtaining second progress information of the first content; determining a second playback resource based on the second progress information and the first playback resource, the second playback resource including a second text and a second picture in the first content, the second text and the second picture both being related to the second progress information; playing a second audio based on the second playback resource, displaying the second picture based on the second playback resource, the second audio being the audio corresponding to the second text; wherein the first progress information and the second progress information are different.

[0007] In the above method, the first device can obtain the first progress information of the first content, determine the first playback resource based on the first progress information, and the first device can play the first audio based on the first playback resource and display the first picture. Subsequently, the first device can update the progress information of reading the first content, and re-determine the updated playback resource (i.e., the second playback resource) based on the updated progress information (i.e., the second progress information), and repeat the process of playing audio and displaying pictures. This method can synchronously display pictures that match the text currently being read during the audio playback process, solve the problem of inconsistency between pictures and text during the listening process, improve functional usability, and enhance user experience.

[0008] In one possible implementation, the method further includes: determining whether the third playback resource includes a picture, the third playback resource being determined based on the acquired third progress information of the first content and the second playback resource; when the third playback resource includes the third picture, switching the second picture displayed on the first device to the third picture, the third picture being different from the second picture; and when the third playback resource does not include the picture, displaying the second picture.

[0009] In the above method, after the first device determines the updated playback resource (i.e., the third playback resource) based on the updated progress information (i.e., the third progress information), it can determine whether the third playback resource includes an image. If the third playback resource includes an image (i.e., the third image), the second image displayed on the display can be switched to the third image; if the third image is not included, the second image displayed on the display can remain unchanged. This method can update the displayed image in real time during audio playback, ensuring that the image and text are consistent, and improving the user experience.

[0010] In a possible implementation, the first playback resource is determined according to a text position corresponding to the first progress information, the second playback resource is determined according to a text position corresponding to the second progress information, and the second playback resource is different from the first playback resource.

[0011] In the above method, the first device determines the playback resources in real time based on the playback progress information during the audio playback process. That is, the playback resources determined under different playback progress are different. By obtaining the playback resources in real time through the playback progress, it can be ensured that the text corresponding to the played audio and the displayed picture match, thereby improving functional availability and user experience.

[0012] In a possible implementation, the first playback resource includes the first picture and a fourth picture, and displaying the first picture according to the first playback resource includes: displaying the first picture when a display position of the first picture is before a display position of the fourth picture.

[0013] In the above method, when the first playback resource includes multiple pictures, they can be displayed on the display screen according to the display positions of the multiple pictures (for example, in order from front to back), so that the text corresponding to the played audio matches the displayed picture, thereby improving the user experience.

[0014] In one possible implementation, the second playback resource includes the second picture and the first picture, the first progress information is before the second progress information, and the display position of the first picture is before the display position of the second picture; displaying the second picture according to the second playback resource includes: when the display time of the first picture is greater than or equal to a preset time, switching the first picture displayed by the first device to the second picture; the method also includes: when the display time of the first picture is less than the preset time, displaying the first picture.

[0015] In the above method, when the second playback resource contains pictures that have been displayed (i.e., the first picture) and pictures that have not been displayed (i.e., the second picture), it can be determined whether to switch to displaying the pictures that have not been displayed based on the display duration of the displayed pictures. In this way, when multiple pictures in the second playback resource are close to each other, it can be ensured that each picture has enough time (for example, 10 seconds to 30 seconds) to be displayed on the display screen, and the display duration of the picture is determined based on the text related to the picture, which further improves the effect of synchronous display.

[0016] In a possible implementation, displaying the first picture according to the first playback resource includes: displaying the first picture when the current playback progress in the first playback resource is located at a first trigger display position of the first picture.

[0017] In the above method, when the first playback resource includes a picture, the first device does not directly display the included picture, but further determines whether the current playback progress is at the trigger display position of the picture. This method accurately controls the moment of picture display, ensures that the displayed picture and text match, further improves the effect of synchronous display of pictures, improves functional availability, and improves user experience.

[0018] In a possible implementation, the method further includes: obtaining the first audio and the first picture according to the first playback resource; and sending the first audio and the first picture to a second device, wherein the first audio is played by the second device and the first picture is displayed by the second device.

[0019] In the above method, when the first device and the second device are connected, after the first device obtains the first audio and the first picture, the first audio and the first picture can be synchronously pushed to the second device (external remote device), so that the second device can synchronously display the picture matching the text currently being read during the audio playback process, thereby expanding the range of applicable devices, further improving functional availability, and enhancing user experience.

[0020] In one possible implementation, the method further includes: when the second playback resource includes the second picture, sending a first message to the second device, the first message indicating an updated displayed picture; receiving a second message sent by the second device, the second message being used to request an updated picture; and sending the second picture to the second device according to the second message, the second picture being used for display on the second device.

[0021] In the above method, for different second devices (external remote devices), the way in which the first device pushes audio and pictures to the second device may be different. Through different interaction methods, it can be ensured that the text corresponding to the audio played by the second device and the displayed picture match, which expands the scope of applicable devices, further improves functional availability, and enhances user experience.

[0022] In one possible implementation, determining the first playback resource based on the first progress information includes: determining the first playback resource based on the first progress information and a sliding window, the sliding window being an interval in the first content determined based on preset conditions, and the preset conditions including at least one of the following: the number of words in the preset text, the number of sentences in the preset text, and the duration corresponding to the preset text.

[0023] In the above method, the determination of the sliding window can directly affect the effect of synchronous display of pictures. Therefore, the size of the sliding window can be accurately controlled by preset conditions, thereby accurately controlling the time of picture display, ensuring that the displayed pictures and texts match, further improving the effect of synchronous display of pictures, improving functional availability, and improving user experience.

[0024] In a possible implementation, the method further includes: sending the first text to a network device; and receiving the first audio sent by the network device, where the first audio is obtained by the network device based on the first text.

[0025] In the above method, the first device can obtain the first audio corresponding to the first text through the network device, which can reduce the processing pressure of the first device. In the case where the processing capacity of the first device is insufficient, it can still ensure that the first device synchronously displays the picture matching the currently read text when playing the audio, thereby expanding the scope of applicable devices, further improving functional availability, and enhancing user experience.

[0026] In a possible implementation, the first content is an e-book, and obtaining the first progress information of the first content includes: when a user operation triggering a book listening function is received, obtaining the first progress information of the first content.

[0027] In the above method, in a real-time book listening scenario, when the user triggers the book listening function, the first device can turn on the book listening function according to the user's reading progress information, which simplifies the triggering operation of the book listening function and eliminates the need for the user to perform additional operations, making it easier for the user to use the book listening function and further improving the function usability and user experience.

[0028] In a second aspect, the present application provides an electronic device comprising a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the audio playback method in any possible implementation of the first aspect.

[0029] In a third aspect, the present application provides an electronic device comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, wherein the computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the audio playback method in any possible implementation of the first aspect.

[0030] In a fourth aspect, the present application provides a computer storage medium storing a computer program. When the computer program is executed by a processor, the audio playback method in any possible implementation of any of the above aspects is implemented.

[0031] In a fifth aspect, the present application provides a computer program product, which, when running on an electronic device, enables the electronic device to execute the audio playback method in any possible implementation of the first aspect above.

[0032] In a sixth aspect, the present application provides an electronic device, the electronic device including a method or apparatus for executing any one of the implementations of the first aspect of the present application. The electronic device is, for example, a chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The following is an introduction to the drawings used in this application.

[0034] FIG1A is a schematic diagram of the architecture of an audio playback system 10 provided by the present application;

[0035] FIG1B is an interactive schematic diagram of an audio playback system 10 provided by the present application;

[0036] FIG2 is a schematic diagram of the hardware structure of an electronic device 100 provided in this application;

[0037] FIG3 is a schematic diagram of a software architecture of an electronic device 100 provided in this application;

[0038] FIG4 is a schematic diagram of the hardware structure of a network device 200 provided in this application;

[0039] FIG5 is a flow chart of an audio playback method provided by the present application;

[0040] FIG6 is a flowchart of another audio playback method provided by the present application;

[0041] FIG7 is a schematic diagram of some application scenarios provided by this application;

[0042] FIG8 is a flowchart of another audio playback method provided by the present application;

[0043] FIG9 is a schematic diagram of some further application scenarios provided by this application;

[0044] FIG10 is a flow chart of another audio playback method provided by the present application;

[0045] FIG11 is a schematic diagram of some further application scenarios provided by this application;

[0046] FIG12 is a flow chart of another audio playback method provided in the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents or, for example, A / B can represent A or B; "and / or" in the text is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0048] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0049] Electronic devices can provide text-to-speech (TTS) functionality. For example, in a real-time book listening scenario, the electronic device can send the text portion of the content to be read to a server to obtain the corresponding audio and play the audio. During the audio playback process, the electronic device can randomly display an image on the display screen, such as the cover image of the e-book obtained by the electronic device, or display no image at all, resulting in a mismatch between the image and the text, low functional usability, and a poor user experience.

[0050] Currently, e-books can be converted and produced in advance. Specifically, the e-book is parsed and the parsed text is converted using TTS technology or manual reading to obtain an audio file (including the timeline information of the audio). The corresponding relationship between the illustration and the audio playback timestamp is determined based on the chapter position corresponding to each parsed illustration, and this information is integrated to obtain the audio book file. When the user listens to the book, the electronic device can obtain the corresponding illustration information based on the timestamp of the audio playback, such as the uniform resource locator (URL) of the illustration, and send the illustration information to the server to obtain the corresponding illustration, which is displayed synchronously on the display screen. However, this method requires a large amount of labor and time cost to be invested in each e-book in advance. If the user wants to enjoy the illustrations synchronously while listening to the book (i.e., to achieve the effect of unified text and graphics), he can only choose from the e-books that have been produced, and the user's choice is limited. For e-books that have not been pre-produced, it is impossible to display the illustrations synchronously when playing the audio. This is not suitable for real-time listening scenarios, cannot meet user needs, and has a poor user experience. In addition, when the electronic device downloads the illustration from the server, it needs to incur traffic costs. Therefore, when there is an abnormal network disconnection such as an abnormal disconnection between the electronic device and the server, a delay will occur, affecting the effect of synchronously displaying the illustration.

[0051] The embodiment of the present application proposes an audio playback method, which is applied to an audio playback system. The audio playback system may include an electronic device and a network device. In a real-time listening scenario, the electronic device can obtain the progress information of the first content. The first content may include text and pictures / illustrations. The initial position of the sliding window in the first content is determined based on the progress information. The sliding window can be used to indicate the position where the electronic device is currently performing TTS processing. The sliding window is a window that can be moved in the first content. For example, the sliding window is moved according to a preset step. The electronic device can control the sliding window to move from the initial position to the first direction (for example, the direction of TTS processing). During the movement of the sliding window, the electronic device can obtain the text in the sliding window in real time, and send the text to the network device to obtain the corresponding audio and play the audio. At the same time, the electronic device can also identify the illustrations in the sliding window and display the identified illustrations on the display screen. This method can synchronously display illustrations related to the text currently being read during the audio playback process, solve the problem of inconsistency between text and images during the listening process, improve functional usability, and improve user experience. Furthermore, this approach can be applied to any pre-produced primary content, eliminating the need for additional labor and time to pre-produce it. This expands the range of primary content available to users, meets their diverse needs, and further enhances the user experience. Furthermore, when listening to books in real time, electronic devices no longer need to be connected to the internet, reducing illustration loading delays, network calls, and data costs, further ensuring the simultaneous display of illustrations.

[0052] The following introduces an audio playback system 10 involved in an embodiment of the present application.

[0053] FIG1A exemplarily shows a schematic diagram of the architecture of an audio playback system 10 .

[0054] As shown in Figure 1A, the audio playback system 10 may include an electronic device 100, a network device 200, and an electronic device 300. The electronic device 100 may communicate with the network device 200 and the electronic device 300 via wired (e.g., universal serial bus (USB), twisted pair, coaxial cable, optical fiber, etc.) and / or wireless (e.g., wireless local area networks (WLAN), Bluetooth, and cellular communication networks, etc.).

[0055] The electronic device 100 may be a mobile phone, a tablet computer, an all-in-one computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as smart TVs and projectors, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), or in-vehicle devices. The description of the electronic device 300 is consistent with that of the electronic device 100. The embodiments of the present application do not impose any particular restrictions on the specific type of the electronic device.

[0056] Network device 200 may include at least one server. In an optional embodiment, any server may be a hardware server. In an optional embodiment, any server may be a cloud server. In an optional embodiment, any server may be a Linux server, a Windows server, or other server device capable of providing simultaneous access to multiple devices. In an optional embodiment, network device 200 may be a server cluster consisting of multiple servers in multiple locations, multiple computer rooms, and multiple servers.

[0057] In one embodiment, the electronic device 100 can communicate with the network device 200 based on a browser / server (B / S) architecture or a client / server (C / S) architecture. In one embodiment, the electronic device 100 can send the acquired text to the network device 200, and the network device 200 can convert the text into audio and return it to the electronic device 100. In some examples, after receiving the audio, the electronic device 100 can play the audio. During the audio playback process, when the electronic device 100 identifies an illustration to be displayed through a sliding window and acquires the illustration, the illustration can be displayed.

[0058] In other examples, after receiving the audio, the electronic device 100 can obtain a corresponding audio stream based on the audio and push the audio stream to the electronic device 300. During the process of the electronic device 300 playing the audio stream, when the electronic device 100 identifies an illustration to be displayed through the sliding window and obtains the illustration, the electronic device 300 can send a first message to the electronic device 300, where the first message is used to instruct the electronic device 300 to update the displayed illustration. After receiving the first message, the electronic device 300 can send a request message to the electronic device 100, receive the illustration returned by the electronic device 100, and display the illustration.

[0059] It can be understood that the number and form of the electronic devices shown in Figure 1A are only examples. In specific implementations, the number may be more or less. For example, the audio playback system 10 may also include other electronic devices. For example, the audio playback system 10 may not include the electronic device 300. This application does not limit this.

[0060] FIG. 1B exemplarily shows an interactive diagram of an audio playback system 10 .

[0061] As shown in Figure 1B, the electronic device 100 in the audio playback system 10 may include a reading application 101 (including a reader 102 and a TTS player 103), a storage module 104, a Bluetooth target terminal 201, and a MediaSession 202. The network device 200 in the audio playback system 10 may include a text-to-audio module 301. The electronic device 300 in the audio playback system 10 may include at least one of a Bluetooth playback device (including a Bluetooth control terminal 401 and a processing module 402) and a car playback device (Hicar terminal 501 and a processing module 502). Among them:

[0062] The reader 102 may include a format parsing module 1021 and a content extraction module 1022 (including a sliding window module 1023). The TTS player 103 may include an audio synthesis module 1031, an audio playback module 1032, and a display module 1033. The storage module 104 may be used to store first content. The format of the first content may be, for example, but not limited to, epub format. The first content may include text and illustrations. The first content may include a starting position, an ending position, and a reading position. The reading position may indicate the position where the user is currently reading the first content. The storage module 104 may send the first content to the format parsing module 1021. The format parsing module 1021 may parse the first content, such as decrypting, decompressing, format parsing, etc., and send the parsed first content to the sliding window module 1023 in the content extraction module 1022. The content extraction module 1022 can determine the initial position of the sliding window in the first content based on the progress information of the first content (i.e., the reading position described above). The sliding window module 1023 can identify the text and illustrations in the sliding window and send the extracted text to the audio synthesis module 1031. When an illustration to be displayed is identified, the illustration is obtained and sent to the display module 1033. After receiving the text, the audio synthesis module 1031 can send the text to the text-to-audio module 301 in the network device 200. The text-to-audio module 301 can generate audio corresponding to the text using TTS technology and send the generated audio to the audio synthesis module 1031. After receiving the audio, the audio synthesis module 1031 can send the audio to the audio playback module 1032, which can play the audio through the speaker. The display module 1033 can display the illustration on the display screen. The sliding window module 1023 can also control the sliding window to move from an initial position to a first direction. During the movement of the sliding window, the sliding window module 1023, the audio synthesis module 1031, the text-to-audio module 301, the audio playback module 1032 and the display module 1033 can repeatedly execute the above process to convert the text in the sliding window into audio in real time and play it, and display the illustrations to be displayed in the sliding window.

[0063] In one embodiment, after the sliding window module 1023 identifies an illustration to be displayed and obtains the illustration, it can send a notification message and the illustration to the audio playback module 1032, indicating an update of the displayed illustration. After receiving the notification message and the illustration, the audio playback module 1032 can send the notification message to the Bluetooth control terminal 401 of the Bluetooth playback device via the Bluetooth target terminal 201. The Bluetooth control terminal 401 can then send the notification message to the processing module 402. The processing module 402 can parse the notification message and, upon determining that an updated illustration is currently required, can send a request message to the Bluetooth target terminal 201 of the electronic device 100 via the Bluetooth control terminal 401, requesting the updated illustration. After receiving the request message, the Bluetooth target terminal 201 can obtain the illustration from the audio playback module 1032 and return the illustration to the Bluetooth control terminal 401. The Bluetooth control terminal 401 can then send the illustration to the processing module 402, which can then display the illustration. After the audio playback module 1032 receives the audio sent by the audio synthesis module 1031, it can send the audio to the Bluetooth target terminal 201. The Bluetooth target terminal 201 can convert the audio into a Bluetooth signal and send it to the Bluetooth control terminal 401 of the Bluetooth playback device. The Bluetooth control terminal 401 can send the Bluetooth signal to the processing module 402. The processing module 402 can parse the Bluetooth signal, obtain the parsed audio, and play the audio through the speaker.

[0064] In one embodiment, after the sliding window module 1023 identifies the illustration to be displayed and obtains the illustration, it can send a notification message and an illustration to the audio playback module 1032. The notification message is used to indicate the update of the displayed illustration. After the audio playback module 1032 receives the notification message and the illustration, it can send the notification message and the illustration to the Hicar end 501 of the vehicle playback device through MediaSession202. The Hicar end 501 can send the notification message and the illustration to the processing module 502. The processing module 502 can parse the notification message, determine that the displayed illustration needs to be updated, and display the received illustration. After the audio playback module 1032 receives the audio sent by the audio synthesis module 1031, it can send the audio to MediaSession202. MediaSession202 can obtain the corresponding audio stream based on the audio and send it to the Hicar end 501 of the vehicle playback device. The Hicar end 501 can send the audio stream to the processing module 502. The processing module 502 can parse the audio stream to obtain the parsed audio and play the audio through the speaker.

[0065] Optionally, the Bluetooth control terminal 401 of the Bluetooth playback device can receive a playback control instruction input by a user, where the playback control instruction is used to control the playback of audio, for example, to pause the playback of audio, to start the playback of audio, to play audio at a multiple of times, etc. The Bluetooth control terminal 401 can send the playback control instruction to the Bluetooth target terminal 201 of the electronic device 100. After receiving the playback control instruction, the Bluetooth target terminal 201 can control the state of converting the audio into a Bluetooth signal, for example, to pause the conversion of the audio into a Bluetooth signal, to start the conversion of the audio into a Bluetooth signal, to play audio at a multiple of times, etc.

[0066] It is understandable that the audio playback system 10 shown in FIG1B may include more or fewer modules. Any module in the audio playback system 10 may be a hardware module or a software module. Any of the above modules may be a separate module, or at least one of the above modules may be integrated together. This application does not limit the name of any of the above modules, and other names may be used to implement the functions corresponding to any of the above modules.

[0067] Next, the structure of an exemplary electronic device provided by an embodiment of the present application is introduced.

[0068] FIG2 exemplarily shows a schematic diagram of the hardware structure of an electronic device 100 .

[0069] As shown in Figure 2, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0070] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0071] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0072] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0073] Processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0074] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In another embodiment, the power management module 141 can also be set in the processor 110. In another embodiment, the power management module 141 and the charging management module 140 can also be set in the same device.

[0075] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0076] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In another embodiment, the antenna can be used in conjunction with a tuning switch.

[0077] The mobile communication module 150 can provide wireless communication solutions for the electronic device 100, including second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G), and sixth generation (6G). The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via the antenna 1. In one embodiment, at least some of the functional modules of the mobile communication module 150 can be located in the processor 110. In another embodiment, at least some of the functional modules of the mobile communication module 150 and at least some of the modules of the processor 110 can be located in the same device.

[0078] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In one embodiment, the modem processor may be an independent device. In another embodiment, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0079] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0080] In one embodiment, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0081] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0082] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In one embodiment, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0083] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0084] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and color. It can also optimize parameters such as exposure and color temperature of the captured scene. In one embodiment, the ISP can be located within camera 193.

[0085] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In one embodiment, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0086] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0087] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0088] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0089] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals.

[0090] The speaker 170A, also called a "horn", is used to convert audio electrical signals into sound signals.

[0091] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals.

[0092] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0093] The headphone jack 170D is used to connect a wired headphone.

[0094] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In one embodiment, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device comprising at least two parallel plates with conductive material. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In one embodiment, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0095] The gyro sensor 180B may be used to determine the motion posture of the electronic device 100. In one embodiment, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180B.

[0096] The air pressure sensor 180C is used to measure air pressure.

[0097] The magnetic sensor 180D includes a Hall sensor, and the electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case.

[0098] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes).

[0099] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In one embodiment, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0100] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100.

[0101] The ambient light sensor 180L is used to sense the brightness of the ambient light.

[0102] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0103] The temperature sensor 180J is used to detect temperature.

[0104] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0105] The bone conduction sensor 180M can acquire vibration signals.

[0106] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0107] The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card.

[0108] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. For example, the software system with a layered architecture can be an Android system, a Harmony operating system (OS), or other software systems. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100.

[0109] FIG3 exemplarily shows a schematic diagram of a software architecture of an electronic device 100 .

[0110] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In one embodiment, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0111] The application layer can include a series of application packages.

[0112] As shown in Figure 3, the application package may include applications such as camera, gallery, music, calendar, short message, call, navigation, Bluetooth, browser, and screen projection.

[0113] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0114] As shown in FIG3 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0115] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0116] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0117] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0118] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0119] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0120] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0121] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0122] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0123] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0124] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0125] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0126] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0127] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0128] A 2D graphics engine is a drawing engine for 2D drawings.

[0129] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0130] The following describes the workflow of the software and hardware of the electronic device 100 in conjunction with a book listening scenario.

[0131] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, timestamp of the touch operation, and other information). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input events from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is an icon control of a reading application, the reading application calls the interface of the application framework layer to start the reading application, and then controls the display driver by calling the kernel layer to display the user interface of the reading application through the display screen 194.

[0132] Next, an exemplary network device 200 provided in an embodiment of the present application is introduced.

[0133] FIG4 exemplarily shows a schematic diagram of the hardware structure of a network device 200 .

[0134] As shown in Figure 4, the network device 200 may include one or more processors 201, a communication interface 202, and a memory 203, wherein the processor 201, the communication interface 202, and the memory 203 can be connected via a bus or other means. The embodiment of the present application takes the connection via bus 204 as an example.

[0135] in:

[0136] The processor 201 may be composed of one or more general-purpose processors, such as a CPU, and may be configured to execute program codes related to the device control method.

[0137] The communication interface 202 may be a wired interface (eg, an Ethernet interface) or a wireless interface (eg, a cellular network interface or a wireless local area network interface) for communicating with other nodes. In the embodiment of the present application, the communication interface 202 may be specifically used to communicate with the electronic device 100.

[0138] The memory 203 may include a volatile memory, such as a RAM; the memory may also include a non-volatile memory, such as a ROM, a flash memory, an HDD, or a solid-state drive (SSD). The memory 203 may also include a combination of the above-mentioned types of memories. The memory 203 may be used to store a set of program codes so that the processor 201 calls the program codes stored in the memory 203 to implement the implementation method of the embodiment of the present application on the server. In the embodiment of the present application, the memory 203 may also be a storage array, etc.

[0139] In one embodiment, the network device 200 may include multiple servers, such as a web server, a background server, a download server, etc. The hardware structures of these multiple servers can refer to the hardware structure of the network device 200 shown in Figure 4.

[0140] It should be noted that the network device 200 shown in FIG4 is an example implementation of an embodiment of the present application. In actual applications, the network device 200 may also include more or fewer components, which is not limited here.

[0141] Next, the audio playback method provided by the embodiment of the present application is introduced.

[0142] Please refer to Figure 5, which is a flowchart of an audio playback method provided by an embodiment of the present application. The method can be applied to the audio playback system 10 shown in Figure 1A. The method can be applied to the audio playback system 10 shown in Figure 1B. The method can be applied to the electronic device 100 shown in Figure 2. The method can be applied to the electronic device 100 shown in Figure 3. The method can include, but is not limited to, the following steps:

[0143] S101: The electronic device 100 obtains progress information of a first content.

[0144] In one embodiment, the first content may include text and pictures / illustrations, and the first content may be, for example but not limited to, a document file, an electronic book, and the like.

[0145] In one embodiment, the first content may include multiple positions, including a start position, an end position, and a reading position. The reading position may indicate the position where the user is currently reading the first content, ie, the position corresponding to the progress information.

[0146] In one embodiment, the position in the first content may be represented by, but is not limited to, at least one of a percentage, a position number, a chapter, a page number, and a line number. For example, when the position is represented by a percentage, 0% may represent the starting position of the first content, 100% may represent the ending position of the first content, and x% may represent the reading position, where x is greater than or equal to 0 and less than or equal to 100. In one embodiment, any two or more positions in the first content may be classified as front or back. For example, the smaller the percentage, the closer the position is to the front, e.g., the starting position is before the ending position.

[0147] In some examples, the text in the first content includes the characters "A lot of lotus flowers have already bloomed. The lotus leaves are crowded together, like big green discs, with white lotus flowers emerging between these big discs. Some have only two or three petals unfolded. Some have all their petals unfolded, revealing tender yellow lotus pods. Some are still buds, looking so full that they seem about to burst.", where the starting characters "Lotus flowers have already" of the text correspond to a position of 20% in the first content, the middle characters "Some have just unfolded" of the text correspond to a position of 25% in the first content, and the ending characters "Burst like" of the text correspond to a position of 30% in the first content.

[0148] In one embodiment, the illustrations in the first content may be embedded in the text. In some examples, as shown in FIG6 below, the first content may include multiple lines / paragraphs of text and multiple illustrations, and the multiple lines / paragraphs of text are interspersed with one or more illustrations. The illustrations in the first content may be presented in the form of illustration identifiers, such as image embedding elements ( )Label.

[0149] In one embodiment, the electronic device 100 may display a first interface, which may be an interface of a first application, such as an interface for reading first content. The electronic device 100 may then receive a user operation (e.g., a touch operation) on a function control in the first interface, and in response to the user operation, activate the audiobook function of the first application and obtain progress information of the first content.

[0150] S102: The electronic device 100 determines the position of the sliding window according to the progress information.

[0151] In one embodiment, the sliding window may be used to indicate the current position of the electronic device performing TTS processing.

[0152] In one implementation, the electronic device 100 may use the position corresponding to the progress information as the position of the sliding window in the first content.

[0153] In one embodiment, the sliding window may include two edges perpendicular to the first direction (e.g., the direction of TTS processing), wherein the edge at the front position may be referred to as the starting edge, and the edge at the back position may be referred to as the ending edge. In some examples, the electronic device 100 may use the position corresponding to the progress information as the starting edge of the sliding window. It is understandable that at this time, the sliding window only includes unread text. In other examples, the electronic device 100 may use the position corresponding to the progress information as the ending edge of the sliding window. It is understandable that at this time, the sliding window only includes read text.

[0154] Without limitation, in another embodiment, the electronic device 100 may also use the position corresponding to the progress information as any other position in the sliding window other than the starting edge and the ending edge. In some examples, the electronic device 100 may use the position of the text that has been read within a preset time period before the position corresponding to the progress information as the starting edge of the sliding window. It is understandable that the sliding window includes both read text and unread text. This application does not limit the positional relationship between the starting edge of the sliding window and the position corresponding to the current playback progress.

[0155] In one embodiment, the size of the sliding window can be pre-set. For example, the size of the sliding window can be related to the number of lines / paragraphs, words, sentences, duration of the text to be included, and optionally the number of illustrations / illustration identifiers.

[0156] S103: The electronic device 100 obtains the audio and illustration corresponding to the text in the sliding window.

[0157] In one embodiment, after determining the position of the sliding window, the electronic device 100 may identify part or all of the content in the sliding window, extract the first text in the sliding window, and obtain the first audio corresponding to the first text. It is understandable that the first text may be part or all of the content in the sliding window.

[0158] In one embodiment, after obtaining the first text, the electronic device 100 may send the first text to the network device 200. The network device 200 may process the first text using the TTS technology to obtain the first audio corresponding to the first text and return it to the electronic device 100. Without limitation, in another embodiment, the electronic device 100 may directly process the first text using the TTS technology to obtain the first audio.

[0159] In one embodiment, the electronic device 100 can determine whether the sliding window contains illustrations. When the sliding window contains illustrations, the electronic device 100 can obtain a first illustration. The first illustration can be an illustration in the sliding window that has not been displayed on the display screen in the listening scene and is at the front (located at the front position). For specific instructions, please refer to S202-S204 of Figure 6 below, which will not be repeated here.

[0160] S104: The electronic device 100 plays audio and displays illustrations.

[0161] In one embodiment, after the electronic device 100 obtains the first audio and the first illustration, it can play the first audio through the speaker and display the first illustration on the display screen. It can be understood that the electronic device 100 updates and displays the first illustration when playing the first audio.

[0162] S105: The electronic device 100 updates progress information.

[0163] In one embodiment, while the electronic device 100 is playing audio and displaying illustrations, the electronic device 100 can obtain the latest progress information, such as obtaining the position of the text corresponding to the audio currently played by the electronic device 100 in the first content, that is, updating the progress of the current reading of the first content (reading position).

[0164] In one embodiment, the electronic device 100 obtains the latest progress information and re-executes S102-S105. At this time, the updated progress information is different from the previous one, the position of the sliding window corresponding to the updated progress information (which may be simply referred to as the updated sliding window) is different from the previous one, and the text contained in the updated sliding window is also different from the previous one.

[0165] In the method shown in Figure 5, in a real-time listening scenario, the electronic device can obtain the progress information of the first content and determine the position of the sliding window in the first content based on the progress information. The electronic device can obtain the text in the sliding window, and send the text to the network device to obtain the corresponding audio, and play the audio. At the same time, the electronic device can also identify the illustrations in the sliding window and display the identified illustrations on the display screen. Subsequently, the electronic device can update the progress information of reading the first content, and re-determine the position of the sliding window based on the updated progress information, and repeat the process of playing audio and displaying illustrations. This method can synchronously display illustrations related to the text currently being read during the audio playback process, solve the problem of inconsistency between text and images during the listening process, improve functional usability, and enhance user experience.

[0166] Please refer to Figure 6, which is a flow chart of another audio playback method provided in an embodiment of the present application. The method can be applied to the audio playback system 10 shown in Figure 1A. The method can be applied to the audio playback system 10 shown in Figure 1B. The method can be applied to the electronic device 100 shown in Figure 2. The method can be applied to the electronic device 100 shown in Figure 3. The method can include, but is not limited to, the following steps:

[0167] S201: The electronic device 100 obtains the audio corresponding to the text in the sliding window.

[0168] In one embodiment, before S201 , S101 - S102 in FIG. 5 may also be included.

[0169] In one embodiment, the description of S201 is similar to the specific description of the electronic device 100 obtaining the audio corresponding to the text in the sliding window in S103 of Figure 5. Please refer to the specific description in S103 of Figure 5 and no further details will be given.

[0170] S202: The electronic device 100 determines whether the sliding window contains an illustration.

[0171] In one embodiment, the electronic device 100 may identify the content in the sliding window and determine whether the sliding window contains an illustration. In one embodiment, when an illustration identifier is identified, the electronic device 100 may determine that the sliding window contains an illustration, and in this case, the electronic device 100 may execute S203; when no illustration identifier is identified, the electronic device 100 may determine that the sliding window does not contain an illustration, and in this case, the electronic device 100 may execute S206.

[0172] S203: The electronic device 100 determines whether there is an undisplayed illustration in the sliding window.

[0173] In one embodiment, the electronic device 100 can determine whether the illustrations in the sliding window have been displayed on the display screen in the listening scene (for example, the location of the picture 611 in (B) of Figure 7 below). When the sliding window includes illustrations that have not been displayed on the display screen, the electronic device 100 can execute S203-S204-S205; when the sliding window includes illustrations that have been displayed on the display screen, the electronic device 100 can execute S206.

[0174] S204: The electronic device 100 obtains a first illustration.

[0175] In one embodiment, the first illustration may be an illustration that has not been displayed in the sliding window and is located at the front (at the frontmost position). The frontmost position may be a position farthest from the first direction. The first direction may be, for example, the direction of TTS processing, which may be understood as the direction in which the first content extends.

[0176] In one embodiment, when the number of undisplayed illustrations in the sliding window is greater than one, the electronic device 100 may select the first illustration corresponding to the frontmost position based on the multiple positions corresponding to the multiple undisplayed illustrations. For a specific example, see (A) in FIG9 below. When the number of undisplayed illustrations in the sliding window is equal to one, the electronic device 100 may directly obtain the illustration (the first illustration). For a specific example, see FIG7 below.

[0177] In one embodiment, the electronic device 100 may obtain a first illustration identifier, and parse the first illustration identifier to obtain a parsed first illustration.

[0178] The order of S201 and S202-S204 is not limited.

[0179] S205: The electronic device 100 plays audio and displays a first illustration.

[0180] In one embodiment, after the electronic device 100 obtains the audio and the first illustration, it can play the audio through the speaker and display the first illustration on the display screen. It can be understood that the electronic device 100 updates the display of the first illustration while playing the audio.

[0181] S206: The electronic device 100 plays the audio and keeps the illustration displayed on the display unchanged.

[0182] In one embodiment, after the electronic device 100 obtains the audio, it can play the audio through the speaker and keep the illustration currently displayed on the display unchanged. Without limitation, when the display screen of the electronic device 100 does not display the illustration, the electronic device 100 can keep the display screen from displaying the illustration.

[0183] S207: The electronic device 100 moves the sliding window according to a first preset step.

[0184] In one embodiment, the first preset step may be pre-set. For example, the first preset step may be related to the size of the sliding window, the number of lines / paragraphs, words, or sentences of the TTS-processed text within the sliding window, etc. For example, every time the number of TTS-processed text sentences within the sliding window increases by N sentences, where N is a positive integer, the electronic device 100 moves the sliding window once.

[0185] In one embodiment, the electronic device 100 can control the sliding window to move from an initial position (for example, the position of the sliding window determined for the first time based on progress information) to a first direction. The first direction is, for example, the direction of TTS processing, which can be understood as the direction of extension of the first content.

[0186] In one embodiment, the electronic device 100 can move the sliding window from the current position to the first direction according to the first preset step and re-execute S201-S207. At this time, the position of the sliding window after the move is different from before, and the text contained in the sliding window after the move is also different from before.

[0187] Not limited to the embodiment shown in FIG6 , in another embodiment, after S201 , electronic device 100 may obtain an audio stream based on the audio and send the audio stream to electronic device 300 . After receiving the audio stream, electronic device 300 may play the audio stream through a speaker. In some examples, after S204 , electronic device 100 may send a first message to electronic device 300 , indicating an update of the displayed artwork. The first message may be, for example, AVRC_MEDIA_ATTR_ID_COVER_ART. After receiving the first message, electronic device 300 may parse the first message and determine that the displayed artwork needs to be updated. Next, electronic device 300 may send a first request message to electronic device 100 , requesting the artwork to be updated. Electronic device 100 may send the first artwork to electronic device 300 , and electronic device 300 may display the first artwork on its display. In other examples, after S204 , electronic device 100 may send a second message to electronic device 300 , including the first artwork. The second message indicates an update of the displayed artwork. The second message may be, for example, MediaDescription.setIconBitmap . After receiving the second message, the electronic device 300 can parse the second message, determine that the displayed illustration needs to be updated, and then display the received first illustration on the display screen. In this way, the electronic device 100 can simultaneously push the illustrations related to the text currently being read to the electronic device 300 (external remote device), synchronously update the displayed illustrations, and improve the user experience.

[0188] In the method shown in FIG6 , in a real-time listening scenario, the electronic device can obtain progress information of the first content and determine the initial position of a sliding window in the first content based on the progress information. The sliding window can be used to indicate the position where the electronic device is currently performing TTS processing. The electronic device can control the sliding window to move from the initial position to a first direction (e.g., the direction of TTS processing). During the movement of the sliding window, the electronic device can obtain the text in the sliding window in real time and send the text to a network device to obtain the corresponding audio and play the audio. At the same time, the electronic device can also identify illustrations in the sliding window and display the identified illustrations on the display screen in a display order (e.g., in order from front to back). This method can synchronously display illustrations related to the text currently being read while the audio is playing, solving the problem of inconsistent text and image during listening, improving functional usability, and enhancing the user experience. Moreover, this method can be applied to any first content that has not been pre-produced, eliminating the need for additional labor costs and time costs to pre-produce the first content, expanding the range of first content that users can choose, meeting the diverse needs of users, and further improving the user experience. In addition, when listening to books in real time, electronic devices do not need to be connected to the Internet in real time, which reduces the delay in loading illustrations, reduces the number of network calls, saves traffic costs, and further ensures the effect of synchronous display of illustrations.

[0189] Next, the application scenario involved and the user interface diagram in this scenario will be introduced in conjunction with the implementation shown in FIG6 .

[0190] Scenario 1: When the electronic device 100 recognizes that the sliding window includes an illustration and the illustration has not been displayed on the display screen, the electronic device 100 can display the illustration on the display screen. For specific examples, see Figures 7 (A) to 7 (F).

[0191] As shown in (A) of FIG7 , assuming that the progress information in the first content corresponds to position 1, the electronic device 100 can use position 1 as the position of the sliding window, that is, the sliding window in the first content is located at position 1, and the sliding window includes illustration 1. The electronic device 100 can execute S201 of FIG6 , that is, the electronic device 100 can obtain audio 1 corresponding to text 1 in the sliding window. At the same time, the electronic device 100 can execute S202 of FIG6 , and recognize that the sliding window includes an identifier of illustration 1, and then execute S203 of FIG6 , and determine that illustration 1 has not been displayed on the display screen. Therefore, the electronic device 100 can execute S204-S205 of FIG6 , that is, obtain illustration 1, play audio 1 and display illustration 1 on the display screen. For a specific example, see the user interface 610 shown in (B) of FIG7 .

[0192] As shown in FIG7(B), user interface 610 may be the audiobook interface of the first application. User interface 610 may include an image 611, a preview image 612A, a preview image 612B, text content 613, a progress bar 614, and a time 615. Image 611 may correspond to illustration 1 in FIG7(A). The location of image 611 may be the display position of the illustration in the audiobook scenario. This display position remains unchanged, but the displayed content (i.e., the image) may change. Preview image 612A may be a preview of illustration 1 in FIG7(A). Preview image 612A is selected, indicating that illustration 1 is currently being displayed. Preview image 612B may be a preview of illustration 2 in FIG7(A). Text content 613 may include text 6131, text 6132, and a text box 6133. Text 6131 (including the characters "There are many things worth trying in life") may correspond to text 1 in FIG7(A). Text 6131 (corresponding to text 1) is in the read / played state. Text 6132 (including the characters "Go try practicing yoga") can correspond to text 2 in FIG. 7 (A). Text 6132 (corresponding to text 2) is framed by text box 6133, which can indicate that text 6132 is the text corresponding to audio 1 currently being played by electronic device 100, and text 6132 is related to image 611. Progress bar 614 and time 615 (including the characters "01:00") can indicate the progress of the audio currently being played by electronic device 100, i.e., electronic device 100 has played 1 minute of audio. In one embodiment, electronic device 100 can use the position of the text that has been read within a preset time period before the current playback progress as the starting edge of the sliding window based on the current playback progress (e.g., time 615 or text 6132). For example, if the time corresponding to the current playback progress is 01:00 (corresponding to text 6132), electronic device 100 can use the position of the text that has been read within 3 seconds before 01:00 (e.g., text 6131) as the starting edge of the sliding window, i.e., the position of the read text 1 as the starting edge of the sliding window.

[0193] In one embodiment, after (A) and (B) of FIG. 7 , the electronic device 100 may execute S207 of FIG. 6 , i.e., move the sliding window in a first direction (e.g., the direction of TTS processing) according to step 1. Assume that the sliding window after the movement is at position 2, as shown in (C) of FIG. 7 , and the sliding window does not include an illustration at this time. The electronic device 100 may execute S201 of FIG. 6 , i.e., the electronic device 100 obtains the audio 2 corresponding to the text 2 in the sliding window. At the same time, the electronic device 100 may execute S202 of FIG. 6 and obtain a judgment result that the illustration is not included. At this time, the electronic device 100 may execute S206 of FIG. 6 , i.e., play the audio 2 and keep the illustration 1 displayed on the display unchanged. At this time, the electronic device 100 may display the user interface 620 shown in (D) of FIG. 7 .

[0194] As shown in FIG7(D), user interface 620 is similar to user interface 610 shown in FIG7(B). For example, picture 611 displayed in user interface 620 is the same as picture 611 displayed in user interface 610, i.e., picture 611 (corresponding to FIG1) remains displayed. The difference lies in that text content 623 in user interface 620 is different from text content 613 in user interface 610, and progress bar 624 and time 625 in user interface 620 are different from progress bar 614 and time 615 in user interface 610. Text content 623 may include text 6231, text 6232, and text box 6233. Text 6231 (including the characters "Practicing yoga must persist") may correspond to text 3 in FIG7(C). Text 6231 (corresponding to text 3) is in a read / played state. Text 6232 (including the characters "You can start two or three times a week") may correspond to text 4 in FIG. 7 (C). Text 6232 (corresponding to text 3) is framed by text box 6233, indicating that text 6232 corresponds to audio 2 currently being played by electronic device 100. Text 6232 is associated with image 611. Progress bar 624 and time 625 (including the characters "02:13") may indicate the progress of the audio currently being played by electronic device 100, i.e., electronic device 100 has played the audio for 2 minutes and 13 seconds.

[0195] In one embodiment, after FIG7 (C) and FIG7 (D), the electronic device 100 may execute S207 of FIG6, i.e., move the sliding window again in the first direction (e.g., the direction of TTS processing) according to step 1. Assume that the sliding window after the movement is at position 3, as shown in FIG7 (E), and the sliding window now includes illustration 2. The electronic device 100 may execute S201 of FIG6, i.e., obtain audio 3 corresponding to text 3 in the sliding window. At the same time, the electronic device may execute S202 of FIG6 and recognize that the sliding window includes an identifier of illustration 2. Then, the electronic device 100 may execute S203 of FIG6 and determine that illustration 2 has not been displayed on the display screen. Therefore, the electronic device 100 may execute S204-S205 of FIG6, i.e., obtain illustration 2, play audio 3, and display illustration 2 on the display screen. For a specific example, see user interface 630 shown in FIG7 (F).

[0196] As shown in FIG7(F), user interface 630 may include an image 631, text content 633, a progress bar 634, and a time 635. Image 631 may correspond to illustration 2 in FIG7(E), and preview image 612B is selected, indicating that illustration 2 is currently being displayed. Text content 633 may include text 6331, text 6332, and a text box 6333. Text 6331 (including the characters "Go try to get a pet dog") may correspond to text 5 in FIG7(E), and text 6331 (corresponding to text 5) is in a read / played state. Text 6332 (including the characters "Dogs are humans' most loyal companions") may correspond to text 6 in FIG7(E). Text 6332 (corresponding to text 6) is selected by text box 6333, indicating that text 6332 corresponds to text corresponding to audio 3 currently being played on electronic device 100, and that text 6332 is associated with image 631. The progress bar 634 and the time 635 (including the characters “06:50”) may indicate the progress of the current audio playback by the electronic device 100 , that is, the electronic device 100 has played the audio for 6 minutes and 50 seconds.

[0197] Without limitation, in another embodiment, when the electronic device 100 receives a user operation acting on the progress bar of the audio, the progress of the audio played by the electronic device 100 will change, the text corresponding to the played audio will change (the displayed text changes accordingly), and the displayed illustrations may change. In some examples, assuming that the electronic device 100 displays the user interface 610 shown in (B) of Figure 7, the electronic device 100 receives a user operation acting on the progress bar 614 in the user interface 610 (for example, the user operation is a touch operation). In response to the user operation, the electronic device 100 can adjust the progress of the audio playback. Assuming that the electronic device 100 displays the user interface 630 shown in (F) of Figure 7 after the progress is adjusted, the progress of the audio playback, the displayed text, and the illustrations in the user interface 630 are all different from those in the user interface 610. It is understandable that when the electronic device 100 does not receive a user operation on the progress bar of the audio, the electronic device 100 may move the sliding window from position 1 shown in FIG. 7 (A) to position 2 shown in FIG. 7 (C) and position 3 shown in FIG. 7 (E) in sequence, and the display screen also sequentially displays user interface 610, user interface 620 shown in FIG. 7 (D), and user interface 630. When the electronic device 100 receives a user operation on the progress bar of the audio, the electronic device 100 may directly move the sliding window from position 1 to position 3 (no longer to position 2), and the display screen does not display user interface 620, but directly switches from user interface 610 to user interface 630.

[0198] Scenario 2: When the electronic device 100 recognizes that the sliding window includes multiple illustrations and these multiple illustrations have not been displayed on the display screen, the electronic device 100 can display the illustration at the front position on the display screen. For specific examples, please refer to (A) and (B) of Figure 9 below.

[0199] As shown in FIG9 (A), the sliding window in the first content is at position 4 (assuming that the starting edge of the sliding window corresponds to the current playback progress of the electronic device 100). At this time, the sliding window includes illustrations 3 and 4. The electronic device 100 can execute S201 of FIG6, that is, the electronic device 100 can obtain audio 4 corresponding to text 4 in the sliding window. At the same time, the electronic device 100 can execute S202 of FIG6 and recognize that the sliding window includes the identifier of illustration 3 and the identifier of illustration 4. Then, the electronic device 100 executes S203 of FIG6 and determines that neither illustration 3 nor illustration 4 has been displayed on the display screen. Therefore, the electronic device 100 can execute S204-S205 of FIG6. Since the position of illustration 3 is before the position of illustration 4, the electronic device 100 can obtain illustration 3, play audio 4, and display illustration 3 on the display screen. For a specific example, see user interface 910 shown in FIG9 (B).

[0200] As shown in FIG9(B), user interface 910 may be an audiobook interface for the first application. User interface 910 may include image 911, preview image 912A, preview image 912B, text content 913, progress bar 914, and time 915. Image 911 may correspond to illustration 3 in FIG9(A). The location of image 911 may be the display location of the illustration in the audiobook scenario. The display location remains unchanged, but the displayed content (i.e., image) may change. Preview image 912A may be a preview of illustration 3 in FIG9(A). Preview image 912A is in a selected state, indicating that illustration 3 is currently being displayed. Preview image 912B may be a preview of illustration 4 in FIG9(A). The text content 913 may include text 9131 and a text box 9132. The text content 913 is related to the picture 911. The text 9131 (including the characters "cultivate one's character and communication skills") may correspond to the text 7 in (A) of FIG9 . The text 9131 (corresponding to the text 7) is framed by the text box 9132, which may indicate that the text 9131 is the text corresponding to the audio 4 currently being played by the electronic device 100. The text 9131 is related to the picture 911. The progress bar 914 and the time 915 (including the characters "02:55") may indicate the progress of the audio currently being played by the electronic device 100, i.e., the electronic device 100 has played 2 minutes and 55 seconds of audio. It can be understood that the current playback progress of the electronic device 100 corresponds to the starting edge of the sliding window.

[0201] Please refer to Figure 8, which is a flowchart of another audio playback method provided by an embodiment of the present application. The method can be applied to the audio playback system 10 shown in Figure 1A. The method can be applied to the audio playback system 10 shown in Figure 1B. The method can be applied to the electronic device 100 shown in Figure 2. The method can be applied to the electronic device 100 shown in Figure 3. The method can include but is not limited to the following steps:

[0202] S301: The electronic device 100 obtains the audio corresponding to the text in the sliding window.

[0203] In one embodiment, before S301 , S101 - S102 in FIG. 5 may also be included.

[0204] In one embodiment, the description of S301 is similar to the description of the electronic device 100 obtaining the audio corresponding to the text in the sliding window in S103 of Figure 5. Please refer to the specific description in S103 of Figure 5 and will not be repeated here.

[0205] S302: The electronic device 100 determines whether the sliding window contains an illustration.

[0206] In one embodiment, the description of S302 is similar to the description of S202 in Figure 6. Please refer to the specific description of S202 in Figure 6. When it is determined that the sliding window contains illustrations, the electronic device 100 can execute S303; when it is determined that the sliding window does not contain illustrations, the electronic device 100 can execute S306.

[0207] S303: The electronic device 100 determines whether there is an undisplayed illustration in the sliding window.

[0208] In one embodiment, the description of S303 is similar to the description of S203 in Figure 6. Please refer to the specific description of S203 in Figure 6. When there are illustrations that have not been displayed, the electronic device 100 can execute S304; when there are no illustrations that have not been displayed (that is, there are illustrations that have been displayed), the electronic device 100 can execute S306.

[0209] S304: The electronic device 100 determines whether there is an illustration that has been displayed in the sliding window.

[0210] In one embodiment, when there are illustrations that have been displayed, the electronic device 100 can determine that the sliding window includes illustrations that have been displayed on the display screen and illustrations that have not been displayed on the display screen, and the electronic device 100 can execute S305; when there are no illustrations that have been displayed, the electronic device 100 can determine that the sliding window only includes illustrations that have been displayed on the display screen, and the electronic device 100 can execute S307-S308.

[0211] S305: The electronic device 100 determines whether the first display duration is less than a first preset duration.

[0212] In one embodiment, after the electronic device 100 determines that the sliding window includes illustrations that have been displayed on the display screen and illustrations that have not been displayed on the display screen, the electronic device 100 can obtain the first display duration of the second illustration, and the second illustration can be the illustration currently displayed on the display screen of the electronic device 100.

[0213] In one embodiment, the electronic device 100 can determine whether the first display time is less than the first preset time. When the first display time is less than the first preset time, the electronic device 100 can execute S306; when the first display time is greater than or equal to the first preset time, the electronic device 100 can execute S307-S308.

[0214] In one embodiment, the first preset duration may be determined by the electronic device 100 based on the number of lines / paragraphs, words, or sentences of the text related to the second illustration in the sliding window.

[0215] S306: The electronic device 100 plays the audio and keeps the illustration displayed on the display unchanged.

[0216] In one embodiment, after the electronic device 100 obtains the audio, it can play the audio through the speaker and keep the illustration currently displayed on the display unchanged. Without limitation, when the display screen of the electronic device 100 does not display the illustration, the electronic device 100 can keep the display screen from displaying the illustration.

[0217] S307: The electronic device 100 obtains a third illustration.

[0218] In one embodiment, the description of the third illustration is similar to the description of the first illustration in S204 of FIG. 6 , and reference may be made to the specific description of the first illustration in S204 of FIG. 6 , which will not be repeated here.

[0219] The order of S301 and S302-S305 is not limited.

[0220] S308: The electronic device 100 plays audio and displays a third illustration.

[0221] In one embodiment, after the electronic device 100 obtains the audio and the third illustration, it can play the audio through the speaker and display the third illustration on the display screen. It can be understood that the electronic device 100 updates and displays the third illustration while playing the audio.

[0222] In one embodiment, after the electronic device 100 executes S305, it executes S307. At this time, the electronic device 100 may switch the displayed second illustration to a third illustration.

[0223] S309: The electronic device 100 moves the sliding window according to a second preset step.

[0224] In one embodiment, the description of S309 is similar to the description of S207 in Figure 6. Please refer to the specific description of S207 in Figure 6. The electronic device 100 can move the sliding window according to the second preset step and re-execute S301-S308. At this time, the position of the sliding window after moving is different from before, and the text contained in the sliding window after moving is also different from before.

[0225] In the method shown in Figure 8, when the sliding window contains illustrations that have been displayed and illustrations that have not been displayed, it can be determined whether to switch to displaying the illustrations that have not been displayed based on the display duration of the displayed illustrations. In this way, when multiple illustrations in the sliding window are close to each other, it can be ensured that each illustration has enough time (for example, 10 seconds to 30 seconds) to be displayed on the display screen, and the display duration of the illustration is determined based on the text related to the illustration, which further improves the effect of synchronous display.

[0226] Next, the application scenario involved and the user interface diagram in this scenario will be introduced in conjunction with the implementation shown in FIG8 .

[0227] Scenario 3: When the electronic device 100 recognizes that the sliding window includes multiple illustrations, and these multiple illustrations include illustrations that have been displayed on the display screen and illustrations that have not been displayed on the display screen, the electronic device 100 can determine whether to switch the display of the illustrations based on the display duration of the displayed illustrations. For specific examples, please refer to Figure 9 (C) to Figure 9 (F).

[0228] For example, after (A) and (B) of FIG. 9 shown in scenario 2, the electronic device 100 may execute S309 of FIG. 8 , that is, the electronic device 100 may move the sliding window in the first direction (e.g., the direction of TTS processing) according to step 2. Assume that the sliding window after the movement is at position 5, as shown in (C) of FIG. 9 , at this time, the sliding window includes illustrations 3 and 4. The electronic device 100 may execute S301 of FIG. 8 , that is, obtain audio 5 corresponding to text 5 in the sliding window. At the same time, the electronic device 100 can execute S302 of Figure 8 and recognize that the sliding window includes the identifier of illustration 3 and the identifier of illustration 4, then execute S303 of Figure 8 and determine that illustration 4 has not been displayed on the display screen, then execute S304 of Figure 8 and determine that illustration 3 has been displayed on the display screen. At this time, the electronic device 100 can execute S305 of Figure 8 and determine that the display time 1 of illustration 3 is less than the preset time 1. Therefore, the electronic device 100 can execute S306 of Figure 8, that is, play audio 5 and keep illustration 3 displayed on the display screen unchanged. At this time, the electronic device 100 can display the user interface 920 shown in (D) of Figure 9.

[0229] As shown in (D) of FIG9 , user interface 920 is similar to user interface 910 shown in (A) of FIG9 . For example, picture 911 displayed in user interface 920 is the same as picture 911 displayed in user interface 910, that is, picture 911 is kept displayed (corresponding to FIG3 ). The difference is that user interface 920 may include text content 923, progress bar 924, and time 925. Text content 923 may include text 9231, text 9232, and text box 9233. Text 9231 (including the characters “known as “a fashionable and elegant sport”) may correspond to FIG9 (C) 8, text 9231 (corresponding to text 8) is selected by text box 9233, which can indicate that text 9231 is the text corresponding to audio 5 currently being played by electronic device 100, and text 9231 is related to picture 911. Text 9232 (including the characters "Dance is a performing art") can correspond to text 9 in (C) of Figure 9, and text 9232 is in an unread / unplayed state. Progress bar 924 and time 925 (including the characters "02:58") can indicate the progress of the audio currently being played by electronic device 100, that is, electronic device 100 has played 2 minutes and 58 seconds of audio.

[0230] In one embodiment, after FIG9 (C) and FIG9 (D), the electronic device 100 may execute S309 of FIG8, that is, the electronic device 100 may again move the sliding window in the first direction according to step 2. Assume that the sliding window after the movement is at position 6, as shown in FIG9 (E), and the sliding window now includes illustrations 3 and 4. The electronic device 100 may execute S301 of FIG8, that is, obtain audio 6 corresponding to text 6 in the sliding window. At the same time, the electronic device 100 can execute S302-S304 of Figure 8 again, and determine that illustration 3 has been displayed on the display screen, and illustration 4 has not been displayed on the display screen. At this time, the electronic device 100 can execute S305 of Figure 8, and determine that the display time 2 of illustration 3 (assuming that the display time 2 is two seconds longer than the display time 1) is greater than or equal to the preset time 1. Therefore, the electronic device 100 can execute S307-S308 of Figure 8, that is, obtain illustration 4, play audio 6 and switch illustration 3 displayed on the display screen to illustration 4. At this time, the electronic device 100 can display the user interface 930 shown in (F) of Figure 9.

[0231] As shown in FIG9(F), user interface 930 may include an image 931, text content 933, a progress bar 934, and a time 935. Image 931 may correspond to illustration 4 in FIG9(E), and preview image 912B is selected, indicating that illustration 4 is currently being displayed. Text 9231 (corresponding to text 8) in text content 933 has been read / played, while text 9232 (corresponding to text 9) is framed by text box 9331, indicating that text 9232 corresponds to audio 6 currently being played on electronic device 100. Text 9232 is associated with image 931. Progress bar 934 and time 935 (including the characters "03:00") may indicate the progress of the audio currently being played on electronic device 100, i.e., electronic device 100 has played three minutes of audio.

[0232] It can be understood that in Figure 9 (C) and Figure 9 (D), the electronic device 100 is playing the text 8 corresponding to the audio 5 (the total duration corresponding to the playback progress is 2 minutes and 58 seconds). At this time, the duration for which the electronic device 100 displays illustration 3 (display duration 1) is less than the preset duration 1, so the electronic device 100 keeps the displayed illustration 3 unchanged. In Figure 9 (E) and Figure 9 (F), the electronic device 100 is playing the text 9 corresponding to the audio 6 (the total duration corresponding to the playback progress is 3 minutes). At this time, the electronic device 100 displays illustration 3 for a duration (display duration 2) greater than or equal to the preset duration 1, so the electronic device 100 switches the displayed illustration 3 to illustration 4.

[0233] Please refer to Figure 10, which is a flowchart of another audio playback method provided by an embodiment of the present application. The method can be applied to the audio playback system 10 shown in Figure 1A. The method can be applied to the audio playback system 10 shown in Figure 1B. The method can be applied to the electronic device 100 shown in Figure 2. The method can be applied to the electronic device 100 shown in Figure 3. The method can include but is not limited to the following steps:

[0234] S401: The electronic device 100 obtains the audio corresponding to the text in the sliding window.

[0235] In one embodiment, before S401 , S101 - S102 in FIG. 5 may also be included.

[0236] In one embodiment, the description of S401 is similar to the description of the electronic device 100 acquiring the audio corresponding to the text in the sliding window in S103 of FIG. 5 . Please refer to the specific description in S103 of FIG. 5 and will not be repeated here.

[0237] S402: The electronic device 100 determines whether the sliding window contains an illustration.

[0238] In one embodiment, the description of S402 is similar to the description of S202 in Figure 6. Please refer to the specific description of S202 in Figure 6. When it is determined that the sliding window contains illustrations, the electronic device 100 can execute S403; when it is determined that the sliding window does not contain illustrations, the electronic device 100 can execute S407.

[0239] S403: The electronic device 100 determines a trigger display position of the illustration.

[0240] In one embodiment, the trigger display position of the illustration can be used to trigger the display screen to update the illustration.

[0241] In one embodiment, the electronic device 100 may determine the trigger display position of the illustration based on the number of lines / paragraphs, words, or sentences in the text associated with the illustration in the sliding window. In some examples, assuming that the two lines of text before the illustration and the three lines of text after the illustration are all associated with the illustration, the electronic device 100 may determine the positions of the two lines of text as the trigger display position of the illustration.

[0242] In one embodiment, the sliding window may include multiple illustrations, each of which may correspond to a trigger display position. Optionally, the multiple illustrations may correspond to different trigger display positions.

[0243] S404: The electronic device 100 determines whether the current playback progress is at the trigger display position of the fourth illustration.

[0244] In one embodiment, the current playback progress may be the position of the text corresponding to the audio currently being played by the electronic device 100 in the sliding window. For example, the current playback progress may correspond to the starting edge of the sliding window. For an explanation of the positional relationship between the current playback progress and the starting edge of the sliding window, see the relevant description in S102 of FIG. 5 , which will not be repeated here. For ease of explanation, FIG. 10 uses the current playback progress corresponding to the starting edge of the sliding window as an example for explanation.

[0245] In one embodiment, the fourth illustration may be any illustration in the sliding window. When the starting edge of the sliding window (i.e., the current playback progress of the electronic device 100) is located at the trigger display position of the fourth illustration, the electronic device 100 may execute S405; when the starting edge of the sliding window is not located at the trigger display position of the fourth illustration, the electronic device 100 may execute S407.

[0246] S405: The electronic device 100 obtains a fourth illustration.

[0247] In one embodiment, the electronic device 100 may obtain a fourth illustration identifier, and parse the fourth illustration identifier to obtain a parsed fourth illustration.

[0248] S406: The electronic device 100 plays audio and displays a fourth illustration.

[0249] In one embodiment, after the electronic device 100 obtains the audio and the fourth illustration, it can play the audio through the speaker and display the fourth illustration through the display screen. It can be understood that the electronic device 100 updates and displays the fourth illustration while playing the audio.

[0250] S407: The electronic device 100 plays the audio and keeps the illustration displayed on the display unchanged.

[0251] In one embodiment, the description of S407 is similar to the description of S206 in FIG. 6 , and reference may be made to the specific description of S306 in FIG. 8 , which will not be repeated here.

[0252] S408: The electronic device 100 moves the sliding window according to a third preset step.

[0253] In one embodiment, the description of S408 is similar to the description of S207 in Figure 6, except that the third preset step can also be related to the current playback progress. The electronic device 100 can move the sliding window according to the third preset step and re-execute S401-S407. At this time, the position of the sliding window after the move is different from before, and the text contained in the sliding window after the move is also different from before.

[0254] In the method shown in Figure 10, when the sliding window contains illustrations, the electronic device does not directly display the contained illustrations, but further determines whether the current playback progress (for example, the starting edge of the sliding window) is at the trigger display position of the illustration. This method accurately controls the moment of illustration display, ensures that the displayed illustrations and text match, further improves the effect of synchronous display of illustrations, improves functional availability, and improves user experience.

[0255] Next, the application scenario involved and the user interface diagram in this scenario will be introduced in conjunction with the implementation shown in FIG10 .

[0256] Scenario 4: When the electronic device 100 recognizes that the sliding window includes an illustration, the electronic device 100 may display the illustration according to the trigger display position of the illustration. For specific examples, see (A) to (F) of FIG. 11 .

[0257] As shown in FIG11(A), the sliding window in the first content is located at position 7 (assuming that the starting edge of the sliding window corresponds to the current playback progress of the electronic device 100), and the sliding window does not include an illustration at this time. The electronic device 100 can execute S401 of FIG10, that is, the electronic device 100 can obtain the audio 7 corresponding to the text 7 in the sliding window. At the same time, the electronic device 100 can execute S402 of FIG10 and recognize that the sliding window does not include an illustration. Then, it can execute S407 of FIG10, that is, play the audio 7 and keep the illustration displayed on the display screen (the illustration located before the sliding window, assuming it is illustration 5) unchanged. At this time, the electronic device 100 can display the user interface 1110 shown in FIG11(B).

[0258] As shown in FIG11(B), user interface 1110 may be an audiobook interface for the first application. User interface 1110 may include image 1111, preview image 1112A, preview image 1112B, text content 1113, progress bar 1116, and time 1117. Image 1111 may correspond to illustration 5 in FIG11(A). The location of image 1111 may be the display location of the illustration in the audiobook scenario. The display location remains unchanged, but the displayed content (i.e., image) may change. Preview image 1112A may be a preview of illustration 5 in FIG11(A). Preview image 1112A is selected, indicating that illustration 5 is currently being displayed. Preview image 1112B may be a preview of illustration 6 in FIG11(A). Text content 1113 may include text 1114 and a text box 1115. Text 1114 (including the characters "Spring plants begin to sprout and grow") may correspond to text 10 in FIG. 11 (A). Text 1114 (corresponding to text 10) is framed by text box 1115, indicating that text 1114 is the text corresponding to audio 7 currently being played by electronic device 100. Text 1114 is associated with image 1111. Progress bar 1116 and time 1117 (including the characters "01:00") may indicate the progress of the audio currently being played by electronic device 100, i.e., electronic device 100 has played 1 minute of audio.

[0259] In one embodiment, after FIG11(A) and FIG11(B), the electronic device 100 may execute S408 of FIG10, i.e., move the sliding window in the first direction (e.g., the direction of TTS processing) according to step 3. Assume that the sliding window after the movement is at position 8, as shown in FIG11(C), and the sliding window includes illustration 6. The electronic device 100 may execute S401 of FIG10, i.e., obtain audio 8 corresponding to text 8 in the sliding window. At the same time, the electronic device 100 can execute S402 of Figure 10 and recognize that the sliding window includes the identification of illustration 6, and then execute S403 of Figure 10, that is, determine the trigger display position of illustration 6 (assuming it is position 10 in (C) of Figure 11), and then execute S404 of Figure 10, and judge that the current playback progress (for example, the position of text 11 corresponding to the starting edge of the sliding window in (C) of Figure 11) is not located at the trigger display position (position 10) of illustration 6. Therefore, the electronic device 100 can execute S407 of Figure 10, that is, play audio 8 and keep illustration 5 displayed on the display screen unchanged. At this time, the electronic device 100 can display the user interface 1120 shown in (D) of Figure 11.

[0260] As shown in FIG11(D), user interface 1120 is similar to user interface 1110 shown in FIG11(B). For example, picture 1111 displayed in user interface 1120 is the same as picture 1111 displayed in user interface 1110, i.e., picture 1111 (corresponding to FIG5) is displayed. The difference is that text content 1123 in user interface 1120 may include text 1124 and text box 1125. Text 1124 (including the characters "All things have come here, gradually recovering") may correspond to text 11 in FIG11(C). Text 1124 (corresponding to text 11) is framed by text box 1125, which may indicate that text 1124 is the text corresponding to audio 8 currently being played by electronic device 100, and text 1124 is related to picture 1111. Progress bar 1126 and time 1127 (including the characters "02:00") may indicate the progress of the audio currently being played by electronic device 100, i.e., electronic device 100 has played 2 minutes of audio.

[0261] In one embodiment, after (C) in FIG. 11 and (D) in FIG. 11, the electronic device 100 may execute S408 in FIG. 10, that is, move the sliding window in the first direction again according to step 3. Assume that the moved sliding window is located at position 9, as shown in (E) in FIG. 11. At this time, the sliding window includes illustration 6. The electronic device 100 may execute S401 in FIG. 10, that is, obtain the audio 9 corresponding to the text 9 within the sliding window. At the same time, the electronic device 100 may execute S402 in FIG. 10, and recognize the identifier of illustration 6 included in the sliding window, then execute S403 in FIG. 10, that is, determine the trigger display position of illustration 6 (the above position 10), then execute S404 in FIG. 10, and determine that the current playing progress (for example, the position of the text 12 corresponding to the starting edge of the sliding window in (E) in FIG. 11) is located at the trigger display position of illustration 6 (position 10). Therefore, the electronic device 100 may execute S406 in FIG. 10, that is, obtain illustration 6, play audio 9 and display illustration 6 on the display screen. At this time, the electronic device 100 may display the user interface 1130 shown in (F) in FIG. 11.

[0262] As shown in (F) in FIG. 11, the user interface 1130 may include a picture 1131, text content 1133, a progress bar 1136, and time 1137. Among them, the picture 1131 may correspond to illustration 6 in (E) in FIG. 11, and the preview picture 1112B is in a selected state, which may represent that illustration 6 is currently being displayed. The text content 1133 may include text 1134 and a text box 1135. The text 1134 (including the characters "Yang moves upward and Yin is forced downward") may correspond to the text 12 in (E) in FIG. 11. The text 1134 (corresponding to the text 12) is framed by the text box 1135, which may represent that the text 1134 is the text corresponding to the audio 9 currently being played by the electronic device 100, and the text 1134 is related to the picture 1131. The progress bar 1136 and time 1137 (including the characters "02:50") may indicate the progress of the audio currently being played by the electronic device 100, that is, the electronic device 100 has played the audio for 2 minutes and 50 seconds.

[0263] It can be understood that at (C) in FIG. 11 and (D) in FIG. 11, the electronic device 100 is playing the text 11 corresponding to the audio 8. At this time, the position of the text 11 corresponding to the sliding window is not located at the trigger display position of illustration 6 (position 10). Therefore, the electronic device 100 keeps the displayed illustration 5 unchanged. At (E) in FIG. 11 and (F) in FIG. 11, the electronic device 100 is playing the text 12 corresponding to the audio 9. At this time, the position of the text 12 corresponding to the sliding window is located at the trigger display position of illustration 6 (position 10). Therefore, the electronic device 100 obtains illustration 6 and displays illustration 6.

[0264] Please refer to Figure 12, which is a flowchart of another audio playback method provided by an embodiment of the present application. The method can be applied to the audio playback system 10 shown in Figure 1A. The method can be applied to the audio playback system 10 shown in Figure 1B. The method can be applied to the electronic device 100 shown in Figure 2. The method can be applied to the electronic device 100 shown in Figure 3. The method can include but is not limited to the following steps:

[0265] S501: A first device obtains first progress information of first content.

[0266] In one embodiment, the first content may be an e-book. When the first device receives a user operation that triggers the book listening function, the first progress information of the first content may be obtained. The first progress information is similar to the description of the progress information in S101 of Figure 5. The description of S501 can be found in the description of S101 of Figure 5 and will not be repeated here.

[0267] S502: The first device determines a first playback resource according to the first progress information.

[0268] In one implementation, the first playback resource may include a first text and a first picture of the first content, and the first playback resource is determined according to a text position corresponding to the first progress information.

[0269] In one embodiment, the first device can determine the first playback resource based on the first progress information and the sliding window. The sliding window is an interval in the first content determined according to preset conditions. The preset conditions may include at least one of the following: the number of words in the preset text, the number of sentences in the preset text, and the duration corresponding to the preset text. The sliding window is similar to the description of the sliding window in S102 of Figure 5. The description of S502 can be found in the description of S102 of Figure 5 and will not be repeated here.

[0270] S503: The first device plays the first audio according to the first playback resource, and displays the first picture according to the first playback resource.

[0271] In one embodiment, the first audio may be the audio corresponding to the first text. The first device may send the first text to the network device and receive the first audio returned by the network device. The method for obtaining the first audio may refer to the relevant description in S103 of Figure 5. The description of S503 may refer to the description of S104 of Figure 5, and will not be repeated here.

[0272] In one embodiment, the description of the first picture can refer to the specific description of the first illustration in S103 of Figure 5. The first playback resource may include the first picture and the fourth picture. When the display position of the first picture is before the display position of the fourth picture, the first device can display the first picture.

[0273] In one embodiment, when the current playback progress in the first playback resource is located at the trigger display position of the first picture, the first device can display the first picture. For specific instructions, please refer to S404-S407 of Figure 10, which will not be repeated here.

[0274] S504: The first device obtains second progress information of the first content.

[0275] In one embodiment, the second progress information is different from the first progress information. For the description of S504 , reference may be made to the description of S105 in FIG. 5 , and details are omitted for brevity.

[0276] S505: The first device determines a second playback resource according to the second progress information and the first playback resource.

[0277] In one embodiment, the second playback resource includes the second text and the second image in the first content. The second playback resource is determined based on the text position corresponding to the second progress information. The second playback resource is different from the first playback resource. The description of S505 can be found in the description of S502 in Figure 12 and will not be repeated here.

[0278] S506: The first device plays the second audio according to the second playback resource, and displays the second picture according to the second playback resource.

[0279] In one implementation, the second audio may be the audio corresponding to the second text. For detailed description, please refer to the description of the first audio in S503 of FIG. 12 , which will not be repeated here.

[0280] In one embodiment, the description of the second picture can refer to the specific description of the first illustration in S103 of Figure 5. The second playback resource may include the first picture and the second picture. The first progress information is before the second progress information. The display position of the first picture is before the display position of the second picture. When the display time of the first picture is greater than or equal to the preset time, the first picture displayed on the first device is switched to the second picture; when the display time of the first picture is less than the preset time, the first device displays the first picture. For specific descriptions, please refer to the descriptions of S305-S308 of Figure 8, and no further details will be given.

[0281] In one embodiment, after S506, the first device can obtain the third progress information of the first content, determine the third playback resource based on the third progress information and the second playback resource, and determine whether the third playback resource includes a picture. When the third playback resource includes a third picture, the second picture displayed on the first device is switched to the third picture, and the second picture and the third picture are different; when the third playback resource does not include a picture, the second picture is displayed.

[0282] In the method shown in Figure 12, the first device can synchronously display illustrations related to the text currently being read while playing audio, solving the problem of inconsistency between images and text during listening to books, improving functional usability, and enhancing user experience. Moreover, this method can be applied to any first content that has not been pre-produced, without the need to spend additional labor costs and time costs to pre-produce the first content, expanding the range of first content that users can choose, meeting the diverse needs of users, and further improving the user experience. In addition, when listening to books in real time, the first device does not need to be connected to the Internet in real time, which reduces the delay in loading illustrations, reduces the number of network calls, saves traffic costs, and further ensures the effect of synchronous display of illustrations.

[0283] The methods provided in the various embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital video disc (DWD), or a semiconductor medium (e.g., a solid state drive (SSD)). As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the same. Although the present application has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments may be modified, or some of the technical features thereof may be replaced by equivalents. However, such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio playback method, characterized in that: Applied to a first device, the method includes: Acquire first progress information of first content, where the first content includes text and at least one picture; determining a first playback resource according to the first progress information, where the first playback resource includes a first text and a first picture in the first content, and both the first text and the first picture are related to the first progress information; Play a first audio according to the first playback resource, display the first picture according to the first playback resource, the first audio being the audio corresponding to the first text; Acquire second progress information of the first content; determining a second playback resource according to the second progress information and the first playback resource, where the second playback resource includes a second text and a second picture in the first content, and both the second text and the second picture are related to the second progress information; Play the second audio according to the second playback resource, display the second picture according to the second playback resource, and the second audio is the audio corresponding to the second text; The first progress information and the second progress information are different.

2. The method according to claim 1, characterized in that The method further comprises: determining whether a third playback resource includes a picture, wherein the third playback resource is determined according to the acquired third progress information of the first content and the second playback resource; When the third playback resource includes a third picture, switching the second picture displayed by the first device to the third picture, where the third picture is different from the second picture; When the third playback resource does not include the picture, the second picture is displayed.

3. The method according to claim 1 or 2, characterized in that The first playback resource is determined according to a text position corresponding to the first progress information, the second playback resource is determined according to a text position corresponding to the second progress information, and the second playback resource is different from the first playback resource.

4. The method according to any one of claims 1 to 3, characterized in that: The first playback resource includes the first picture and the fourth picture, The displaying the first picture according to the first playback resource includes: When the display position of the first picture is before the display position of the fourth picture, the first picture is displayed.

5. The method according to any one of claims 1 to 4, characterized in that: The second playback resource includes the second picture and the first picture, the first progress information is before the second progress information, and the display position of the first picture is before the display position of the second picture; The displaying the second picture according to the second playback resource includes: When the display time of the first picture is greater than or equal to a preset time, switching the first picture displayed by the first device to the second picture; The method further comprises: When the display time of the first picture is less than the preset time, the first picture is displayed.

6. The method according to claim 1, characterized in that The displaying the first picture according to the first playback resource includes: When the current playback progress in the first playback resource is located at a trigger display position of the first picture, the first picture is displayed.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire the first audio and the first picture according to the first playback resource; The first audio and the first picture are sent to a second device, where the first audio is played by the second device and the first picture is displayed by the second device.

8. The method according to claim 2, characterized in that The method further comprises: When the second playback resource includes the second picture, sending a first message to the second device, where the first message indicates updating the displayed picture; receiving a second message sent by the second device, where the second message is used to request to obtain an updated picture; The second picture is sent to the second device according to the second message, and the second picture is used for display by the second device.

9. The method according to any one of claims 1 to 8, characterized in that The determining the first playback resource according to the first progress information includes: The first playback resource is determined according to the first progress information and a sliding window, where the sliding window is an interval in the first content determined according to preset conditions, and the preset conditions include at least one of the following: the number of words in a preset text, the number of sentences in a preset text, and the duration corresponding to the preset text.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Sending the first text to a network device; The first audio sent by the network device is received, where the first audio is obtained by the network device according to the first text.

11. The method according to claims 1 to 10, characterized in that The first content is an e-book, and the obtaining of first progress information of the first content includes: When a user operation for triggering the audiobook function is received, the first progress information of the first content is acquired.

12. An electronic device, characterized in that: The method comprises a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the method according to any one of claims 1 to 11.

13. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images