Audio playing method, electronic equipment and system

Abstract

The invention provides an audio playing method, electronic equipment and an audio playing system. The method comprises the following steps: acquiring first progress information of first content; determining a first playing resource according to the first progress information; playing a first audio according to the first playing resource, and displaying a first picture according to the first playing resource, the first audio being an audio corresponding to the first text; acquiring second progress information of the first content; determining a second playing resource according to the second progress information and the first playing resource; playing a second audio according to the second playing resource, and displaying a second picture according to the second playing resource, the second audio being an audio corresponding to the second text; wherein the first progress information is different from the second progress information. According to the method and the device, the illustration related to the currently read text can be synchronously displayed in the audio playing process, so that the problem that images and texts are not consistent in the book listening process is solved, the function availability is improved, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to an audio playing method, an electronic device, and a system. Background Art

[0002] With the development of Artificial Intelligence (AI) technology, the Text to Speech (TTS) technology for converting the text content of e-books into audio for playing is widely used. In the real-time audiobook scenario, an electronic device can obtain the audio corresponding to the text part of the content to be read in real time and play the audio. During the process of playing the audio, the electronic device may display a picture that does not match the audio, such as the cover picture of an e-book, or not display any picture, resulting in the problem of inconsistent text and picture, and low functional availability. Summary of the Invention

[0003] This application discloses an audio playing method, an electronic device, and a system, which can synchronously display the illustrations related to the text being read during the process of playing the audio, solve the problem of inconsistent text and picture during the audiobook process, improve the functional availability, and enhance the user experience.

[0004] In a first aspect, an embodiment of this application provides an audio playing method, which is applied to a first device. The method includes: obtaining first progress information of first content, where the first content includes text and at least one picture; determining a first playing resource according to the first progress information, where the first playing resource includes first text and a first picture in the first content, and both the first text and the first picture are related to the first progress information; playing a first audio according to the first playing resource, and displaying the first picture according to the first playing resource, where the first audio is the audio corresponding to the first text; obtaining second progress information of the first content; determining a second playing resource according to the second progress information and the first playing resource, where the second playing resource includes second text and a second picture in the first content, and both the second text and the second picture are related to the second progress information; playing a second audio according to the second playing resource, and displaying the second picture according to the second playing resource, where the second audio is the audio corresponding to the second text; where the first progress information is different from the second progress information.

[0005] In the above method, the first device can obtain the first progress information of the first content, determine the first playback resource according to the first progress information. The first device can play the first audio according to the first playback resource and display the first picture. Subsequently, the first device can update the progress information of reading the first content, and re-determine the updated playback resource (i.e., the second playback resource) according to the updated progress information (i.e., the second progress information), and repeat the process of playing the audio and displaying the picture. This method can synchronously display the picture matching the text being read currently during the audio playback process, solve the problem of inconsistent pictures and texts during the audiobook listening process, improve the functional availability, and enhance the user experience.

[0006] In a possible implementation manner, the method further includes: determining whether the third playback resource includes a picture, where the third playback resource is determined according to the third progress information of the first content obtained and the second playback resource; when the third playback resource includes the third picture, switching the second picture displayed by the first device to the third picture, where the third picture is different from the second picture; when the third playback resource does not include the picture, displaying the second picture.

[0007] In the above method, after the first device determines the updated playback resource (i.e., the third playback resource) according to the progress information updated again (i.e., the third progress information), it can determine whether the third playback resource includes a picture. When it includes a picture (i.e., the third picture), it can switch the second picture displayed on the display screen to the third picture; when it does not include a picture, it can keep the second picture displayed on the display screen unchanged. This method can update the displayed picture in real time during the audio playback process, ensure the consistency of pictures and texts, and enhance the user experience.

[0008] In a possible implementation manner, the first playback resource is determined according to the text position corresponding to the first progress information, the second playback resource is determined according to the text position corresponding to the second progress information, and the second playback resource is different from the first playback resource.

[0009] In the above method, the first device determines the playback resource in real time according to the playback progress information during the audio playback process. That is to say, the playback resources determined at different playback progress are different. Obtaining the playback resource in real time through the playback progress can ensure the matching of the text corresponding to the played audio and the displayed picture, improve the functional availability, and enhance the user experience.

[0010] In a possible implementation manner, the first playback resource includes the first picture and the fourth picture. The displaying of the first picture according to the first playback resource includes: when the display position of the first picture is before the display position of the fourth picture, displaying the first picture.

[0011] In the above method, when the first playback resource includes multiple pictures, the pictures can be displayed on the display screen according to the display positions of the multiple pictures (for example, in the order from the front to the back), so that the text corresponding to the played audio matches the displayed pictures, improving the user experience.

[0012] In a possible implementation manner, the second playback resource includes the second picture and the first picture, the first progress information is before the second progress information, and the display position of the first picture is before the display position of the second picture; the step of displaying the second picture according to the second playback resource includes: when the display duration of the first picture is greater than or equal to a preset duration, switching the first picture displayed on the first device to the second picture; the method further includes: when the display duration of the first picture is less than the preset duration, displaying the first picture.

[0013] In the above method, when the second playback resource includes a picture that has been displayed (i.e., the first picture) and a picture that has not been displayed (i.e., the second picture), it is possible to determine whether to switch the display to the picture that has not been displayed according to the display duration of the picture that has been displayed. In this way, when the positions of multiple pictures in the second playback resource are relatively close, it can be ensured that each picture has sufficient time (for example, 10 seconds - 30 seconds) to be displayed on the display screen, and the display duration of the picture is determined according to the text related to the picture, further improving the effect of synchronous display.

[0014] In a possible implementation manner, the step of displaying the first picture according to the first playback resource includes: when the current playback progress in the first playback resource is at the first trigger display position of the first picture, displaying the first picture.

[0015] In the above method, when the first playback resource includes pictures, the first device does not directly display the included pictures, but further determines whether the current playback progress is at the trigger display position of the pictures. This method precisely controls the moment of picture display, ensures that the displayed pictures match the text, further improves the effect of synchronous picture display, improves the functional availability, and improves the user experience.

[0016] In a possible implementation manner, the method further includes: obtaining the first audio and the first picture according to the first playback resource; sending the first audio and the first picture to a second device, where the first audio is used for the second device to play, and the first picture is used for the second device to display.

[0017] In the above method, when the first device and the second device are connected, after the first device obtains the first audio and the first picture, the first audio and the first picture can be synchronously pushed to the second device (external remote device), so that when the second device plays the audio, the picture matching the text being read currently is synchronously displayed, expanding the scope of applicable devices, further improving the functional availability, and enhancing the user experience.

[0018] In a possible implementation manner, the method further includes: when the second playback resource includes the second picture, sending a first message to the second device, where the first message instructs to update the displayed picture; receiving a second message sent by the second device, where the second message is used to request to obtain the updated picture; and sending the second picture to the second device according to the second message, where the second picture is used for the second device to display.

[0019] In the above method, for different second devices (external remote devices), the way the first device pushes the audio and the picture to the second device may be different. Through different interaction methods, it can be ensured that the text corresponding to the audio played by the second device and the displayed picture match, expanding the scope of applicable devices, further improving the functional availability, and enhancing the user experience.

[0020] In a possible implementation manner, the determining the first playback resource according to the first progress information includes: determining the first playback resource according to the first progress information and a sliding window, where the sliding window is an interval in the first content determined according to a preset condition, and the preset condition includes at least one of the following: the number of characters of a preset text, the number of sentences of a preset text, and the duration corresponding to a preset text.

[0021] In the above method, the determination of the sliding window can directly affect the effect of synchronously displaying the picture. Therefore, the size of the sliding window can be accurately controlled through the preset condition, so as to accurately control the moment of picture display, ensure that the displayed picture and the text match, further improve the effect of synchronously displaying the picture, improve the functional availability, and enhance the user experience.

[0022] In a possible implementation manner, the method further includes: sending the first text to a network device; and receiving the first audio sent by the network device, where the first audio is obtained by the network device according to the first text.

[0023] In the above method, the first device can obtain the first audio corresponding to the first text through a network device, which can reduce the processing pressure of the first device. In the case where the processing capacity of the first device is insufficient, it can still ensure that when the first device plays the audio, a picture matching the currently read text is synchronously displayed, expanding the scope of applicable devices, further improving the functional availability, and enhancing the user experience.

[0024] In a possible implementation manner, the first content is an e-book, and obtaining the first progress information of the first content includes: when receiving a user operation that triggers the audiobook function, obtaining the first progress information of the first content.

[0025] In the above method, in a real-time audiobook scenario, when the user triggers the audiobook function, the first device can enable the audiobook function according to the user's reading progress information, simplifying the triggering operation of the audiobook function. There is no need for the user to perform additional operations, facilitating the user to use the audiobook function, further improving the functional availability and the user experience.

[0026] In a second aspect, the present application provides an electronic device, including a transceiver, a processor, and a memory. The above memory is used to store a computer program, and the above processor calls the above computer program to execute the audio playback method in any possible implementation manner of the first aspect.

[0027] In a third aspect, the present application provides an electronic device, including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code. The computer program code includes computer instructions. When the one or more processors execute the computer instructions, the electronic device is caused to execute the audio playback method in any possible implementation manner of the first aspect.

[0028] In a fourth aspect, the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, it realizes the execution of the audio playback method in any possible implementation manner of any aspect above.

[0029] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is caused to execute the audio playback method in any possible implementation manner of the first aspect.

[0030] In a sixth aspect, the present application provides an electronic device, which includes a method or device described in any implementation manner of the first aspect of the present application. The above electronic device is, for example, a chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings used in this application are introduced below.

[0032] Figure 1A It is a schematic architecture diagram of an audio playback system 10 provided by this application;

[0033] Figure 1B It is an interaction schematic diagram of an audio playback system 10 provided by this application;

[0034] Figure 2 It is a schematic hardware structure diagram of an electronic device 100 provided by this application;

[0035] Figure 3 It is a schematic software architecture diagram of an electronic device 100 provided by this application;

[0036] Figure 4 It is a schematic hardware structure diagram of a network device 200 provided by this application;

[0037] Figure 5 It is a schematic flowchart of an audio playback method provided by this application;

[0038] Figure 6 It is a schematic flowchart of another audio playback method provided by this application;

[0039] Figure 7 It is a schematic diagram of some application scenarios provided by this application;

[0040] Figure 8 It is a schematic flowchart of another audio playback method provided by this application;

[0041] Figure 9 It is a schematic diagram of some other application scenarios provided by this application;

[0042] Figure 10 It is a schematic flowchart of another audio playback method provided by this application;

[0043] Figure 11 It is a schematic diagram of some other application scenarios provided by this application;

[0044] Figure 12 It is a schematic flowchart of another audio playback method provided by this application. Detailed implementation manners

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality of" means two or more than two.

[0046] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0047] An electronic device can provide a text-to-speech (TTS) function. For example, in a real-time audiobook scenario, the electronic device can send the text part of the content to be read to the server to obtain the corresponding audio and play the audio. During the playback of the audio, the electronic device can randomly display a picture on the display screen, such as the cover picture of the e-book obtained by the electronic device, or not display any picture, resulting in a problem of inconsistent text and pictures, low functional availability, and poor user experience.

[0048] Currently, the e-books can be pre-converted and produced. Specifically, the e-books are parsed, and the parsed text is converted by means of TTS technology or manual reading to obtain an audio file (including the timeline information of the audio), and according to the chapter position corresponding to each illustration parsed, the corresponding relationship between the illustration and the audio playback timestamp is determined, and these information are integrated to obtain an audiobook file. When a user listens to an audiobook, the electronic device can obtain the corresponding illustration information according to the timestamp of the played audio, such as the Uniform Resource Locator (URL) of the illustration, and send the illustration information to the server to obtain the corresponding illustration, which is synchronously displayed on the display screen. However, this method requires a large amount of labor cost and time cost for each e-book in advance. If the user wants to synchronously enjoy the illustrations when listening to the audiobook (that is, to achieve the effect of unified text and pictures), the user can only choose from the pre-produced e-books, and the user's choice is limited. For e-books that have not been pre-produced, it is impossible to synchronously display illustrations when playing the audio, which is not applicable to the scenario of real-time audiobook listening, cannot meet the user's needs, and the user experience is poor. In addition, when the electronic device downloads the illustration from the server, it needs to spend traffic cost. Therefore, when there is a network anomaly such as an abnormal disconnection between the electronic device and the server, there will be a delay, which affects the effect of synchronously displaying the illustration.

[0049] An embodiment of the present application provides an audio playback method, which is applied to an audio playback system. The audio playback system may include an electronic device and a network device. In a real-time audiobook scenario, the electronic device can obtain the progress information of the first content. The first content may include text and pictures / illustrations. According to the progress information, the initial position of the sliding window in the first content is determined. The sliding window can be used to indicate the position where the electronic device is currently performing TTS processing. The sliding window is a window that can move in the first content. For example, the sliding window is moved according to a preset step. The electronic device can control the sliding window to move from the initial position in the first direction (for example, the direction of TTS processing). During the movement of the sliding window, the electronic device can obtain the text in the sliding window in real time, send the text to the network device to obtain the corresponding audio, and play the audio. At the same time, the electronic device can also recognize the illustrations in the sliding window and display the recognized illustrations on the display screen. This method can synchronously display the illustrations related to the text being read currently during the audio playback process, solve the problem of inconsistent text and pictures during the audiobook process, improve the functional availability, and enhance the user experience. Moreover, this method can be applied to any first content that has not been pre-produced, without the need to spend additional labor costs and time costs to pre-produce the first content in advance, expand the range of first content that users can choose, meet the diverse needs of users, and further enhance the user experience. In addition, during real-time audiobook listening, the electronic device does not need to remain connected to the network in real time, reducing the delay of illustration loading, reducing the number of network calls, and saving traffic costs, further ensuring the effect of synchronously displaying illustrations.

[0050] The following introduces an audio playback system 10 involved in an embodiment of the present application.

[0051] Figure 1A An exemplary schematic diagram of the architecture of an audio playback system 10 is shown.

[0052] As Figure 1A shown, the audio playback system 10 may include an electronic device 100, a network device 200, and an electronic device 300. The electronic device 100 can communicate with the network device 200 and the electronic device 300 in a wired manner (for example, universal serial bus (USB), twisted pair, coaxial cable, and optical fiber, etc.) and / or a wireless manner (for example, wireless local area networks (WLAN), Bluetooth, and cellular communication networks, etc.).

[0053] The electronic device 100 can be a mobile phone, a tablet computer, an all-in-one computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as a smart TV and a projector, wearable devices such as a smart bracelet, a smart watch, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), or in-vehicle devices. The description of the electronic device 300 is the same as that of the electronic device 100. The specific type of the electronic device is not particularly limited in the embodiments of the present application.

[0054] The network device 200 may include at least one server. In an optional embodiment, any one of the servers can be a hardware server. In an optional embodiment, any one of the servers can be a cloud server. In an optional embodiment, any one of the servers can be a Linux server, a Windows server, or other server devices that can provide simultaneous access for multiple devices. In an optional embodiment, the network device 200 can be a server cluster composed of multiple regions, multiple computer rooms, and multiple servers.

[0055] In one implementation, the electronic device 100 can communicate with the network device 200 based on the browser / server (B / S) architecture or based on the client / server (C / S) architecture. In one implementation, the electronic device 100 can send the obtained text to the network device 200, and the network device 200 can convert the text into audio and return it to the electronic device 100. In some examples, after receiving the audio, the electronic device 100 can play the audio. During the playback of the audio, when the electronic device 100 identifies the illustration to be displayed through a sliding window and obtains the illustration, the illustration can be displayed.

[0056] In some other examples, after receiving the audio, the electronic device 100 may obtain a corresponding audio stream according to the audio and push the audio stream to the electronic device 300. During the process of the electronic device 300 playing the audio stream, when the electronic device 100 identifies an illustration to be displayed through a sliding window and obtains the illustration, it may send a first message to the electronic device 300, and the first message is used to indicate an update to display the illustration. After receiving the first message, the electronic device 300 may send a request message to the electronic device 100, receive the illustration returned by the electronic device 100, and display the illustration.

[0057] Understandably, Figure 1A The number and form of the illustrated electronic devices are only examples. In specific implementations, the number may be more or less. For example, the audio playback system 10 may further include other electronic devices. For another example, the audio playback system 10 may not include the electronic device 300. This application does not make any limitations in this regard.

[0058] Figure 1B An interaction schematic diagram of an audio playback system 10 is exemplarily shown.

[0059] As Figure 1B As shown, the electronic device 100 in the audio playback system 10 may include a reading application 101 (including a reader 102 and a TTS player 103), a storage module 104, a Bluetooth target end 201, and a MediaSession 202. The network device 200 in the audio playback system 10 may include a text-to-audio module 301. The electronic device 300 in the audio playback system 10 may include at least one of a Bluetooth playback device (including a Bluetooth control end 401 and a processing module 402) and a vehicle head unit playback device (Hicar end 501 and a processing module 502). Among them:

[0060] The reader 102 may include a format parsing module 1021 and a content extraction module 1022 (including a sliding window module 1023). The TTS player 103 may include an audio synthesis module 1031, an audio playback module 1032, and a display module 1033. The storage module 104 may be used to store the first content. The format of the first content is, for example but not limited to, the epub format. The first content may include text and illustrations. The first content may include a start position, an end position, and a reading position. The reading position may indicate the position where the user is currently reading the first content. The storage module 104 may send the above-mentioned first content to the format parsing module 1021. The format parsing module 1021 may perform parsing processing on the first content, such as decryption, decompression, format parsing, etc., and send the parsed first content to the sliding window module 1023 in the content extraction module 1022. The content extraction module 1022 may determine the initial position of the sliding window in the first content according to the progress information of the first content (i.e., the above-mentioned reading position). The sliding window module 1023 may identify the text and illustrations in the sliding window, send the extracted text to the audio synthesis module 1031, and when an illustration to be displayed is recognized, obtain the illustration and send the illustration to the display module 1033. After receiving the text, the audio synthesis module 1031 may send the text to the text-to-audio module 301 in the network device 200. The text-to-audio module 301 may generate the audio corresponding to the text through TTS technology and send the generated audio to the audio synthesis module 1031. After receiving the audio, the audio synthesis module 1031 may send the audio to the audio playback module 1032. The audio playback module 1032 may play the audio through a speaker. The display module 1033 may display the illustration through a display screen. The sliding window module 1023 may also control the sliding window to move in the first direction. During the movement of the sliding window, the sliding window module 1023, the audio synthesis module 1031, the text-to-audio module 301, the audio playback module 1032, and the display module 1033 may repeat the above process to play the text in the sliding window in real time after converting it into audio and display the illustrations to be displayed in the sliding window.

[0061] In one implementation, after the sliding window module 1023 identifies the illustration to be displayed and obtains the illustration, it can send a notification message and the illustration to the audio playback module 1032. The notification message is used to indicate an update to the displayed illustration. After receiving the notification message and the illustration, the audio playback module 1032 can send the notification message to the Bluetooth control end 401 of the Bluetooth playback device through the Bluetooth target end 201. The Bluetooth control end 401 can send the notification message to the processing module 402. The processing module 402 can parse the notification message, and after determining that the illustration to be updated is to be displayed currently, it can send a request message to the Bluetooth target end 201 of the electronic device 100 through the Bluetooth control end 401. The request message is used to obtain the illustration to be updated. After receiving the request message, the Bluetooth target end 201 can obtain the illustration from the audio playback module 1032 and return the illustration to the Bluetooth control end 401. The Bluetooth control end 401 can send the illustration to the processing module 402, and the processing module 402 can display the illustration. After receiving the audio sent by the audio synthesis module 1031, the audio playback module 1032 can send the audio to the Bluetooth target end 201. The Bluetooth target end 201 can convert the audio into a Bluetooth signal and send it to the Bluetooth control end 401 of the Bluetooth playback device. The Bluetooth control end 401 can send the Bluetooth signal to the processing module 402. The processing module 402 can parse the Bluetooth signal to obtain the parsed audio and play the audio through the speaker.

[0062] In one implementation, after the sliding window module 1023 identifies the illustration to be displayed and obtains the illustration, it can send a notification message and the illustration to the audio playback module 1032. The notification message is used to indicate an update to the displayed illustration. After receiving the notification message and the illustration, the audio playback module 1032 can send the notification message and the illustration to the Hicar end 501 of the in-vehicle playback device through the MediaSession202. The Hicar end 501 can send the notification message and the illustration to the processing module 502. The processing module 502 can parse the notification message, determine that the illustration to be updated is to be displayed currently, and display the received illustration. After receiving the audio sent by the audio synthesis module 1031, the audio playback module 1032 can send the audio to the MediaSession202. The MediaSession202 can obtain the corresponding audio stream based on the audio and send it to the Hicar end 501 of the in-vehicle playback device. The Hicar end 501 can send the audio stream to the processing module 502. The processing module 502 can parse the audio stream to obtain the parsed audio and play the audio through the speaker.

[0063] Optionally, the Bluetooth control end 401 of the Bluetooth playback device can receive the playback control instruction input by the user, and the playback control instruction is used to control the playback of audio. For example, pausing the playback of audio, starting the playback of audio, playing the audio by a multiple, etc. The Bluetooth control end 401 can send the playback control instruction to the Bluetooth target end 201 of the electronic device 100. After receiving the playback control instruction, the Bluetooth target end 201 can control the state of converting the audio into a Bluetooth signal. For example, pausing the conversion of the audio into a Bluetooth signal, starting the conversion of the audio into a Bluetooth signal, converting the audio into a Bluetooth signal by a multiple, etc.

[0064] Understandably, Figure 1B The illustrated audio playback system 10 may include more or fewer modules. Any one of the modules in the audio playback system 10 may be a hardware module or a software module. Any one of the above modules may be a separate module, and at least one of the above modules may also be integrated together. This application does not make any limitations in this regard. This application does not limit the name of any one of the above modules, and it may also be other names for implementing the functions corresponding to any one of the above modules.

[0065] Next, the structure of the exemplary electronic device provided in the embodiments of the present application will be introduced.

[0066] Figure 2 An exemplary schematic diagram of the hardware structure of an electronic device 100 is shown.

[0067] As Figure 2 shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0068] It can be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0069] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0070] The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0071] A memory may also be provided in the processor 110 for storing instructions and data. In one implementation, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0072] The charging management module 140 is used to receive charging input from a charger. The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery charge cycles, and the battery health status (leakage, impedance). In another implementation, the power management module 141 may also be provided in the processor 110. In another implementation, the power management module 141 and the charging management module 140 may also be provided in the same device.

[0073] The wireless communication function of the electronic device 100 can be implemented by antenna 1, antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0074] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In another implementation, the antenna can be used in combination with a tuning switch.

[0075] The mobile communication module 150 can provide solutions for wireless communication including second generation (2G) / third generation (3G) / fourth-generation (4G) / fifth generation (5G) / sixth generation (6G), etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through antenna 1 for radiation. In one implementation, at least some functional modules of the mobile communication module 150 can be provided in the processor 110. In one implementation, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be provided in the same device.

[0076] The modulation and demodulation processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 194. In one implementation, the modulation and demodulation processor can be an independent device. In another implementation, the modulation and demodulation processor can be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0077] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0078] In one embodiment, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0079] Electronic device 100 implements the display function through the GPU, display screen 194, and application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0080] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In one implementation, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0081] The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, etc.

[0082] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera sensor. The light signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In one implementation, the ISP can be set in the camera 193.

[0083] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the sensor. The sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The sensor converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In one implementation, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0084] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to implement the storage capacity expansion of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0085] The internal memory 121 can be used to store computer-executable program codes, and the executable program codes include instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor.

[0086] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc.

[0087] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals.

[0088] The speaker 170A, also known as a "loudspeaker", is used to convert an audio electrical signal into a sound signal.

[0089] The receiver 170B, also known as a "handset", is used to convert an audio electrical signal into a sound signal.

[0090] The microphone 170C, also known as a "microphone" or "transmitter", is used to convert a sound signal into an electrical signal.

[0091] The headphone jack 170D is used to connect a wired headphone.

[0092] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In one implementation, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In one implementation, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0093] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In one implementation, the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B.

[0094] The barometric pressure sensor 180C is used to measure barometric pressure.

[0095] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case.

[0096] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes).

[0097] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance by infrared or laser. In one implementation, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve rapid focusing.

[0098] The proximity light sensor 180G can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode can be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100.

[0099] The ambient light sensor 180L is used to sense the ambient light brightness.

[0100] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

[0101] The temperature sensor 180J is used to detect the temperature.

[0102] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0103] Bone conduction sensor 180M can obtain vibration signals.

[0104] Key 190 includes a power button, a volume button, etc. Key 190 can be a mechanical button. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0105] The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power change, messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect a SIM card.

[0106] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. For example, the software system of the layered architecture may be an Android system, or a Harmony operating system (OS), or other software systems. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 100.

[0107] Figure 3 A schematic diagram of the software architecture of an electronic device 100 is shown as an example.

[0108] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In one embodiment, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0109] The application layer may include a series of application packages.

[0110] As Figure 3 shown, the application packages may include applications such as camera, gallery, music, calendar, short message, call, navigation, Bluetooth, browser, screen mirroring, etc.

[0111] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0112] As Figure 3 shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0113] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0114] The content provider is used to store and obtain data, and make this data accessible to applications. The data may include videos, images, audio, dialed and answered calls, browsing history and bookmarks, phone books, etc.

[0115] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon may include a view for displaying text and a view for displaying pictures.

[0116] The phone manager is used to provide the communication function of the electronic device 100. For example, the management of call status (including connection, disconnection, etc.).

[0117] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0118] The notification manager enables applications to display notification information in the status bar, can be used to convey notification-type messages, can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the completion of a download, message reminder, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background-running application, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompt text information in the status bar, emit a prompt sound, the electronic device vibrates, the indicator light flashes, etc.

[0119] The Android Runtime includes the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0120] The core libraries consist of two parts: one is the functional functions that the Java language needs to call, and the other is the core libraries of Android.

[0121] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0122] The system libraries can include multiple functional modules. For example: surface manager, Media Libraries, 3D graphics processing libraries (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0123] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0124] The media libraries support the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0125] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0126] The 2D graphics engine is the graphics engine for 2D drawing.

[0127] The kernel layer is the layer between the hardware and the software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

[0128] Next, in combination with the scenario of listening to books, the working processes of the software and hardware of the electronic device 100 will be exemplarily described.

[0129] When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation and the control corresponding to the click operation as the icon control of the reading application as an example, the reading application calls the interface of the application framework layer to start the reading application, and then controls the display driver through the kernel layer to display the user interface of the reading application on the display screen 194.

[0130] Next, an exemplary network device 200 provided by an embodiment of the present application will be introduced.

[0131] Figure 4 An exemplary hardware structure diagram of a network device 200 is shown.

[0132] As Figure 4 shown, the network device 200 may include one or more processors 201, a communication interface 202, and a memory 203. The processors 201, the communication interface 202, and the memory 203 may be connected through a bus or other means. In this embodiment of the present application, it is taken as an example that they are connected through a bus 204. Among them:

[0133] The processor 201 may be composed of one or more general-purpose processors, such as a CPU. The processor 201 can be used to run the program code related to the device control method.

[0134] The communication interface 202 may be a wired interface (such as an Ethernet interface) or a wireless interface (such as a cellular network interface or a wireless local area network interface) for communicating with other nodes. In this embodiment of the present application, the communication interface 202 is specifically used to communicate with the electronic device 100.

[0135] The memory 203 may include a volatile memory, such as RAM; the memory may also include a non-volatile memory, such as ROM, flash memory, HDD, or solid-state drive SSD. The memory 203 may also include a combination of the above types of memories. The memory 203 can be used to store a set of program code so that the processor 201 can call the program code stored in the memory 203 to implement the implementation method of the present application embodiment on the server. In this embodiment of the present application, the memory 203 may also be a storage array, and so on.

[0136] In one embodiment, the network device 200 may include multiple servers, such as a web server, a background server, a download server, etc. The hardware structures of these multiple servers can all refer to Figure 4 the hardware structure of the network device 200 shown.

[0137] It should be noted that Figure 4 the network device 200 shown is an implementation example of the embodiments of this application. In practical applications, the network device 200 may also include more or fewer components, which are not limited here.

[0138] Next, the audio playback method provided by the embodiments of this application will be introduced.

[0139] Please refer to Figure 5 , Figure 5 which is a schematic flowchart of an audio playback method provided by the embodiments of this application. This method can be applied to Figure 1A the audio playback system 10 shown. This method can be applied to Figure 1B the audio playback system 10 shown. This method can be applied to Figure 2 the electronic device 100 shown. This method can be applied to Figure 3 the electronic device 100 shown. This method may include but is not limited to the following steps:

[0140] S101: The electronic device 100 obtains the progress information of the first content.

[0141] In one embodiment, the first content may include text and pictures / illustrations, etc. The first content is, for example but not limited to, a document file, an e-book, etc.

[0142] In one embodiment, the first content may include multiple positions, which include a start position, an end position, and a reading position. The reading position can indicate the position where the user is currently reading the first content, that is, the position corresponding to the progress information.

[0143] In one embodiment, the positions in the first content can be characterized by at least one of a percentage, a position number, a chapter, a page number, and a line number. For example, when characterizing the position by a percentage, 0% can represent the start position of the first content, 100% can represent the end position of the first content, and x% can represent the reading position, where x is greater than or equal to 0 and less than or equal to 100. In one embodiment, there is a front-back distinction between any two / multiple positions in the first content. For example, the smaller the percentage, the more forward the position. For example, the start position is before the end position.

[0144] In some examples, the text in the first content, for example, includes the characters "There are already quite a few lotus flowers in bloom. The lotus leaves are crowded together, like big green disks. White lotus flowers emerge among these disks. Some have only two or three petals unfolded. Some have all their petals unfolded, revealing the small yellowish lotus pods. Some are still flower buds, looking so full that they seem about to burst." Among them, the starting character "There are already" of this text corresponds to 20% of the position in the first content, the middle character "Some have only" corresponds to 25% of the position in the first content, and the ending character "seem about to burst" corresponds to 30% of the position in the first content.

[0145] In one embodiment, the illustration in the first content can be embedded in the text. In some examples, as follows Figure 6 shown, the first content may include multiple lines / paragraphs of text and multiple illustrations, with one or more illustrations interspersed among the multiple lines / paragraphs of text. Among them, the illustrations in the first content can be presented in the form of illustration identifiers, such as image embedding elements ( ) tags.

[0146] In one embodiment, the electronic device 100 can display a first interface, and the first interface can be the interface of a first application, such as an interface for reading the first content. Then, the electronic device 100 can receive a user operation on a function control in the first interface (for example, the user operation is a touch operation), and in response to the user operation, start the audiobook function of the first application and obtain the progress information of the first content.

[0147] S102: The electronic device 100 determines the position of the sliding window according to the progress information.

[0148] In one embodiment, the sliding window can be used to indicate the position where the electronic device is currently performing TTS processing.

[0149] In one embodiment, the electronic device 100 can use the position corresponding to the progress information as the position of the sliding window in the first content.

[0150] In one embodiment, the sliding window can include two sides perpendicular to the first direction (for example, the direction of TTS processing). Among them, the side at the relatively front position can be called the starting side, and the side at the relatively rear position can be called the ending side. In some examples, the electronic device 100 can use the position corresponding to the progress information as the starting side of the sliding window. Understandably, at this time, only the unread text is included in the sliding window. In other examples, the electronic device 100 can use the position corresponding to the progress information as the ending side of the sliding window. Understandably, at this time, only the read text is included in the sliding window.

[0151] Not limited thereto, in another embodiment, the electronic device 100 may also use the position corresponding to the progress information as any other position in the sliding window except the start edge and the end edge. In some examples, the electronic device 100 may use the position of the text read within a preset duration before the position corresponding to the progress information as the start edge of the sliding window. Understandably, at this time, the sliding window includes the read text and the unread text. This application does not limit the positional relationship between the start edge of the sliding window and the position corresponding to the current playback progress.

[0152] In one embodiment, the size of the sliding window may be pre-set. For example, the size of the sliding window may be related to the number of lines / paragraphs, the number of words, the number of sentences, the duration corresponding to the text, and optionally the number of illustrations / illustration identifiers of the text to be included.

[0153] S103: The electronic device 100 obtains the audio and illustrations corresponding to the text within the sliding window.

[0154] In one embodiment, after the electronic device 100 determines the position of the sliding window, it may identify some / all of the content in the sliding window, extract the first text in the sliding window, and obtain the first audio corresponding to the first text. Understandably, the first text may be some / all of the content within the sliding window.

[0155] In one embodiment, after the electronic device 100 obtains the first text, it may send the first text to the network device 200. The network device 200 may process the first text through TTS technology to obtain the first audio corresponding to the first text and return it to the electronic device 100. Not limited thereto, in another embodiment, the electronic device 100 may directly process the first text through TTS technology to obtain the first audio.

[0156] In one embodiment, the electronic device 100 may determine whether the sliding window contains an illustration. When the sliding window contains an illustration, the electronic device 100 may obtain the first illustration. The first illustration may be the illustration that has not been displayed on the display screen in the audiobook scenario and is in the foremost position (at the foremost position). For specific descriptions, please refer to S202 - S204 below. Figure 6 Here, it will not be elaborated for the time being.

[0157] S104: The electronic device 100 plays the audio and displays the illustration.

[0158] In one embodiment, after the electronic device 100 obtains the first audio and the first illustration, it may play the first audio through the speaker and display the first illustration on the display screen. It can be understood that the electronic device 100 updates the display of the first illustration when playing the first audio.

[0159] S105: The electronic device 100 updates the progress information.

[0160] In one embodiment, during the process of playing audio and displaying illustrations, the electronic device 100 can obtain the latest progress information. For example, the electronic device 100 can obtain the position of the text corresponding to the currently played audio in the first content, that is, update the progress (reading position) of currently reading the first content.

[0161] In one embodiment, after the electronic device 100 obtains the latest progress information, it re-executes S102 - S105. At this time, the updated progress information is different from the previous one, the position of the sliding window corresponding to the updated progress information (which can be simply referred to as the updated sliding window) is different from the previous one, and the text included in the updated sliding window is also different from the previous one.

[0162] In Figure 5 In the method shown, in the real-time audiobook scenario, the electronic device can obtain the progress information of the first content and determine the position of the sliding window in the first content according to the progress information. The electronic device can obtain the text in the sliding window, send the text to the network device to obtain the corresponding audio, and play the audio. At the same time, the electronic device can also recognize the illustrations in the sliding window and display the recognized illustrations on the display screen. Subsequently, the electronic device can update the progress information of reading the first content, re-determine the position of the sliding window according to the updated progress information, and repeat the process of playing audio and displaying illustrations. This way can synchronously display the illustrations related to the currently read text during the process of playing audio, solve the problem of inconsistent pictures and texts during the audiobook process, improve the functional usability, and enhance the user experience.

[0163] Please refer to Figure 6 , Figure 6 which is a schematic flowchart of another audio playing method provided by an embodiment of the present application. This method can be applied to Figure 1A the audio playing system 10 shown. This method can be applied to Figure 1B the audio playing system 10 shown. This method can be applied to Figure 2 the electronic device 100 shown. This method can be applied to Figure 3 the electronic device 100 shown. This method can include but is not limited to the following steps:

[0164] S201: The electronic device 100 obtains the audio corresponding to the text within the sliding window.

[0165] In one embodiment, before S201, it may further include Figure 5 S101 - S102.

[0166] In one implementation, the description of S201 and Figure 5 is similar to the specific description in S103 of Figure 5 where the electronic device 100 obtains the audio corresponding to the text within the sliding window. For the specific description, refer to

[0167] S202: The electronic device 100 determines whether the sliding window contains an illustration.

[0168] In one implementation, the electronic device 100 can identify the content in the sliding window to determine whether the sliding window contains an illustration. In one implementation, when an illustration identifier is recognized, the electronic device 100 can determine that the sliding window contains an illustration. At this time, the electronic device 100 can execute S203; when no illustration identifier is recognized, the electronic device 100 can determine that the sliding window does not contain an illustration. At this time, the electronic device 100 can execute S206.

[0169] S203: The electronic device 100 determines whether there is an unshown illustration in the sliding window.

[0170] In one implementation, the electronic device 100 can determine whether the illustration in the sliding window has been shown on the display screen in the audiobook scenario (for example, at the position where the picture 611 in Figure 7 (B) is located). When the sliding window includes an illustration that has not been shown on the display screen, the electronic device 100 can execute S203 - S204 - S205; when the sliding window includes an illustration that has been shown on the display screen, the electronic device 100 can execute S206.

[0171] S204: The electronic device 100 obtains a first illustration.

[0172] In one implementation, the first illustration can be the unshown illustration in the sliding window that is in the foremost position (at the most forward position). The most forward position can be the position farthest from the first direction. The first direction is, for example, the direction of TTS processing, which can be understood as the direction of the first content extension.

[0173] In one implementation, when the number of unshown illustrations in the sliding window is greater than 1, the electronic device 100 can select the first illustration corresponding to the foremost position according to the multiple positions corresponding to these multiple unshown illustrations. For a specific example, refer to Figure 9 (A) below. When the number of unshown illustrations in the sliding window is equal to 1, the electronic device 100 can directly obtain this illustration (the first illustration). For a specific example, refer to Figure 7 .

[0174] In one embodiment, the electronic device 100 may obtain a first illustration identifier, parse the first illustration identifier, and obtain the parsed first illustration.

[0175] Among them, the order of S201 and S202 - S204 is not limited.

[0176] S205: The electronic device 100 plays the audio and displays the first illustration.

[0177] In one embodiment, after the electronic device 100 obtains the audio and the first illustration, it may play the audio through the speaker and display the first illustration on the display screen. It can be understood that the electronic device 100 updates the display of the first illustration while playing the audio.

[0178] S206: The electronic device 100 plays the audio and keeps the illustration displayed on the display screen unchanged.

[0179] In one embodiment, after the electronic device 100 obtains the audio, it may play the audio through the speaker and keep the illustration currently displayed on the display screen unchanged. Not limited to this, when the display screen of the electronic device 100 does not display an illustration, the electronic device 100 may keep the display screen without displaying an illustration.

[0180] S207: The electronic device 100 moves the sliding window according to a first preset step.

[0181] In one embodiment, the first preset step may be pre - set. For example, the first preset step may be related to the size of the sliding window, the number of lines / paragraphs, the number of words, the number of sentences, etc. of the text processed by TTS within the sliding window. For example, when the number of sentences of the text processed by TTS within the sliding window increases by N sentences, where N is a positive integer, the electronic device 100 moves the sliding window once.

[0182] In one embodiment, the electronic device 100 may control the sliding window to move in a first direction from an initial position (for example, the position of the sliding window determined for the first time according to the progress information). The first direction is, for example, the direction of TTS processing, which can be understood as the direction of the first content extension.

[0183] In one embodiment, the electronic device 100 may move the sliding window in a first direction from the current position according to the first preset step and re - execute S201 - S207. At this time, the position of the moved sliding window is different from before, and the text contained in the moved sliding window is also different from before.

[0184] Not limited to Figure 6In the embodiment shown, in another embodiment, after S201, the electronic device 100 may obtain an audio stream according to the audio and send the audio stream to the electronic device 300. After receiving the audio stream, the electronic device 300 may play the audio stream through the speaker. In some examples, after S204, the electronic device 100 may send a first message to the electronic device 300, where the first message is used to indicate an update to the displayed illustration. The first message is, for example, AVRC_MEDIA_ATTR_ID_COVER_ART. After receiving the first message, the electronic device 300 may parse the first message to determine that the displayed illustration needs to be updated currently. Then, the electronic device 300 may send a first request message to the electronic device 100, where the first request message is used to request to obtain the illustration to be updated. The electronic device 100 may send a first illustration to the electronic device 300, and the electronic device 300 may display the first illustration through the display screen. In other examples, after S204, the electronic device 100 may send a second message to the electronic device 300. The second message may include the first illustration, and the second message is used to indicate an update to the displayed illustration. The second message is, for example, MediaDescription.setIconBitmap. After receiving the second message, the electronic device 300 may parse the second message to determine that the displayed illustration needs to be updated currently and display the received first illustration through the display screen. In this way, the electronic device 100 can synchronously push the illustration related to the text being read currently to the electronic device 300 (external remote device), synchronously update the displayed illustration, and improve the user experience.

[0185] In Figure 6In the method shown, in the real-time audiobook scenario, the electronic device can obtain the progress information of the first content, determine the initial position of the sliding window in the first content according to the progress information, and the sliding window can be used to indicate the position where the electronic device is currently performing TTS processing. The electronic device can control the sliding window to move from the initial position in the first direction (for example, the direction of TTS processing). During the movement of the sliding window, the electronic device can obtain the text in the sliding window in real time, send the text to the network device to obtain the corresponding audio, and play the audio. At the same time, the electronic device can also recognize the illustrations in the sliding window and display the recognized illustrations on the display screen in the display order (for example, the order from front to back). This method can synchronously display the illustrations related to the text being read during the audio playback, solve the problem of inconsistent text and pictures during audiobook listening, improve the functional availability, and enhance the user experience. Moreover, this method can be applied to any first content that has not been pre-produced, without the need to spend additional labor costs and time costs to pre-produce the first content in advance, expand the range of first content that users can choose, meet the diverse needs of users, and further enhance the user experience. In addition, during real-time audiobook listening, the electronic device does not need to remain connected to the network in real time, reducing the delay of illustration loading, reducing the number of network calls, saving traffic costs, and further ensuring the effect of synchronously displaying illustrations.

[0186] Next, the application scenarios involved and the schematic diagram of the user interface in this scenario will be introduced in combination with Figure 6 the shown implementation manners.

[0187] Scenario 1: When the electronic device 100 recognizes that the sliding window includes an illustration and the illustration has not been displayed on the display screen, the electronic device 100 can display the illustration on the display screen. For specific examples, please refer to Figure 7 (A)- Figure 7 (F) of

[0188] As shown in Figure 7 (A) of Figure 6 , assuming that the progress information in the first content corresponds to position 1, the electronic device 100 can use position 1 as the position of the sliding window, that is, the sliding window in the first content is located at position 1, and at this time, the sliding window includes illustration 1. The electronic device 100 can execute Figure 6 S201 of Figure 6 , that is, the electronic device 100 can obtain the audio 1 corresponding to the text 1 in the sliding window. At the same time, the electronic device 100 can execute Figure 6In S204 - S205, that is, obtain Illustration 1, play Audio 1 and display Illustration 1 on the display screen. For a specific example, refer to Figure 7 the user interface 610 shown in (B) of

[0189] such as Figure 7 As shown in (B) of Figure 7 , the user interface 610 can be an audiobook interface of the first application. The user interface 610 can include a picture 611, a preview picture 612A, a preview picture 612B, text content 613, a progress bar 614, and time 615. Among them, the picture 611 can correspond to Figure 7 Illustration 1 in (A) of Figure 7 . The position where the picture 611 is located can be the display position of the illustration in the audiobook scenario. This display position remains unchanged, but the displayed content (i.e., the picture) can change. The preview picture 612A can be Figure 2 a preview of Illustration 1 in (A) of Figure 7 . The preview picture 612A is in a selected state, which can represent that Illustration 1 is currently being displayed. The preview picture 612B can be Figure 7 a preview of

[0190] In one implementation, in Figure 7 (A) and Figure 7 (B) of Figure 6In S207, the sliding window is moved in the first direction (e.g., the direction of TTS processing) by one step. Assume that the moved sliding window is at position 2, as shown in Figure 7 (C) of this figure shows that there is no illustration in the sliding window at this time. The electronic device 100 can execute Figure 6 In S201, the electronic device 100 obtains the audio 2 corresponding to the text 2 within the sliding window. At the same time, the electronic device 100 can execute Figure 6 In S202, and the judgment result is that there is no illustration. At this time, the electronic device 100 can execute Figure 6 In S206, that is, play the audio 2 and keep the illustration 1 displayed on the display screen unchanged. At this time, the electronic device 100 can display Figure 7 The user interface 620 shown in (D) of this figure.

[0191] As shown in Figure 7 (D) of this figure shows that the user interface 620 is similar to the user interface 610 shown in Figure 7 (B) of this figure. For example, the picture 611 displayed in the user interface 620 is the same as the picture 611 displayed in the user interface 610, that is, the picture 611 (corresponding to the illustration 1) is kept displayed. The difference is that the text content 623 in the user interface 620 is different from the text content 613 in the user interface 610, and the progress bar 624 and time 625 in the user interface 620 are different from the progress bar 614 and time 615 in the user interface 610. The text content 623 may include the text 6231, the text 6232, and the text box 6233. The text 6231 (including the character "You must persevere in practicing yoga") may correspond to Figure 7 The text 3 in (C) of this figure. The text 6231 (corresponding to the text 3) is in the state of being read / played. The text 6232 (including the character "At the beginning, you can do it two or three times a week") may correspond to Figure 7 The text 4 in (C) of this figure. The text 6232 (corresponding to the text 3) is framed by the text box 6233, which can indicate that the text 6232 is the text corresponding to the audio 2 currently being played by the electronic device 100, and the text 6232 is related to the picture 611. The progress bar 624 and time 625 (including the character "02:13") can indicate the progress of the audio currently being played by the electronic device 100, that is, the electronic device 100 has played the audio for 2 minutes and 13 seconds.

[0192] In one implementation, after Figure 7 (C) of this figure and Figure 7 (D) of this figure, the electronic device 100 can execute Figure 6 In S207, that is, the sliding window is moved again in the first direction (e.g., the direction of TTS processing) by one step. Assume that the moved sliding window is at position 3, as shown in Figure 7 (E) of this figure shows that there is an illustration in the sliding window at this time.Figure 2 。The electronic device 100 can execute Figure 6 S201, that is, obtain the audio 3 corresponding to the text 3 within the sliding window. Meanwhile, the electronic device can execute Figure 6 S202, and recognize the identifier included in the sliding window, and then execute Figure 2 S203, and determine that the insertion Figure 6 has not been displayed on the display screen. Therefore, the electronic device 100 can execute Figure 2 S204 - S205, that is, obtain the insertion Figure 6 , play the audio 3 and display the insertion Figure 2 on the display screen. For a specific example, reference can be made to Figure 2 the user interface 630 shown in (F) of Figure 7 .

[0193] As Figure 7 shown in (F) of Figure 7 , the user interface 630 may include a picture 631, text content 633, a progress bar 634, and time 635. Among them, the picture 631 may correspond to Figure 2 the insertion in (E) of Figure 2 . The preview picture 612B is in a selected state, which can indicate that the current insertion Figure 7 is being displayed. The text content 633 may include text 6331, text 6332, and a text box 6333. The text 6331 (including the character "Go and try to raise a pet dog") may correspond to Figure 7 the text 5 in (E) of

[0194] the text 6 in (E) of Figure 7When the electronic device 100 receives a user operation acting on the progress bar 614 in the user interface 610 as shown in (B), assuming the user operation is a touch operation, in response to this user operation, the electronic device 100 can adjust the progress of the playing audio. Suppose after adjusting the progress, the electronic device 100 displays Figure 7 the user interface 630 as shown in (F) of. The progress of playing audio, the displayed text, and the illustrations in the user interface 630 are all different from those in the user interface 610. It can be understood that when the electronic device 100 does not receive a user operation acting on the progress bar of the audio, the electronic device 100 can move the sliding window from Figure 7 the position 1 shown in (A) of to Figure 7 the position 2 shown in (C) of, Figure 7 the position 3 shown in (E) of. The display screen also sequentially displays the user interface 610, Figure 7 the user interface 620 shown in (D) of, and the user interface 630. When the electronic device 100 receives a user operation acting on the progress bar of the audio, the electronic device 100 can directly move the sliding window from position 1 to position 3 (without moving to position 2). The display screen does not display the user interface 620, but directly switches from the user interface 610 to display the user interface 630.

[0195] Scenario 2: When the electronic device 100 recognizes that the sliding window includes multiple illustrations and none of these multiple illustrations have been displayed on the display screen, the electronic device 100 can display the illustration at the forefront position on the display screen. For specific examples, refer to Figure 9 (A) of and Figure 9 (B) of.

[0196] As Figure 9 shown in (A) of, the sliding window in the first content is located at position 4 (assuming the starting edge of the sliding window corresponds to the current playing progress of the electronic device 100). At this time, the sliding window includes the inserted Figure 3 and the inserted Figure 4 . The electronic device 100 can execute Figure 6 S201 of, that is, the electronic device 100 can obtain the audio 4 corresponding to the text 4 within the sliding window. At the same time, the electronic device 100 can execute Figure 6 S202 of, and recognize that the sliding window includes the identifier of the inserted Figure 3 and the identifier of the inserted Figure 4 . Then execute Figure 6 S203 of, and determine that neither the inserted Figure 3 nor the inserted Figure 4 has been displayed on the display screen. Therefore, the electronic device 100 can execute Figure 6 S204 - S205 of. Since the position of the inserted Figure 3 is located at the inserted Figure 4Before the position, the electronic device 100 can obtain the inserted Figure 3 , play the audio 4 and display the inserted Figure 3 , for specific examples, please refer to Figure 9 The user interface 910 shown in (B) of

[0197] Such as Figure 9 As shown in (B) of , the user interface 910 can be the audiobook interface of the first application. The user interface 910 can include a picture 911, a preview picture 912A, a preview picture 912B, text content 913, a progress bar 914, and time 915. Among them, the picture 911 can correspond to Figure 9 The inserted Figure 3 in (A) of . The position where the picture 911 is located can be the display position of the illustration in the audiobook scenario. This display position remains unchanged, but the display content (i.e., the picture) can be changed. The preview picture 912A can be Figure 9 The preview picture of the inserted Figure 3 in (A) of . The preview picture 912A is in the selected state, which can represent that the inserted Figure 3 is currently being displayed. The preview picture 912B can be Figure 9 The preview picture of the inserted Figure 4 in (A) of . The text content 913 can include text 9131 and a text box 9132. The text content 913 is related to the picture 911. The text 9131 (including the characters "cultivate one's morality and refine one's temperament, communication skills") can correspond to Figure 9 The text 7 in (A) of . The text 9131 (corresponding to the text 7) is framed by the text box 9132, which can represent that the text 9131 is the text corresponding to the audio 4 currently being played by the electronic device 100. The text 9131 is related to the picture 911. The progress bar 914 and time 915 (including the characters "02:55") can indicate the progress of the audio currently being played by the electronic device 100, that is, the electronic device 100 has played 2 minutes and 55 seconds of audio. It can be understood that the current playback progress of the electronic device 100 corresponds to the starting edge of the sliding window.

[0198] Please refer to Figure 8 , Figure 8 is a schematic flowchart of another audio playback method provided by an embodiment of the present application. This method can be applied to Figure 1A The audio playback system 10 shown in Figure 1B The audio playback system 10 shown in Figure 2 The electronic device 100 shown in Figure 3 The electronic device 100 shown in . This method can include but is not limited to the following steps:

[0199] S301: The electronic device 100 obtains the audio corresponding to the text within the sliding window.

[0200] In one embodiment, before S301, it may further include Figure 5 S101 - S102.

[0201] In one embodiment, the description of S301 and Figure 5 the description of S103 in which the electronic device 100 obtains the audio corresponding to the text within the sliding window are similar. For the specific description in S103, reference can be made to Figure 5 the specific description in S103, which will not be elaborated here.

[0202] S302: The electronic device 100 determines whether the sliding window contains an illustration.

[0203] In one embodiment, the description of S302 and Figure 6 the description of S202 are similar. For the specific description in S202, reference can be made to Figure 6 the specific description in S202. When it is determined that the sliding window contains an illustration, the electronic device 100 may execute S303; when it is determined that the sliding window does not contain an illustration, the electronic device 100 may execute S306.

[0204] S303: The electronic device 100 determines whether there is an illustration that has not been displayed in the sliding window.

[0205] In one embodiment, the description of S303 and Figure 6 the description of S203 are similar. For the specific description in S203, reference can be made to Figure 6 the specific description in S203. When there is an illustration that has not been displayed, the electronic device 100 may execute S304; when there is no illustration that has not been displayed (i.e., there is an illustration that has been displayed), the electronic device 100 may execute S306.

[0206] S304: The electronic device 100 determines whether there is an illustration that has been displayed in the sliding window.

[0207] In one embodiment, when there is an illustration that has been displayed, the electronic device 100 may determine that the sliding window includes an illustration that has been displayed on the display screen and an illustration that has not been displayed on the display screen. At this time, the electronic device 100 may execute S305; when there is no illustration that has been displayed, the electronic device 100 may determine that the sliding window only includes an illustration that has been displayed on the display screen. At this time, the electronic device 100 may execute S307 - S308.

[0208] S305: The electronic device 100 determines whether the first display duration is less than the first preset duration.

[0209] In one embodiment, after the electronic device 100 determines that the sliding window includes illustrations that have been displayed on the display screen and illustrations that have not been displayed on the display screen, the electronic device 100 may obtain the first display duration of the second illustration, where the second illustration may be the illustration currently displayed on the display screen of the electronic device 100.

[0210] In one embodiment, the electronic device 100 may determine whether the first display duration is less than a first preset duration. When the first display duration is less than the first preset duration, the electronic device 100 may execute S306; when the first display duration is greater than or equal to the first preset duration, the electronic device 100 may execute S307 - S308.

[0211] In one embodiment, the first preset duration may be determined by the electronic device 100 according to the number of lines / paragraphs, number of words, and number of sentences of the text related to the second illustration in the sliding window.

[0212] S306: The electronic device 100 plays audio and keeps the illustration displayed on the display screen unchanged.

[0213] In one embodiment, after the electronic device 100 obtains the audio, it may play the audio through the speaker and keep the illustration currently displayed on the display screen unchanged. Without limitation, when the display screen of the electronic device 100 does not display an illustration, the electronic device 100 may keep the display screen without displaying an illustration.

[0214] S307: The electronic device 100 obtains a third illustration.

[0215] In one embodiment, the description of the third illustration is Figure 6 similar to the description of the first illustration in S204, and reference may be made to Figure 6 the specific description of the first illustration in S204, which will not be elaborated here.

[0216] Among them, the order of S301 and S302 - S305 is not limited.

[0217] S308: The electronic device 100 plays audio and displays the third illustration.

[0218] In one embodiment, after the electronic device 100 obtains the audio and the third illustration, it may play the audio through the speaker and display the third illustration on the display screen. It can be understood that the electronic device 100 updates the display of the third illustration while playing the audio.

[0219] In one embodiment, when the electronic device 100 executes S305 and then executes S307, at this time, the electronic device 100 may switch the displayed second illustration to the third illustration.

[0220] S309: The electronic device 100 moves the sliding window in accordance with a second preset step.

[0221] In one implementation, the description of S309 is Figure 6 similar to that of S207. For the specific description of S207, see Figure 6 . The electronic device 100 may move the sliding window in accordance with a second preset step and re-execute S301 - S308. At this time, the position of the moved sliding window is different from before, and the text contained in the moved sliding window is also different from before.

[0222] In Figure 8 the method shown, when the sliding window contains both displayed illustrations and non-displayed illustrations, it is possible to determine whether to switch to a non-displayed illustration based on the display duration of the displayed illustration. In this way, when multiple illustrations in the sliding window are close to each other, it is possible to ensure that each illustration has sufficient time (e.g., 10 seconds - 30 seconds) to be displayed on the display screen, and the display duration of the illustration is determined based on the text related to the illustration, thus further enhancing the effect of synchronous display.

[0223] Next, the application scenarios involved and the schematic diagram of the user interface in this scenario will be introduced in combination with Figure 8 the implementation shown.

[0224] Scenario 3: When the electronic device 100 recognizes that the sliding window includes multiple illustrations, and these multiple illustrations include both illustrations that have been displayed on the display screen and illustrations that have not been displayed on the display screen, the electronic device 100 may determine whether to switch the displayed illustration based on the display duration of the displayed illustration. For specific examples, see Figure 9 (C)- Figure 9 (F).

[0225] Exemplarily, after (A) and Figure 9 (B) shown in Scenario 2, the electronic device 100 may execute Figure 9 S309, that is, the electronic device 100 may move the sliding window in a first direction (e.g., the direction of TTS processing) in accordance with step 2. Assume that the moved sliding window is located at position 5, as shown in Figure 8 (C). At this time, the sliding window includes illustration Figure 9 and illustration Figure 3 and illustration Figure 4 . The electronic device 100 may execute Figure 8 S301, that is, obtain the audio 5 corresponding to the text 5 within the sliding window. At the same time, the electronic device 100 may execute Figure 8 S302, and recognize that the sliding window includes the identifier of illustration Figure 3 and the identifier of illustration Figure 4 , and then executeFigure 8 of S303, and determine the inserted Figure 4 has not been displayed on the display screen, and then execute Figure 8 of S304, and determine the inserted Figure 3 has been displayed on the display screen. At this time, the electronic device 100 can execute Figure 8 of S305, and determine the inserted Figure 3 The display duration 1 is less than the preset duration 1. Therefore, the electronic device 100 can execute Figure 8 of S306, that is, play the audio 5 and keep the inserted displayed on the display screen Figure 3 unchanged. At this time, the electronic device 100 can display Figure 9 the user interface 920 shown in (D).

[0226] Such as Figure 9 shown in (D), the user interface 920 is similar to the user interface 910 shown in Figure 9 (A). For example, the picture 911 displayed in the user interface 920 is the same as the picture 911 displayed in the user interface 910, that is, keep displaying the picture 911 (corresponding to the inserted Figure 3 ). The difference is that the user interface 920 may include text content 923, a progress bar 924, and time 925. The text content 923 may include text 9231, text 9232, and a text box 9233. The text 9231 (including the character "known as 'fashionable and elegant sports'") may correspond to Figure 9 the text 8 in (C). The text 9231 (corresponding to the text 8) is framed by the text box 9233, which can indicate that the text 9231 is the text corresponding to the audio 5 currently being played by the electronic device 100, and the text 9231 is related to the picture 911. The text 9232 (including the character "Dance is a performing art") may correspond to Figure 9 the text 9 in (C). The text 9232 is in the unread / played state. The progress bar 924 and time 925 (including the character "02:58") can indicate the progress of the audio currently being played by the electronic device 100, that is, the electronic device 100 has played 2 minutes and 58 seconds of audio.

[0227] In one embodiment, after Figure 9 (C) and Figure 9 (D), the electronic device 100 can execute Figure 8 of S309, that is, the electronic device 100 can move the sliding window in the first direction again by one step. Assume that the moved sliding window is located at position 6. As Figure 9 shown in (E), at this time, the sliding window includes the inserted Figure 3 and the inserted Figure 4 . The electronic device 100 can execute Figure 8S301, that is, obtaining the audio 6 corresponding to the text 6 in the sliding window. At the same time, the electronic device 100 can execute Figure 8 S302-S304, and determine the insertion Figure 3 has been displayed on the display, insert Figure 4 has not been displayed on the display screen. At this time, the electronic device 100 can execute Figure 8 S305, and determine the insertion Figure 3 's display duration 2 (assuming that display duration 2 is two seconds longer than display duration 1) is greater than or equal to preset duration 1, so the electronic device 100 can execute Figure 8 S307-S308, that is, get the plug-in Figure 4 , play audio 6 and change the plug-in displayed on the display Figure 3 Switch to Plug Figure 4 , at this time the electronic device 100 can display Figure 9 (F) shows the user interface 930.

[0228] If Figure 9 As shown in (F) of , the user interface 930 may include a picture 931, text content 933, a progress bar 934 and time 935, wherein the picture 931 may correspond to Figure 9 Insert in (E) Figure 4 , the preview image 912B is in the selected state, which indicates that the current display is being inserted Figure 4 . The text 9231 (corresponding to the text 8) in the text content 933 is in the state of being read / played, and the text 9232 (corresponding to the text 9) is framed by the text frame 9331, which can indicate that the text 9232 is the text corresponding to the audio 6 currently being played by the electronic device 100, and the text 9232 is related to the picture 931. The progress bar 934 and the time 935 (including the characters "03:00") can indicate the progress of the audio currently being played by the electronic device 100, that is, the electronic device 100 has played 3 minutes of audio.

[0229] Understandably, in Figure 9 of (C) and Figure 9 (D), the electronic device 100 is playing the text 8 corresponding to the audio 5 (the total duration corresponding to the playing progress is 2 minutes and 58 seconds), and the electronic device 100 displays the insertion Figure 3 The duration (display duration 1) is less than the preset duration 1, so the electronic device 100 keeps displaying the plug-in Figure 3 Unchanged. In Figure 9 of (E) and Figure 9 (F), the electronic device 100 is playing the text 9 corresponding to the audio 6 (the total duration corresponding to the playing progress is 3 minutes), and the electronic device 100 displays the insertion Figure 3The duration (display duration 2) is greater than or equal to the preset duration 1, so the electronic device 100 will display the inserted Figure 3 Switch to inserted Figure 4 .

[0230] Please refer to Figure 10 , Figure 10 which is a schematic flowchart of another audio playing method provided by an embodiment of the present application. This method can be applied to Figure 1A the audio playing system 10 shown in Figure 1B the audio playing system 10 shown in Figure 2 the electronic device 100 shown in Figure 3 the electronic device 100 shown in

[0231] S401: The electronic device 100 obtains the audio corresponding to the text within the sliding window.

[0232] In one implementation, before S401, it may further include Figure 5 S101 - S102.

[0233] In one implementation, the description of S401 and Figure 5 the description of the electronic device 100 obtaining the audio corresponding to the text within the sliding window in S103 are similar. For the specific description, please refer to Figure 5 the specific description in S103 and will not be elaborated here.

[0234] S402: The electronic device 100 determines whether the sliding window contains an illustration.

[0235] In one implementation, the description of S402 and Figure 6 the description of S202 are similar. For the specific description, please refer to Figure 6 the specific description in S202. When it is determined that the sliding window contains an illustration, the electronic device 100 may execute S403; when it is determined that the sliding window does not contain an illustration, the electronic device 100 may execute S407.

[0236] S403: The electronic device 100 determines the trigger display position of the illustration.

[0237] In one implementation, the trigger display position of the illustration can be used to trigger the display screen to update and display the illustration.

[0238] In one embodiment, the electronic device 100 may determine the trigger display position of the illustration according to the number of lines / paragraphs, the number of words, and the number of sentences of the text related to the illustration in the sliding window. In some examples, assuming that two lines of text before the position of the illustration and three lines of text after the position of the illustration are both related to the illustration, the electronic device 100 may determine the positions of the above two lines of text as the trigger display position of the illustration.

[0239] In one embodiment, the sliding window may include multiple illustrations, and each illustration may correspond to a trigger display position. Optionally, the multiple trigger display positions corresponding to the multiple illustrations are different from each other.

[0240] S404: The electronic device 100 determines whether the current playback progress is at the trigger display position of the fourth illustration.

[0241] In one embodiment, the current playback progress may be the position of the text corresponding to the audio currently played by the electronic device 100 in the sliding window. For example, the current playback progress may correspond to the starting edge of the sliding window. For the description of the positional relationship between the current playback progress and the starting edge of the sliding window, reference may be made to Figure 5 the relevant description in S102 of Figure 10 which will not be elaborated here. For the sake of convenience of description,

[0242] In one embodiment, the fourth illustration may be any illustration in the sliding window. When the starting edge of the sliding window (i.e., the current playback progress of the electronic device 100) is at the trigger display position of the fourth illustration, the electronic device 100 may execute S405; when the starting edge of the sliding window is not at the trigger display position of the fourth illustration, the electronic device 100 may execute S407.

[0243] S405: The electronic device 100 obtains the fourth illustration.

[0244] In one embodiment, the electronic device 100 may obtain a fourth illustration identifier, parse the fourth illustration identifier, and obtain the parsed fourth illustration.

[0245] S406: The electronic device 100 plays the audio and displays the fourth illustration.

[0246] In one embodiment, after the electronic device 100 obtains the audio and the fourth illustration, it may play the audio through the speaker and display the fourth illustration through the display screen. It can be understood that the electronic device 100 updates the display of the fourth illustration while playing the audio.

[0247] S407: The electronic device 100 plays the audio and keeps the illustration displayed on the display screen unchanged.

[0248] In one embodiment, the description of S407 is similar to that of Figure 6 S206, and reference may be made to the specific description of S306 in Figure 8 , which will not be elaborated here.

[0249] S408: The electronic device 100 moves the sliding window according to a third preset step.

[0250] In one embodiment, the description of S408 is similar to that of Figure 6 S207, except that the third preset step may also be related to the current playback progress. The electronic device 100 can move the sliding window according to the third preset step and re - execute S401 - S407. At this time, the position of the moved sliding window is different from before, and the text contained in the moved sliding window is also different from before.

[0251] In Figure 10 the method shown, when the sliding window contains an illustration, the electronic device does not directly display the contained illustration, but further determines whether the current playback progress (e.g., the starting edge of the sliding window) is located at the trigger display position of the illustration. This method precisely controls the moment of illustration display, ensures that the displayed illustration matches the text, further improves the effect of synchronous illustration display, improves the functional usability, and improves the user experience.

[0252] Next, the application scenarios involved and the schematic diagram of the user interface in the scenario will be introduced in combination with the Figure 10 shown embodiment.

[0253] Scenario 4: When the electronic device 100 recognizes that the sliding window includes an illustration, the electronic device 100 can display the illustration according to the trigger display position of the illustration. For specific examples, reference may be made to Figure 11 (A) - Figure 11 (F) in

[0254] As shown in Figure 11 (A), the sliding window in the first content is located at position 7 (assuming that the starting edge of the sliding window corresponds to the current playback progress of the electronic device 100), and at this time, the sliding window does not include an illustration. The electronic device 100 can execute Figure 10 S401 in Figure 10 , that is, the electronic device 100 can obtain the audio 7 corresponding to the text 7 in the sliding window. At the same time, the electronic device 100 can execute Figure 10 S402 in Figure 5 and recognize the identifier indicating that the sliding window does not include an illustration, and then execute Figure 11The user interface 1110 shown in (B).

[0255] As Figure 11 Shown in (B), the user interface 1110 can be the audiobook interface of the first application. The user interface 1110 can include a picture 1111, a preview picture 1112A, a preview picture 1112B, text content 1113, a progress bar 1116, and a time 1117. Among them, the picture 1111 can correspond to Figure 11 The illustration in (A) of Figure 5 , the position where the picture 1111 is located can be the display position of the illustration in the audiobook scenario. This display position remains unchanged, but the display content (i.e., the picture) can be changed. The preview picture 1112A can be Figure 11 The illustration in (A) of Figure 5 The preview of, the preview picture 1112A is in the selected state, which can represent that the current illustration being displayed is Figure 5 , the preview picture 1112B can be Figure 11 The illustration in (A) of Figure 6 The preview of. The text content 1113 can include text 1114 and a text box 1115. The text 1114 (including the character "Spring plants begin to germinate and grow") can correspond to Figure 11 The text 10 in (A) of, the text 1114 (corresponding to the text 10) is framed by the text box 1115, which can represent that the text 1114 is the text corresponding to the audio 7 currently being played by the electronic device 100, and the text 1114 is related to the picture 1111. The progress bar 1116 and the time 1117 (including the character "01:00") can indicate the progress of the audio currently being played by the electronic device 100, that is, the electronic device 100 has played 1 minute of audio.

[0256] In one implementation, after Figure 11 In (A) of and Figure 11 In (B) of, the electronic device 100 can execute Figure 10 S408 of, that is, move the sliding window in the first direction (for example, the direction of TTS processing) by one step. Assume that the moved sliding window is located at position 8, as Figure 11 Shown in (C) of, at this time, the sliding window includes an illustration Figure 6 . The electronic device 100 can execute Figure 10 S401 of, that is, obtain the audio 8 corresponding to the text 8 in the sliding window. At the same time, the electronic device 100 can execute Figure 10 S402 of, and recognize that the sliding window includes the identifier of an illustration Figure 6 , and then execute Figure 10 S403 of, that is, determine the trigger display position of the illustration Figure 6 (assume it is Figure 11 Position 10 in (C) of), and then execute Figure 10of S404, and determine the current playback progress (for example Figure 11 the position of the text 11 corresponding to the starting edge of the sliding window in (C) of Figure 6 is not located at the trigger display position (position 10). Therefore, the electronic device 100 can execute Figure 10 of S407, that is, play the audio 8 and keep the inserted Figure 5 displayed on the display screen unchanged. At this time, the electronic device 100 can display Figure 11 the user interface 1120 shown in (D) of

[0257] As Figure 11 shown in (D) of Figure 11 the user interface 1120 is similar to the user interface 1110 shown in (B) of Figure 5 For example, the picture 1111 displayed in the user interface 1120 is the same as the picture 1111 displayed in the user interface 1110, that is, keep displaying the picture 1111 (corresponding to the inserted Figure 11 ). The difference is that the text content 1123 in the user interface 1120 may include the text 1124 and the text box 1125. The text 1124 (including the character "All things have come to this point and gradually recovered") may correspond to Figure 11 the text 11 in (C) of

[0258] In one implementation, Figure 11 after (C) of Figure 11 and (D) of Figure 10 the electronic device 100 can execute Figure 11 of S408, that is, move the sliding window in the first direction again by a step of 3. Assume that the moved sliding window is located at position 9. As Figure 6 shown in (E) of Figure 10 At this time, the sliding window includes the inserted Figure 10 . The electronic device 100 can execute Figure 6 of S401, that is, obtain the audio 9 corresponding to the text 9 in the sliding window. At the same time, the electronic device 100 can execute Figure 10 of S402, and recognize that the sliding window includes the inserted Figure 6 identifier, and then execute Figure 10 of S403, that is, determine the trigger display position (the above position 10) of the inserted Figure 11The position of the text 12 corresponding to the starting edge of the sliding window in (E) is located at the insertion Figure 6 The trigger display position (position 10), so the electronic device 100 can execute Figure 10 Of S406, that is, obtain the insertion Figure 6 , play the audio 9 and display the insertion on the display screen Figure 6 , at this time the electronic device 100 can display Figure 11 The user interface 1130 shown in (F).

[0259] As Figure 11 Shown in (F), the user interface 1130 may include a picture 1131, text content 1133, a progress bar 1136, and a time 1137. Among them, the picture 1131 may correspond to Figure 11 The insertion in (E) Figure 6 , the preview picture 1112B is in the selected state, which can represent that the current insertion is being displayed Figure 6 . The text content 1133 may include text 1134 and a text box 1135. The text 1134 (including the characters "Yang moves upward and Yin is forced downward") may correspond to Figure 11 The text 12 in (E), the text 1134 (corresponding to the text 12) is framed by the text box 1135, which can represent that the text 1134 is the text corresponding to the audio 9 currently being played by the electronic device 100, and the text 1134 is related to the picture 1131. The progress bar 1136 and the time 1137 (including the characters "02:50") can indicate the progress of the audio currently being played by the electronic device 100, that is, the electronic device 100 has played 2 minutes and 50 seconds of audio.

[0260] It can be understood that in Figure 11 Of (C) and Figure 11 Of (D), the electronic device 100 is playing the text 11 corresponding to the audio 8. At this time, the position of the text 11 corresponding to in the sliding window is not located at the trigger display position (position 10) of the insertion Figure 6 , so the electronic device 100 keeps the displayed insertion Figure 5 Unchanged. In Figure 11 Of (E) and Figure 11 Of (F), the electronic device 100 is playing the text 12 corresponding to the audio 9. At this time, the position of the text 12 corresponding to in the sliding window is located at the trigger display position (position 10) of the insertion Figure 6 , so the electronic device 100 obtains the insertion Figure 6 And display the insertion Figure 6 .

[0261] Please refer to Figure 12 , [[ID=2 Is a schematic flowchart of another audio playback method provided by an embodiment of the present application. This method can be applied to ​The audio playback system 10 shown. This method can be applied to ​ The audio playback system 10 shown. This method can be applied to ​ The electronic device 100 shown. This method can be applied to ​ The electronic device 100 shown. This method may include but is not limited to the following steps:

[0262] S501: The first device obtains first progress information of the first content.

[0263] In one embodiment, the first content may be an e-book. When the first device receives a user operation triggering the audiobook function, it may obtain the first progress information of the first content. The first progress information is similar to the description of the progress information in S101, and the description of S501 can be referred to ​ the description of S101, which will not be elaborated here. ​ the description of S101, which will not be elaborated here.

[0264] S502: The first device determines a first playback resource according to the first progress information.

[0265] In one embodiment, the first playback resource may include the first text and the first picture of the first content. The first playback resource is determined according to the text position corresponding to the first progress information.

[0266] In one embodiment, the first device may determine the first playback resource according to the first progress information and a sliding window. The sliding window is an interval in the first content determined according to preset conditions. The preset conditions may include at least one of the following: the number of words of a preset text, the number of sentences of a preset text, the duration corresponding to a preset text. The sliding window is similar to the description of the sliding window in ​ S102, and the description of S502 can be referred to ​ the description of S102, which will not be elaborated here.

[0267] S503: The first device plays the first audio according to the first playback resource and displays the first picture according to the first playback resource.

[0268] In one embodiment, the first audio may be the audio corresponding to the first text. The first device may send the first text to a network device and receive the first audio returned by the network device. The acquisition method of the first audio can be referred to ​ the relevant description in S103, and the description of S503 can be referred to ​ the description of S104, which will not be elaborated here.

[0269] In one embodiment, the description of the first picture can be referred to ​Specific description of the first illustration in S103. The first playback resource may include a first picture and a fourth picture. When the display position of the first picture is before the display position of the fourth picture, the first device may display the first picture.

[0270] In one embodiment, when the current playback progress in the first playback resource is at the trigger display position of the first picture, the first device may display the first picture. For specific description, refer to ​ the description of S404 - S407 in

[0271] S504: The first device obtains the second progress information of the first content.

[0272] In one embodiment, the second progress information is different from the first progress information. For the description of S504, refer to ​ the description of S105 in

[0273] S505: The first device determines a second playback resource according to the second progress information and the first playback resource.

[0274] In one embodiment, the second playback resource includes a second text and a second picture in the first content. The second playback resource is determined according to the text position corresponding to the second progress information, and the second playback resource is different from the first playback resource. For the description of S505, refer to ​ the description of S502 in

[0275] S506: The first device plays a second audio according to the second playback resource and displays a second picture according to the second playback resource.

[0276] In one embodiment, the second audio may be the audio corresponding to the second text. For specific description, refer to ​ the description of the first audio in S503 in

[0277] In one embodiment, for the description of the second picture, refer to ​ the specific description of the first illustration in S103. The second playback resource may include a first picture and a second picture. The first progress information is before the second progress information, and the display position of the first picture is before the display position of the second picture. When the display duration of the first picture is greater than or equal to a preset duration, the first picture displayed by the first device is switched to the second picture; when the display duration of the first picture is less than the preset duration, the first device displays the first picture. For specific description, refer to ​ the description of S305 - S308 in

[0278] In one implementation, after S506, the first device may obtain third progress information of the first content, determine a third playback resource according to the third progress information and the second playback resource, and determine whether the third playback resource includes a picture. When the third playback resource includes a third picture, the second picture displayed on the first device is switched to the third picture, and the second picture and the third picture are different; when the third playback resource does not include a picture, the second picture is displayed.

[0279] In ​ In the method shown, the first device can synchronously display illustrations related to the text being read during the audio playback, solving the problem of inconsistent pictures and texts during the audiobook listening process, improving the functional availability, and enhancing the user experience. Moreover, this method can be applied to any first content that has not been pre-produced, without the need to spend additional labor costs and time costs to pre-produce the first content in advance, expanding the range of first content that users can choose from, meeting the diverse needs of users, and further enhancing the user experience. In addition, when listening to an audiobook in real time, the first device does not need to remain connected to the network in real time, reducing the latency of illustration loading, reducing the number of network calls, and saving traffic costs, further ensuring the effect of synchronously displaying illustrations.

[0280] In the methods provided by the embodiments of the present application, they can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD), etc.). As described above, the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio playback method, characterized in that: Applied to a first device, the method includes: Acquire first progress information of first content, where the first content includes text and at least one picture; determining a first playback resource according to the first progress information, where the first playback resource includes a first text and a first picture in the first content, and both the first text and the first picture are related to the first progress information; Play a first audio according to the first playback resource, display the first picture according to the first playback resource, the first audio being the audio corresponding to the first text; Acquire second progress information of the first content; determining a second playback resource according to the second progress information and the first playback resource, where the second playback resource includes a second text and a second picture in the first content, and both the second text and the second picture are related to the second progress information; Play the second audio according to the second playback resource, display the second picture according to the second playback resource, and the second audio is the audio corresponding to the second text; The first progress information and the second progress information are different.

2. The method according to claim 1, characterized in that The method further comprises: determining whether a third playback resource includes a picture, wherein the third playback resource is determined according to the acquired third progress information of the first content and the second playback resource; When the third playback resource includes a third picture, switching the second picture displayed by the first device to the third picture, where the third picture is different from the second picture; When the third playback resource does not include the picture, the second picture is displayed.

3. The method according to claim 1 or 2, characterized in that The first playback resource is determined according to a text position corresponding to the first progress information, the second playback resource is determined according to a text position corresponding to the second progress information, and the second playback resource is different from the first playback resource.

4. The method according to any one of claims 1 to 3, characterized in that: The first playback resource includes the first picture and the fourth picture, The displaying the first picture according to the first playback resource includes: When the display position of the first picture is before the display position of the fourth picture, the first picture is displayed.

5. The method according to any one of claims 1 to 4, characterized in that: The second playback resource includes the second picture and the first picture, the first progress information is before the second progress information, and the display position of the first picture is before the display position of the second picture; The displaying the second picture according to the second playback resource includes: When the display time of the first picture is greater than or equal to a preset time, switching the first picture displayed by the first device to the second picture; The method further comprises: When the display time of the first picture is less than the preset time, the first picture is displayed.

6. The method according to claim 1, characterized in that The displaying the first picture according to the first playback resource includes: When the current playback progress in the first playback resource is located at a trigger display position of the first picture, the first picture is displayed.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire the first audio and the first picture according to the first playback resource; The first audio and the first picture are sent to a second device, where the first audio is played by the second device and the first picture is displayed by the second device.

8. The method according to claim 2, characterized in that The method further comprises: When the second playback resource includes the second picture, sending a first message to the second device, where the first message indicates updating the displayed picture; receiving a second message sent by the second device, where the second message is used to request to obtain an updated picture; The second picture is sent to the second device according to the second message, and the second picture is used for display by the second device.

9. The method according to any one of claims 1 to 8, characterized in that The determining the first playback resource according to the first progress information includes: The first playback resource is determined according to the first progress information and a sliding window, where the sliding window is an interval in the first content determined according to preset conditions, and the preset conditions include at least one of the following: the number of words in a preset text, the number of sentences in a preset text, and the duration corresponding to the preset text.

10. The method according to any one of claims 1 to 9, characterized in that: The method further comprises: Sending the first text to a network device; The first audio sent by the network device is received, where the first audio is obtained by the network device according to the first text.

11. The method according to claims 1 to 10, characterized in that The first content is an e-book, and the obtaining of first progress information of the first content includes: When a user operation for triggering the audiobook function is received, the first progress information of the first content is acquired.

12. An electronic device, characterized in that: The method comprises a transceiver, a processor and a memory, wherein the memory is used to store a computer program, and the processor calls the computer program to execute the method according to any one of claims 1 to 11.

13. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images