A reading companion device and computer readable storage medium

Abstract

This disclosure discloses a reading companion device and a computer-readable storage medium. The reading companion device includes: an image acquisition device, an audio acquisition device, an audio playback device, a processor, and a memory. The memory stores program instructions for reading companionship. When the program instructions are executed by the processor, the reading companion device performs the following operations: in response to detecting a start reading companion signal, it acquires an image of the current reading page through the image acquisition device; based on the image, it generates text-to-speech audio using a pre-trained model as reading companion audio; and it plays the reading companion audio through the audio playback device to achieve reading companionship for the current reading page. Using the solution disclosed herein, reading companionship can be provided for the current reading page of any reading material, exhibiting good compatibility and reducing production and purchase costs while saving storage resources.

Description

Technical Field

[0001] This disclosure generally relates to the field of computer vision technology. More specifically, this disclosure relates to a reading companion device and a computer-readable storage medium. Background Technology

[0002] In the early stages of children's education, many parents use picture books, storybooks, and other illustrated reading materials to entertain or educate their children. This process requires parents to interpret, explain, and guide, which places high demands on them. Without parental cooperation, the reading materials are less attractive to children, making it difficult for them to persist in learning for extended periods. This leads to situations where learning outcomes are poor without parental involvement.

[0003] In response to this situation, many devices have emerged on the market that can play reading materials to provide reading companionship for children, replacing parental involvement. However, existing devices have many limitations on the reading materials. Some require the materials to be in a specially designed format, such as having raised dots, while others require them to be pre-stored, for example, in the device's accessible storage space. In short, existing devices can only play reading companionship for specific types of reading materials; they cannot play reading companionship for ordinary reading materials that children are interested in but do not meet the limitations, resulting in poor compatibility. Furthermore, requiring specially designed reading materials leads to higher production and purchase costs, and requiring pre-stored materials results in greater storage consumption.

[0004] In view of this, there is an urgent need to provide a solution for accompanying reading, so as to enable accompanying reading of ordinary reading materials, thereby improving compatibility, reducing production and purchase costs, and saving storage resources. Summary of the Invention

[0005] In order to at least address one or more of the technical problems mentioned above, this disclosure proposes solutions for reading companions in the following aspects.

[0006] In a first aspect, this disclosure provides a reading companion device, comprising: an image acquisition device, an audio acquisition device, an audio playback device, a processor, and a memory; the memory stores program instructions for reading companionship, which, when executed by the processor, cause the reading companion device to perform the following operations: in response to detecting a start reading companion signal, acquiring an image of the current reading page through the image acquisition device; generating text-to-speech audio as reading companion audio based on the image using a pre-trained model; and playing the reading companion audio through the audio playback device to achieve reading companionship for the current reading page.

[0007] In some embodiments, when the program instructions are executed by the processor, the reading aid device further performs the following operations to determine that a start reading aid signal has been detected: if the audio content acquired by the audio acquisition device contains a preset prompt word, the start reading aid signal is determined to be detected, wherein the preset prompt word is used to prompt the reading aid device to start reading aid; or if the sensing value detected by a preset sensor meets a preset condition, the start reading aid signal is determined to be detected.

[0008] In other embodiments, when the program instructions are executed by the processor, the accompanying reading device further performs the following operations to generate accompanying reading audio: plays a sound selection prompt audio through the audio playback device; acquires the sound selection result fed back by the sound selection prompt audio through the audio acquisition device, and obtains the target sound features based on the sound selection result; and generates text reading audio as accompanying reading audio using the pre-trained model based on the image and the target sound features.

[0009] In some other embodiments, when the program instructions are executed by the processor, the accompanying reading device further performs the following operations to generate accompanying reading audio: generating scene simulation audio using the pre-trained model; and synthesizing audio using a speech synthesis model based on the text reading audio and the scene simulation audio to obtain a mixed audio as the accompanying reading audio.

[0010] In some other embodiments, when the program instructions are executed by the processor, the accompanying reading device further performs the following operations to provide accompanying reading for the current reading page: generating guided audio using the pre-trained model; and playing the guided audio through the audio playback device after the accompanying audio has been played.

[0011] In some other embodiments, when the program instructions are executed by the processor, the accompanying reading device further performs the following operations to provide accompanying reading for the current reading page: in response to acquiring a response result based on the guided reading audio through the audio acquisition device within a preset time period, generating a response evaluation audio using the pre-trained model based on the response result; and playing the response evaluation audio through the audio playback device.

[0012] In some other embodiments, when the program instructions are executed by the processor, the accompanying reading device further performs the following operations to provide accompanying reading for the current reading page: in response to the failure to collect a response result based on the guided reading audio feedback through the audio acquisition device within a preset time period, generating reference audio using the pre-trained model; playing the reference audio through the audio playback device, and setting the number of guided readings to 0.

[0013] In a second aspect, this disclosure provides a computer-readable storage medium having stored program instructions thereon for reading, which, when executed by one or more processors, perform operations performed by the apparatus as described in one or more embodiments of the first aspect above.

[0014] The accompanying reading solution provided above allows for the generation of accompanying audio based on the image of the current reading page of any reading material, using a pre-trained model. Playing this audio enables accompanying reading of any material, demonstrating good compatibility. This disclosed solution does not limit whether the reading material is in a specially designed format or pre-stored, reducing production and purchasing costs and saving storage resources. Furthermore, in some embodiments, the accompanying audio generated by the pre-trained model is not simply text-to-speech audio, but a mixture of text-to-speech audio and scene simulation audio, providing listeners with an immersive reading experience and enhancing the appeal of the accompanying audio to children, thus increasing product stickiness. Attached Figure Description

[0015] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0016] Figure 1 This is an exemplary structural block diagram illustrating a companion reading device according to an embodiment of this disclosure;

[0017] Figure 2 This is an exemplary flowchart illustrating the operations performed by the companion reading device according to an embodiment of this application;

[0018] Figure 3 This is an exemplary flowchart illustrating a reading companion method implemented based on a reading companion system according to an embodiment of this disclosure;

[0019] Figure 4 This is an exemplary flowchart illustrating a reading companion method implemented based on a distributed reading companion system according to an embodiment of this disclosure. Detailed Implementation

[0020] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0021] It should be understood that when the terms "first," "second," "third," and "fourth," etc., are used in the claims, specification, and drawings of this application, they are used only to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0022] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0023] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0024] Figure 1 This is an exemplary structural block diagram illustrating a reading companion device 100 according to an embodiment of this application. Figure 1As shown, the companion reading device 100 may include a processor 101, a memory 102, an image acquisition device 103, an audio acquisition device 104, and an audio playback device 105. The processor 101 may include, for example, a general-purpose processor (“CPU”) or a dedicated graphics processor (“GPU”), and the memory 102 stores program instructions executable on the processor. In some embodiments, the memory 102 may include, but is not limited to, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), and enhanced dynamic random access memory (EDRAM).

[0025] The aforementioned audio acquisition device 104 may include at least one microphone, which can be arranged at any suitable location on the reading device, and can be arranged centrally or distributed, symmetrically or asymmetrically, as long as the purpose of audio acquisition is met. Similarly, the aforementioned audio playback device 105 may include at least one speaker, which can also be arranged at any suitable location on the reading device 100, and can be arranged centrally or distributed, symmetrically or asymmetrically, as long as the purpose of audio playback is met.

[0026] The aforementioned image acquisition device 103 may include at least one camera, which may be positioned to conveniently capture an image of the entire reading page, such as the high end of the reading aid device 100 (i.e., the end farther from the reading page when the reading aid device is in normal use). Alternatively or additionally, one of the at least one cameras may be positioned at the low end of the reading aid device 100 (i.e., the end closer to the reading page when the reading aid device is in normal use). This low-end camera can perform individual image scanning of the text on the reading page to obtain more accurate reading text data, thereby assisting the pre-trained model in generating more accurate and higher-quality accompanying audio.

[0027] Furthermore, the aforementioned memory 102 may store program instructions for accompanying reading. When the program instructions are executed by the processor, the accompanying reading device 100 performs the following operations: upon detecting a start accompanying reading signal, it acquires an image of the current reading page using the image acquisition device 103; based on the image, it generates text reading audio using a pre-trained model as accompanying reading audio; and it plays the accompanying reading audio using the audio playback device 105 to achieve accompanying reading of the current reading page.

[0028] Furthermore, the reading aid device 100 may perform the following operations to determine that a start reading aid signal has been detected: if the audio content acquired by the audio acquisition device 104 contains a preset prompt word, the start reading aid signal is determined to be detected; or if the sensing value detected by the preset sensor meets a preset condition, the start reading aid signal is determined to be detected.

[0029] The aforementioned preset prompts are used to prompt the reading companion device to start reading aloud. The specific content of the preset prompts can be set according to actual needs, such as "Start reading companion", "Activate reading companion", "Enter reading companion", "Read for me", "Read for me", "Speak for me", and "Speak for me", etc. This disclosure does not make specific limitations in this regard. In actual operation, the reading companion device 100 can perform speech recognition and analysis on the audio content collected by the audio acquisition device 104, and can determine that a start reading companion signal has been detected when the audio content contains the preset prompts.

[0030] The aforementioned preset sensors can be pressure sensors and / or distance sensors, and can be arranged at the tip of the pen in the reading device 100 so that when the reading device comes into contact with the reading page, the preset sensors can be triggered to generate a start reading signal. The sensed values ​​detected by the aforementioned preset sensors meet preset conditions, for example, the pressure detected by the pressure sensor is greater than a pressure threshold and / or the distance detected by the distance sensor is less than a distance threshold. It is understood that those skilled in the art can select specific values ​​for the pressure threshold and distance threshold according to actual needs, and this disclosure does not impose specific limitations in this regard.

[0031] The aforementioned current reading page can contain only images, or it can contain both images and text. If the current reading page contains only images, the pre-trained model can generate audio for text reading based on the images in the current reading page's images. If the current reading page contains both images and text, the pre-trained model can generate audio for text reading based on both the images and text in the current reading page's images.

[0032] In addition, when the current reading page contains both images and text, besides capturing the entire image of the current reading page, it's also possible to scan the text within the current reading page to obtain scanned text images. Subsequently, the pre-trained model can generate audio for text reading aloud based on the scanned text images and the entire image of the current reading page.

[0033] The aforementioned pre-trained model can be a large model capable of directly generating accompanying audio from images, such as the Im2Wav model, the VITA model, or the Image to Music V2 model. In this case, the pre-trained model can first analyze the image to obtain accompanying data, and then generate accompanying audio based on the accompanying data. Alternatively, the pre-trained model can be a combination of two large models (which can be referred to as the first model and the second model for distinction). Here, the first model can be used to analyze the image to obtain accompanying data, and the second model can be used to generate accompanying audio based on the accompanying data. In this case, the first model can employ models such as Convolutional image captioning, Image-Textualization, and ClipCap. The second model can employ models such as AudioLDM, Make-An-Audio, and Auffusion.

[0034] The aforementioned accompanying reading data may include at least story text, scene information, and atmosphere information. Story text is used to provide a complete description of the events in the image using natural language. Scene information may include the time, location, and / or weather of the event. Here, time may be, for example, a year, season, specific holiday, or a moment in time; location may be, for example, a beach, forest, riverbank, park, hospital, school, or airport; and weather may be, for example, a sudden downpour, light rain, heavy snow, or thunder and lightning. Atmosphere information describes the atmosphere created by the story, such as relaxed and soothing, eerie and terrifying, or cheerful and lively. It is understood that the various times, locations, weather, and atmospheres listed here are merely exemplary and not limiting; those skilled in the art may add or remove appropriate options as needed.

[0035] After obtaining the story text, scene information, and atmosphere information, the pre-trained model can generate audio for reading the story text aloud, generate scene simulation audio based on the scene information, and generate background music based on the atmosphere information. Here, the scene simulation audio is used to simulate environmental sounds that match the scene information, and the background music is used to create the atmosphere described by the atmosphere information. As an example, suppose the pre-trained model analyzes the image of the target reading page and obtains scene information as a seaside. Since the environmental sound that matches the seaside is the sound of waves, the pre-trained model can generate simulated wave sounds as scene simulation audio. Subsequently, the accompanying reading device 100 can also use a speech synthesis model to synthesize the text reading audio with one or more of the scene simulation audio and background music to obtain a more realistic and vivid mixed audio as accompanying reading audio.

[0036] The above combination Figure 1 The described reading companion device 100 can generate accompanying audio based on the image of the current reading page of any reading material and using a pre-trained model. By playing the accompanying audio, it can provide accompaniment to any reading material, exhibiting good compatibility. This disclosed reading companion device does not limit whether the reading material is in a specially designed format or pre-stored, reducing production and purchase costs and saving storage resources. Furthermore, in some embodiments, the accompanying audio generated by the pre-trained model is not simply text-to-speech audio, but a mixture of text-to-speech audio and scene simulation audio, providing listeners with an immersive reading experience and helping to enhance the attractiveness of the accompanying audio to children, thereby increasing product stickiness.

[0037] Furthermore, considering that the children served by the reading companion device are generally younger, they have a higher affinity and recognition for their favorite animated characters. If the reading companion could mimic the voices of these animated characters, it would be more appealing to younger children, resulting in a better reading companion effect. Based on this, in one implementation scenario, a pre-trained model can identify multiple animated characters that a child likes based on their age, gender, and / or historical reading companion records, and determine the voices of these characters as multiple recommended voices. Next, feature analysis is performed on the voices of these multiple animated characters to obtain their vocal characteristics (e.g., timbre, pitch, and volume) in order to mimic their voices and generate reading companion audio.

[0038] Based on this, the reading companion device 100 can further perform the following operations to generate reading companion audio: play a sound selection prompt audio through the audio playback device 105; collect the sound selection result based on the sound selection prompt audio through the audio acquisition device 104, and obtain the target sound features based on the sound selection result; and generate text reading audio as reading companion audio using a pre-trained model based on the image and the target sound features.

[0039] Here, the audio prompt for voice selection is used to guide the user to choose a preferred voice from multiple recommended voices. Considering the young age of the children served by the reading companion device, the audio prompt for voice selection should use a gentle and clear description to enhance the children's reading experience. As an example, the audio prompt for voice selection could be, "Would you like to hear A tell the story, or would you like to hear B tell the story?" Accordingly, the child's voice selection result is also in audio form. Therefore, before obtaining the target voice features based on the voice selection result, the reading companion device 100 can first perform speech recognition on the audio form of the voice selection result to obtain the selection result text, and then obtain the target voice features based on the selection result text.

[0040] Subsequently, guided by the target voice features, the pre-trained model can mimic the voice of the animated character corresponding to the target voice features to generate audio readings of text. This can give children the feeling that a familiar and friendly animated character is telling them a story, thus increasing the model's appeal to children.

[0041] Figure 2 This is an exemplary flowchart illustrating the operation 200 performed by the companion reading device according to an embodiment of this application. Figure 2 As shown, in step S201, a pre-trained model can be used to generate guided audio. In practice, the pre-trained model can generate guided audio based on the aforementioned accompanying reading data. The guided audio is used to guide children to think about and understand the content of the current reading page, and its content can be any one or more of the following: open-ended questions, true / false questions, and multiple-choice questions.

[0042] Next, in step S202, the guided reading audio can be played through the audio playback device 105, and the child's response can be awaited. Step S202 is typically executed after the aforementioned guided reading audio has finished playing. Next, in step S203, if the response based on the guided reading audio is collected by the audio acquisition device 104 within a preset time, a response evaluation audio can be generated using a pre-trained model based on the response. Then, in step S204, the response evaluation audio can be played through the audio playback device 105. It is understood that those skilled in the art can select a specific value for the preset time according to actual needs, and this disclosure does not limit this selection.

[0043] Here, the child's response is in audio format. When generating the response evaluation audio based on the response, the pre-trained model can first perform speech recognition on the response to obtain the response text data. Then, based on the response text data, the pre-trained model can generate the response evaluation audio. The response evaluation audio is used to evaluate the child's response from one or more of the following aspects: whether the child understood the question, whether the response was timely, whether the response was complete, whether the response was correct, and whether the imagination was reasonable. Understandably, to enhance children's enthusiasm for reading, the response evaluation audio should use a friendly, encouraging, and praising tone, avoiding a harsh, critical, and blaming tone.

[0044] Furthermore, in step S205, if no response is collected by the audio acquisition device 104 within the preset time period, this indicates that the child may be unwilling or unable to answer. In this case, a pre-trained model can be used to generate reference audio, that is, to provide reference answers to the questions in the guided reading audio. Next, in step S206, the reference audio can be played through the audio playback device 105, and the number of guided readings can be set to 0. In addition, similar to the aforementioned response evaluation audio, the reference audio should also use a friendly, encouraging, and praising tone, avoiding a harsh, critical, and blaming tone.

[0045] The aforementioned number of guided questions is used to limit the number of times guided questions can be posed to a child, in order to avoid excessive questioning that could cause the child to feel aversion or resistance. When the number of guided questions is set to 0 or decreases to 0, it indicates that the number of guided questions for the child has been exhausted, and guided questioning should be stopped. It is understood that those skilled in the art can select a specific value for the number of guided questions according to actual needs, or the guardians of the child may be given the permission to set a specific value for the number of guided questions, so that the guardians can set a personalized number of guided questions for the child according to actual needs. Therefore, this disclosure does not specifically limit the specific value of the number of guided questions.

[0046] Based on this, when using a pre-trained model to generate guided reading audio, it's possible to first determine if the number of guided reading attempts is not zero. If the number of attempts is not zero, the pre-trained model can be used to generate guided reading audio, and the number of attempts is decremented by 1. Conversely, if the number of attempts is zero, guided reading audio is not generated; instead, the pre-trained model generates summary and evaluation audio, which is then played through the audio playback device 105. The summary and evaluation audio is used to provide an overall assessment of the child's reading ability. A child's reading ability can be reflected through one or more of the following: expression ability, comprehension ability, memory ability, and imagination ability. In practice, the pre-trained model can generate the summary and evaluation audio based on the accompanying reading audio, previous guided reading audio, previous response results, and evaluations of previous response results.

[0047] The aforementioned expressive ability is primarily evaluated through the results of each response. Specifically, it can be assessed by whether the description in the response is complete, the logic is clear, and the word choice is correct. Comprehension ability is primarily evaluated through the response, the guided audio, and the accompanying audio. Specifically, it can be assessed by whether the response is a response to the guided audio, whether the response matches the content of the accompanying audio, and whether the response is correct. Memory ability is primarily evaluated through the response and the accompanying audio. Specifically, it can be assessed by whether the information described in the response, directly from the accompanying audio, is correct. Imagination ability is primarily evaluated through the content of the response and the guided audio. Specifically, it can be assessed by whether the response is based on the imagination in the guided audio and whether the imagination is reasonable. It is understood that, similar to the aforementioned response evaluation audio and reference audio, the summary evaluation audio should also use a friendly, encouraging, and praising tone, avoiding a harsh, critical, and blaming tone.

[0048] As can be seen from the preceding description, combined with Figure 1 and Figure 2 All the operations described are performed by the reading companion device, which undoubtedly increases the computational burden on the device and places higher demands on its hardware. To balance the device load, in practical applications, some operations that directly interact with the child can be deployed on the reading companion device, while others can be deployed on the server side, thus forming a reading companion system. For ease of understanding, the following will combine... Figure 3 The accompanying reading method 300 based on the accompanying reading system is described in detail.

[0049] like Figure 3 As shown, in step S301, the reading companion device can capture an image of the current reading page using an image acquisition device. Next, in step S302, the reading companion device can send a request to the server to obtain accompanying audio, carrying the image of the current reading page. This allows the server to generate accompanying audio based on the request and send the audio back to the reading companion device.

[0050] Correspondingly, in step S303, the server can receive a request to obtain accompanying audio and can parse the request to obtain an image of the current reading page. Then, in step S304, the server can generate text-to-speech audio as accompanying audio based on the image using a pre-trained model.

[0051] After generating the accompanying audio, in step S305, the server can send the accompanying audio to the accompanying device. Correspondingly, in step S306, the accompanying device can receive the accompanying audio. Further, in step S307, the accompanying device can also play the accompanying audio through an audio playback device to achieve accompanying reading of the current reading page.

[0052] The above combination Figure 3 A companion reading method 300 based on a companion reading system is described. It is understood that the description of the various embodiments in this disclosure emphasizes the differences between the embodiments, while the similarities or corresponding parts can be referred to each other. For the sake of brevity, this disclosure will not elaborate further.

[0053] Based on the preceding description, it can be seen that, in the above combination Figure 3 The pre-trained model in the described reading companion method 300 is a large model. However, in some implementation scenarios, the pre-trained model can also be a combination of two large models (i.e., a first model and a second model). In this scenario, the first model and the second model can be deployed on the same server, or, to enhance the fault tolerance of the reading companion system, the first model and the second model can be distributed and deployed on different servers (a first server and a second server) to form a distributed reading companion system. For ease of understanding, the following will combine... Figure 4 The accompanying reading method 400 based on the distributed accompanying reading system is described in detail.

[0054] like Figure 4 In step S401, the reading companion device can acquire an image of the current reading page using an image acquisition device. Next, in step S402, the reading companion device can send a request to the first server to obtain reading companion data, carrying the image of the current reading page. The first server can then generate reading companion data based on this request and send the data back to the reading companion device. Correspondingly, in step S403, the first server can receive the request to obtain reading companion data and parse it to obtain the image of the current reading page. Next, in step S404, the first server can generate reading companion data based on the image using a first model. This reading companion data may include story text and scene information.

[0055] After generating the accompanying reading data, in step S405, the first server can send the accompanying reading data to the accompanying reading device. Correspondingly, in step S406, the accompanying reading device can receive the accompanying reading data. Further, in step S407, the accompanying reading device can send a request to the second server to obtain accompanying audio carrying the accompanying reading data, so that the second server generates accompanying audio based on the request and feeds back the accompanying audio to the accompanying reading device. Correspondingly, in step S408, the second server can receive the request to obtain accompanying audio and can parse the request to obtain the accompanying reading data it carries. Then, in step S409, the second server can use the second model to generate a mixed audio of text reading audio and scene simulation audio as the accompanying audio based on the accompanying reading data.

[0056] After generating the accompanying audio, in step S410, the second server can send the accompanying audio to the accompanying device. Correspondingly, in step S411, the accompanying device can receive the accompanying audio. Further, in step S412, the accompanying device can also play the accompanying audio through an audio playback device to achieve accompanying reading of the current reading page.

[0057] The above combination Figure 4 A companion reading method 400 based on a distributed companion reading system is described. It is understood that the description of the various embodiments in this disclosure emphasizes the differences between the embodiments, while the similarities or corresponding parts can be referred to mutually. For the sake of brevity, this disclosure will not elaborate further.

[0058] Based on the foregoing description in conjunction with the accompanying drawings, those skilled in the art will understand that the embodiments of this application can also be implemented by software programs. Therefore, this application also provides a computer-readable storage medium. This computer-readable storage medium stores program instructions for reading aloud, which can be used to implement the embodiments disclosed herein. Figure 1 and Figure 2 The operations performed by the described reading companion device.

[0059] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0060] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

[0061] The collection and acquisition of various data in this application comply with relevant laws and regulations and are authorized by the data providers. Any organization or individual that needs to obtain external data shall obtain authorization in accordance with the law and ensure data security, and shall not illegally collect, use, process, or transmit unauthorized or unprotected data, nor shall it illegally buy, sell, provide, or disclose unauthorized or unprotected data.

Claims

1. A reading companion device, characterized in that, include: Image acquisition device, audio acquisition device, audio playback device, processor and memory; The memory stores program instructions for companion reading. When the processor executes the program instructions, the companion reading device performs the following operations: In response to the detection of a start-reading signal, the image acquisition device acquires an image of the currently reading page. Based on the image, a pre-trained model is used to generate text-to-speech audio as accompanying audio. as well as The accompanying audio is played through the audio playback device to provide accompaniment reading for the current reading page.

2. The companion reading device according to claim 1, characterized in that, When the program instructions are executed by the processor, the companion reading device further performs the following operations to determine that a companion reading start signal has been detected: If the audio content acquired by the audio acquisition device contains a preset prompt word, it is determined that the start-reading signal has been detected, wherein the preset prompt word is used to prompt the reading device to start reading; or If the sensing value detected by the preset sensor meets the preset conditions, it is determined that the start reading signal has been detected.

3. The companion reading device according to claim 1 or 2, characterized in that, When the program instructions are executed by the processor, the accompanying reading device further performs the following operations to generate accompanying audio: The audio playback device plays a sound selection prompt audio; The audio acquisition device acquires the sound selection result based on the sound selection prompt audio feedback, and obtains the target sound characteristics based on the sound selection result; as well as Based on the image and the target sound features, the pre-trained model is used to generate text reading audio as accompanying audio.

4. The companion reading device according to claim 3, characterized in that, When the program instructions are executed by the processor, the accompanying reading device further performs the following operations to generate accompanying audio: The pre-trained model is used to generate scene simulation audio; Based on the text-to-speech audio and the scene-simulated audio, an audio synthesis model is used to synthesize audio to obtain a mixed audio as the accompanying audio.

5. The companion reading device according to claim 4, characterized in that, When the program instructions are executed by the processor, the accompanying reading device further performs the following operations to provide accompanying reading for the current reading page: The pre-trained model is used to generate the audio guide. After the accompanying audio has finished playing, the guiding audio is played through the audio playback device.

6. The companion reading device according to claim 5, characterized in that, When the program instructions are executed by the processor, the accompanying reading device further performs the following operations to provide accompanying reading for the current reading page: In response to the acquisition of a response result based on the guided audio by the audio acquisition device within a preset time period, the response evaluation audio is generated using the pre-trained model based on the response result; The audio of the response evaluation is played through the audio playback device.

7. The companion reading device according to claim 6, characterized in that, When the program instructions are executed by the processor, the accompanying reading device further performs the following operations to provide accompanying reading for the current reading page: If no response result based on the guided audio feedback is collected by the audio acquisition device within a preset time period, reference audio is generated using the pre-trained model; The reference audio is played through the audio playback device, and the number of guided readings is set to 0.

8. The companion reading device according to claim 7, characterized in that, When the program instructions are executed by the processor, the accompanying reading device further performs the following operations to generate guided audio using the pre-trained model: Determine if the number of guided readings is not 0; If the number of guided readings is not 0, the pre-trained model is used to generate guided audio, and the number of guided readings is decremented by 1. as well as When the number of guided readings is 0, the pre-trained model is used to generate a summary and evaluation audio. The summary and evaluation audio is played through the audio playback device.

9. A computer-readable storage medium storing computer-readable program instructions for reading aloud, which, when executed by a processor, cause the operation performed by the reading aloud device according to any one of claims 1-8 to be implemented.

Images