Auxiliary reading method and device, electronic equipment, storage medium and computer program product

Abstract

The invention provides an auxiliary reading method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: monitoring whether an auxiliary reading task pointed by an auxiliary reading instruction is completed or not through a monitoring model, switching the monitoring model into a running model under the condition that the task is not completed, and returning to execute a text generation step. The process of generating the to-be-read text corresponding to the real-time to-be-read picture by the monitoring model can be parallel to the process of generating the to-be-read text corresponding to the to-be-read picture by the running model, so that a user does not need to wait for the generation of the to-be-read text corresponding to the real-time to-be-read picture. And after the running model generates the to-be-read text corresponding to the to-be-read picture, the running model serves as the monitoring model, and the task monitoring step is executed, so that whether the auxiliary reading task pointed by the auxiliary reading instruction is completed or not can be continuously monitored, the reading continuity is ensured, the waiting time of a user is shortened, and the interaction fluency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an auxiliary reading method, device, electronic device, storage medium and computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of large models (such as BERT), many assisted reading products have been able to realize text-based story creation, dialogue generation, and assisted reading of picture book scenes.

[0003] At present, most large models generate the text to be read based on user instructions and the picture to be read corresponding to the current page of the picture book, and play the audio file corresponding to the text to be read by voice to achieve assisted reading. However, during the process of the large model playing the audio file corresponding to the text to be read, if the user turns to the next page of the picture book and needs to assist in reading the next page, the user needs to issue a new user instruction for the next page so that the large model can regenerate the corresponding text to be read based on the new user instruction and the picture to be read corresponding to the next page. In this method, when the user issues a new user instruction, the continuity of reading will be interrupted, affecting the reading immersion and reducing the user experience. Summary of the invention

[0004] The present invention provides an auxiliary reading method, device, electronic device, storage medium and computer program product to solve the defects in the prior art.

[0005] The present invention provides an auxiliary reading method, comprising the following steps: The running model executes a text generation step, wherein the text generation step includes generating a to-be-read text based on the auxiliary reading instruction and the to-be-read picture, and storing the to-be-read text in a cache; In the process of the running model executing the text generation step, the monitoring model simultaneously executes the task monitoring step, wherein the task monitoring step includes monitoring whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time to-be-read picture and all the to-be-read texts in the cache; If not, the real-time to-be-read picture is used as the to-be-read picture and the monitoring model is used as the running model, and the text generation step is returned to be executed. Meanwhile, the running model is used as the monitoring model, and the task monitoring step is returned to be executed; Wherein, during the execution of the text generation step and the task monitoring step, the audio file corresponding to the to-be-read text in the cache is synchronously played to assist reading.

[0006] According to an auxiliary reading method provided by the present invention, based on the auxiliary reading instruction, the real-time picture to be read and all the text to be read in the cache, monitoring whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction includes: Based on the auxiliary reading instruction, determine a historical to-be-read text from all to-be-read texts in the cache, wherein the timestamp of the to-be-read picture corresponding to the historical to-be-read text is earlier than the timestamp of the real-time to-be-read picture; Based on the semantic information of the real-time to-be-read pictures and the semantic information of each historical to-be-read text, it is monitored whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction.

[0007] According to an auxiliary reading method provided by the present invention, based on the semantic information of the real-time image to be read and the semantic information of each historical text to be read, monitoring whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction includes: Determining the semantic similarity between the real-time image to be read and each historical text to be read based on the semantic information of the real-time image to be read and the semantic information of each historical text to be read; If the semantic similarity between the real-time to-be-read picture and each historical to-be-read text is less than a threshold, it is determined that the operation model has not completed the auxiliary reading task indicated by the auxiliary reading instruction; If the semantic similarity between the real-time to-be-read picture and any historical to-be-read text is greater than or equal to the threshold, it is determined that the running model completes the auxiliary reading task indicated by the auxiliary reading instruction.

[0008] According to an auxiliary reading method provided by the present invention, the method generates a text to be read based on an auxiliary reading instruction and a picture to be read, including: Performing image recognition on the picture to be read to obtain multiple image elements; Performing text recognition on the to-be-read picture to obtain a plurality of text elements; The to-be-read text is generated based on the auxiliary reading instruction, the plurality of image elements, and the plurality of text elements.

[0009] According to an auxiliary reading method provided by the present invention, the generating of the to-be-read text based on the auxiliary reading instruction, the plurality of image elements and the plurality of text elements comprises: Based on the auxiliary reading instruction, determining a target image element from the plurality of image elements, and determining a target text element from the plurality of text elements; The to-be-read text is generated based on the target image element and the target text element.

[0010] According to an auxiliary reading method provided by the present invention, after determining that the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction, it also includes: It is detected whether all the to-be-read texts in the cache have been played. If not, the monitoring model returns to execute the task monitoring step at a preset time interval.

[0011] According to an auxiliary reading method provided by the present invention, the step of determining the audio file corresponding to the to-be-read text in the cache comprises: Converting the newly added to-be-read text in the cache into a phoneme sequence in real time, and generating a speech signal based on the phoneme sequence; The voice signal is converted into an audio file corresponding to the text to be read in the buffer.

[0012] The present invention also provides an auxiliary reading device, comprising the following modules: An operation unit, configured to operate the model to execute a text generation step, wherein the text generation step includes generating a to-be-read text based on the auxiliary reading instruction and the to-be-read picture, and storing the to-be-read text in a cache; A monitoring unit, used for monitoring the execution of a task monitoring step by the model during the execution of the text generation step by the running model, wherein the task monitoring step includes monitoring whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time picture to be read, and all the text to be read in the cache; If not, the real-time to-be-read picture is used as the to-be-read picture and the monitoring model is used as the running model, and the text generation step is returned to be executed. Meanwhile, the running model is used as the monitoring model, and the task monitoring step is returned to be executed; Wherein, during the execution of the text generation step and the task monitoring step, the audio file corresponding to the to-be-read text in the cache is synchronously played to assist reading.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned auxiliary reading methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the auxiliary reading method described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned auxiliary reading methods.

[0016] The auxiliary reading method, device, electronic device, storage medium and computer program product provided by the present invention monitor whether the auxiliary reading task pointed to by the auxiliary reading instruction is completed through the monitoring model, and when the task is not completed, the monitoring model is switched to the running model, and the text generation step is returned to be executed. Since the process of the monitoring model generating the to-be-read text corresponding to the real-time to-be-read picture can be parallel to the process of the running model generating the to-be-read text corresponding to the to-be-read picture, the user does not need to wait for the generation of the to-be-read text corresponding to the real-time to-be-read picture. After the running model generates the to-be-read text corresponding to the to-be-read picture, the running model, as a monitoring model, returns to the task monitoring step, so that it can continue to monitor whether the auxiliary reading task pointed to by the auxiliary reading instruction is completed until the auxiliary reading task pointed to by the auxiliary reading instruction is completed. In addition, in the process of executing the text generation step and the task monitoring step, the audio file corresponding to the to-be-read text in the cache is played synchronously to perform auxiliary reading, without waiting for the completion of the auxiliary reading task pointed to by the auxiliary reading instruction, and then playing the audio file corresponding to the to-be-read text, so as to achieve the synchronous playback of the audio file in the process of generating the to-be-read text, reducing the user waiting time and improving the fluency of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the auxiliary reading method provided by the present invention.

[0019] Figure 2 It is a flowchart of the task monitoring method provided by the present invention.

[0020] Figure 3 It is a flowchart of the implementation of step 220 in the task monitoring method provided by the present invention.

[0021] Figure 4 It is a flow chart of the text generation method provided by the present invention.

[0022] Figure 5 It is a flowchart diagram of the implementation of step 430 in the text generation method provided by the present invention.

[0023] Figure 6 This is the second flow chart of the auxiliary reading method provided by the present invention.

[0024] Figure 7It is a structural schematic diagram of the auxiliary reading device provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] At present, most large models generate the text to be read based on user instructions and the picture to be read corresponding to the current page of the picture book, and play the audio file corresponding to the text to be read by voice to achieve assisted reading. However, during the process of the large model playing the audio file corresponding to the text to be read, if the user turns to the next page of the picture book and needs assisted reading of the next page, the user needs to issue a new user instruction for the next page so that the large model can regenerate the corresponding text to be read based on the new user instruction and the picture to be read corresponding to the next page. In this method, when the user issues a new user instruction, the continuity of reading will be interrupted, which may lead to a loss of coherent understanding of the storyline, thereby affecting the reading immersion and reducing the user experience.

[0028] In addition, after receiving new user instructions, the large model needs a certain amount of time to process and generate new text to be read and corresponding audio files. This response delay may cause users to feel impatient and affect their enthusiasm for reading.

[0029] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is one of the flow charts of the auxiliary reading method provided by the present invention. The executor of each step in the method can be an auxiliary reading device, which can be implemented by software and / or hardware. The device can be integrated in an electronic device, which can be a smart phone, a personal computer, a learning machine, a local server or a cloud server, and a server cluster with hardware devices of various operating systems.

[0030] It should be noted that the execution subject of the embodiment of the present invention may be, for example, a central processing unit (CPU) in an electronic device in terms of hardware, or may be, for example, a large model-related service in an electronic device in terms of software, without limitation. Figure 1 As shown, the method includes step 110 and step 120.

[0031] Step 110, running the model to execute a text generation step, the text generation step includes generating a to-be-read text based on the auxiliary reading instruction and the to-be-read picture, and storing the to-be-read text in a cache; Step 120: While the running model is executing the text generation step, the monitoring model is also executing the task monitoring step, wherein the task monitoring step includes monitoring whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time to-be-read pictures, and all the to-be-read texts in the cache; If not, the real-time to-be-read picture is used as the to-be-read picture and the monitoring model is used as the running model, and the text generation step is returned to be executed. Meanwhile, the running model is used as the monitoring model, and the task monitoring step is returned to be executed; In the process of executing the text generation step and the task monitoring step, the audio file corresponding to the text to be read in the cache is played synchronously to assist reading.

[0032] Specifically, the to-be-read picture refers to a picture that needs to be read that is detected by the running model, which can be a picture of a page in a picture book or a picture of a page in other books, and the embodiment of the present invention does not specifically limit this. Among them, the to-be-read picture can be a picture directly input by the user, or a picture acquired by an image acquisition device such as a scanner, a mobile phone, or a camera, and the embodiment of the present invention does not specifically limit this.

[0033] The real-time to-be-read picture refers to the picture that needs to be read detected in real time by the monitoring model. The real-time to-be-read picture may be the same as the to-be-read picture detected by the running model, or it may be different from the to-be-read picture detected by the running model. For example, if the user needs to read a picture book, the to-be-read picture detected by the running model at time T1 is the picture of page 1 of the picture book. At this time, the running model generates the to-be-read text corresponding to page 1. At time T2, the user turns to page 2 of the picture book. At this time, the to-be-read picture detected by the monitoring model is the picture of page 2 of the picture book. At this time, the to-be-read picture detected by the running model (page 1) is different from the to-be-read picture detected by the monitoring model (page 2).

[0034] Furthermore, if the real-time to-be-read picture detected by the monitoring model is different from the to-be-read picture detected by the running model, it indicates that the user may not have completed the auxiliary reading task pointed to by the auxiliary reading instruction. For example, if the auxiliary reading instruction is "Tell me about this story", the to-be-read picture detected by the running model is the first page of the story (the story has 10 pages in total), and after the user turns to page 2, the real-time to-be-read picture detected by the monitoring model is the second page of the story. Combined with the auxiliary reading instruction, it can be seen that the running model has not completed the auxiliary reading task pointed to by the auxiliary reading instruction (reading the story, a total of 10 pages).

[0035] The auxiliary reading instruction may be understood as a reading instruction of the user, and the auxiliary reading instruction may be a voice instruction sent by the user. For example, the auxiliary reading instruction may be “Tell me this story”.

[0036] In this regard, an embodiment of the present invention executes a text generation step by running the model to generate the text to be read for the picture to be read, and at the same time monitors the execution of the task monitoring step by the model to monitor whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction during the process of the running model generating the text to be read.

[0037] If it is determined that the running model has not completed the auxiliary reading task pointed to by the auxiliary reading instruction, if the running model is still used to continue to generate the to-be-read text corresponding to the real-time to-be-read picture, the running model needs to wait for the running model to generate the to-be-read text corresponding to the previous to-be-read picture before it can generate the to-be-read text corresponding to the real-time to-be-read picture. Generating the to-be-read text corresponding to the real-time to-be-read picture is time-consuming. At this time, after the user has listened to the to-be-read text corresponding to the previous to-be-read picture, he needs to wait for the running model to generate the to-be-read text corresponding to the picture to be read. During this period, there is an interruption in reading, which affects the user's immersion in reading.

[0038] In this regard, the embodiment of the present invention adopts a monitoring model to generate the to-be-read text corresponding to the real-time to-be-read picture. Since the process of the monitoring model generating the to-be-read text corresponding to the real-time to-be-read picture can be parallel to the process of the running model generating the to-be-read text corresponding to the to-be-read picture, the user does not need to wait for the generation of the to-be-read text corresponding to the real-time to-be-read picture. After the running model generates the to-be-read text corresponding to the to-be-read picture, the running model, as a monitoring model, returns to the execution task monitoring step, so that it can continue to monitor whether the auxiliary reading task pointed to by the auxiliary reading instruction is completed until the auxiliary reading task pointed to by the auxiliary reading instruction is completed.

[0039] Among them, the text generation step includes generating the text to be read based on the auxiliary reading instructions and the picture to be read, and storing the text to be read in the cache. The text to be read is used to describe the image information and text information in the picture to be read, that is, the text to be read can be the text generated after the image information and text information in the picture to be read are fused.

[0040] The task monitoring step includes monitoring whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time to-be-read picture and all the to-be-read texts in the cache. Among them, all the to-be-read texts in the cache can be understood as the execution record of the auxiliary reading task. If the semantic information expressed by the real-time to-be-read picture and any to-be-read text in the cache is highly similar, it indicates that the running model has generated the to-be-read text corresponding to the real-time to-be-read picture. At this time, it is not necessary to generate the to-be-read text corresponding to the real-time to-be-read picture, that is, the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction. If the semantic information expressed by the real-time to-be-read picture and all the to-be-read texts in the cache are different, it indicates that the running model has not generated the to-be-read text corresponding to the real-time to-be-read picture. At this time, it is necessary to generate the to-be-read text corresponding to the real-time to-be-read picture, that is, the running model has not completed the auxiliary reading task pointed to by the auxiliary reading instruction.

[0041] In addition, in order to further ensure the continuity of reading, during the execution of the text generation step and the task monitoring step, the audio file corresponding to the text to be read in the cache is synchronously played for auxiliary reading. That is to say, the embodiment of the present invention does not need to wait for the completion of the auxiliary reading task pointed to by the auxiliary reading instruction before playing the audio file corresponding to the text to be read, but can achieve synchronous playback of the audio file during the process of generating the text to be read, thereby reducing the user's waiting time and improving the fluency of interaction.

[0042] The auxiliary reading method provided by the embodiment of the present invention monitors whether the auxiliary reading task pointed to by the auxiliary reading instruction is completed through the monitoring model, and switches the monitoring model to the running model when the task is not completed, and returns to the execution text generation step. Since the process of the monitoring model generating the to-be-read text corresponding to the real-time to-be-read picture can be parallel to the process of the running model generating the to-be-read text corresponding to the to-be-read picture, the user does not need to wait for the generation of the to-be-read text corresponding to the real-time to-be-read picture. After the running model generates the to-be-read text corresponding to the to-be-read picture, the running model, as the monitoring model, returns to the execution task monitoring step, so that it can continue to monitor whether the auxiliary reading task pointed to by the auxiliary reading instruction is completed until the auxiliary reading task pointed to by the auxiliary reading instruction is completed. In addition, in the process of executing the text generation step and the task monitoring step, the audio file corresponding to the to-be-read text in the cache is played synchronously for auxiliary reading, without waiting for the completion of the auxiliary reading task pointed to by the auxiliary reading instruction, and then playing the audio file corresponding to the to-be-read text, so as to achieve the synchronous playback of the audio file in the process of generating the to-be-read text, reducing the user waiting time and improving the fluency of the interaction.

[0043] Based on any of the above embodiments, Figure 2 is a flow chart of the task monitoring method provided by the present invention, such as Figure 2As shown, based on the auxiliary reading instruction, the real-time to-be-read pictures, and all the to-be-read texts in the cache, the monitoring operation model is used to monitor whether the auxiliary reading task pointed to by the auxiliary reading instruction is completed, including: Step 210: Based on the auxiliary reading instruction, determine the historical text to be read from all the texts to be read in the cache, and the timestamp of the picture to be read corresponding to the historical text to be read is earlier than the timestamp of the real-time picture to be read; Step 220: Based on the semantic information of the real-time to-be-read pictures and the semantic information of each historical to-be-read text, monitor whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction.

[0044] Specifically, the assisted reading instruction can represent the scope requirement of the user's assisted reading. For example, if the assisted reading instruction is "Tell me about this story", it indicates that the scope of the user's assisted reading is "the whole story"; if the assisted reading instruction is "Tell me about the first 3 pages", it indicates that the scope of the user's assisted reading is "the first 3 pages".

[0045] Since the auxiliary reading instruction can represent the range requirement of the user's auxiliary reading, the to-be-read text within the range requirement can be determined in the cache based on the auxiliary reading instruction as the historical to-be-read text. For example, if the auxiliary reading instruction is "Tell me about this story", all the to-be-read texts in the cache can be regarded as historical to-be-read texts; if the auxiliary reading instruction is "Tell me about the first 3 pages", the to-be-read texts corresponding to page 3 and before in the cache can be regarded as historical to-be-read texts, and if there is a to-be-read text corresponding to page 4 in the cache, it will not be included in the historical to-be-read text. Among them, the historical to-be-read text can be understood as the to-be-read text that has been generated under the auxiliary reading range indicated by the auxiliary reading instruction, and the timestamp of the to-be-read picture corresponding to the historical to-be-read text is earlier than the timestamp of the real-time to-be-read picture.

[0046] In addition, the semantic information of the real-time to-be-read picture refers to the content and meaning extracted from the picture, and the semantic information may include objects, scenes, emotions, actions, etc. in the picture. The semantic information of each historical to-be-read text is used to represent the key information of each historical to-be-read text.

[0047] If the semantic information of the real-time to-be-read picture is highly similar to the semantic information of any historical to-be-read text, it indicates that the running model has generated the to-be-read text corresponding to the real-time to-be-read picture, and it is not necessary to generate the to-be-read text corresponding to the real-time to-be-read picture, that is, the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction. If the semantic information of the real-time to-be-read picture is different from the semantic information of each historical to-be-read text, it indicates that the running model has not generated the to-be-read text corresponding to the real-time to-be-read picture, and it is necessary to generate the to-be-read text corresponding to the real-time to-be-read picture, that is, the running model has not completed the auxiliary reading task pointed to by the auxiliary reading instruction.

[0048] Based on any of the above embodiments, Figure 3 is a flow chart of an implementation of step 220 in the task monitoring method provided by the present invention, such as Figure 3 As shown, step 220 monitors whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction based on the semantic information of the real-time to-be-read picture and the semantic information of each historical to-be-read text, including: Step 221: Determine the semantic similarity between the real-time image to be read and each historical text to be read based on the semantic information of the real-time image to be read and the semantic information of each historical text to be read; Step 222: If the semantic similarity between the real-time to-be-read picture and each historical to-be-read text is less than the threshold, it is determined that the running model has not completed the auxiliary reading task indicated by the auxiliary reading instruction; Step 223: If the semantic similarity between the real-time to-be-read image and any historical to-be-read text is greater than or equal to the threshold, it is determined that the running model completes the auxiliary reading task indicated by the auxiliary reading instruction.

[0049] Specifically, the semantic information of the real-time to-be-read picture can be represented by the picture features of the real-time to-be-read picture, and the semantic information of each historical to-be-read text can be represented by the semantic features of each historical to-be-read text. On this basis, the distance between the picture features and each semantic feature is calculated to obtain the semantic similarity between the real-time to-be-read picture and each historical to-be-read text. The distance here can be Euclidean distance, Manhattan distance, etc., which is not specifically limited in the embodiment of the present invention.

[0050] If the semantic similarity between the real-time picture to be read and the historical texts to be read is less than the threshold, it means that the semantic information of the real-time picture to be read is different from the semantic information of the historical texts to be read, that is, the running model has not generated the text to be read corresponding to the real-time picture to be read. At this time, it is necessary to generate the text to be read corresponding to the real-time picture to be read, that is, the running model has not completed the auxiliary reading task indicated by the auxiliary reading instruction.

[0051] If the semantic similarity between the real-time picture to be read and any historical text to be read is greater than or equal to the threshold, it indicates that the historical text to be read is the text to be read corresponding to the real-time picture to be read, that is, the running model has generated the text to be read corresponding to the real-time picture to be read, and there is no need to generate the text to be read corresponding to the real-time picture to be read, that is, the running model completes the auxiliary reading task indicated by the auxiliary reading instruction.

[0052] Based on any of the above embodiments, Figure 4 It is a flow chart of the text generation method provided by the present invention, such as Figure 4 As shown, based on the auxiliary reading instructions and the to-be-read pictures, the to-be-read text is generated, including: Step 410: Perform image recognition on the image to be read to obtain multiple image elements; Step 420: Perform text recognition on the image to be read to obtain multiple text elements; Step 430: Generate a to-be-read text based on the auxiliary reading instruction, the plurality of image elements, and the plurality of text elements.

[0053] Specifically, the picture to be read may include image content and text content. Image elements can be understood as key elements of the image content in the picture to be read, and image elements can include objects, scenes, colors, shapes, etc. Text elements can be understood as key elements of the text content in the picture to be read, and text elements can include key words, titles, etc.

[0054] In addition, the auxiliary reading instruction can also represent the user's auxiliary reading requirements. For example, if the auxiliary reading instruction is "briefly summarize the story", it means that the user needs to briefly listen to the general information of the story. If the auxiliary reading instruction is "combine the pictures and text content to describe the story in detail", it means that the user needs to explain the plot information of the story in detail.

[0055] If the user needs to briefly hear the general information of the story, a brief to-be-read text can be generated based on multiple image elements and multiple text elements to meet the user's needs. If the user needs a detailed explanation of the plot information of the story, a detailed to-be-read text can be generated based on multiple image elements and multiple text elements.

[0056] It can be seen that the embodiments of the present invention can assist in meeting the user needs pointed out by the reading instructions, flexibly generate corresponding texts to be read, and improve user satisfaction.

[0057] Based on any of the above embodiments, Figure 5 is a flow chart of an implementation of step 430 in the text generation method provided by the present invention, such as Figure 5 As shown, step 430 generates a to-be-read text based on the auxiliary reading instruction, the plurality of image elements, and the plurality of text elements, including: Step 431: determining a target image element from a plurality of image elements and determining a target text element from a plurality of text elements based on the auxiliary reading instruction; Step 432: Generate the text to be read based on the target image elements and the target text elements.

[0058] Specifically, the target image element can be understood as the element most relevant to the user's needs among multiple image elements, and the target text element can be understood as the element most relevant to the user's needs among multiple text elements. Since the auxiliary reading instruction can represent the user's needs, the target image element can be determined from multiple image elements and the target text element can be determined from multiple text elements based on the auxiliary reading instruction.

[0059] For example, if the auxiliary reading instruction is "What kind of dog is this", it means that the user is paying attention to "dog", so the target image element may be "dog's color", "dog's shape", etc., and the target text element may be a text element used to describe the dog, such as the target text element may be "This is a retriever".

[0060] After obtaining the target image element and the target text element, the target image element and the target text element can be integrated to generate the corresponding to-be-read text. For example, if the target image element is "golden retriever" and the target text element can be "this is a retriever", the above elements can be integrated to obtain the to-be-read text "this is a golden retriever".

[0061] It can be seen that the target image element and the target text element in the embodiment of the present invention can achieve information complementarity, thereby making the generated text to be read more accurate and detailed.

[0062] Based on any of the above embodiments, after determining that the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction, the method further includes: Check whether all the to-be-read texts in the cache have been played. If not, the monitoring model returns to the task monitoring step at a preset time interval.

[0063] Specifically, if there are unplayed texts to be read in the cache, the auxiliary reading task pointed to by the auxiliary reading instruction can only be considered completed after all the texts to be read are played. During the waiting period, the user may update the current picture to be read, such as turning to the next page of the picture book. At this time, in order to further accurately determine whether the auxiliary reading task pointed to by the auxiliary reading instruction is completed, the monitoring model returns to the task monitoring step at a preset time interval until all the texts to be read in the cache have been played.

[0064] If the monitoring model detects a new picture to be read, it further determines whether the auxiliary reading task pointed to by the auxiliary reading instruction is completed. If not, the real-time picture to be read is used as the picture to be read and the monitoring model is used as the running model, and the text generation step is returned to execute. At the same time, the running model is used as the monitoring model, and the task monitoring step is returned to execute. The preset time interval can be 2s.

[0065] Based on any of the above embodiments, the step of determining the audio file corresponding to the to-be-read text in the cache includes: Convert the newly added text to be read in the cache into a phoneme sequence in real time, and generate a speech signal based on the phoneme sequence; Convert the speech signal into an audio file corresponding to the text to be read in the buffer.

[0066] Specifically, the phoneme sequence refers to the basic sound units into which the newly added text to be read is decomposed. For example, the word "cat" can be decomposed into phonemes / k / , / æ / , / t / . Then, the phoneme sequence can be converted into a speech signal through synthesis technology. Among them, the synthesis technology can be rule-based synthesis, splicing synthesis, deep learning-based synthesis, etc.

[0067] After acquiring the voice signal, the voice signal is converted into an audio file, which is the audio file corresponding to the text to be read in the cache. The timestamp of the voice signal is the same as the timestamp of the corresponding picture to be read, and when playing the corresponding audio file, it is played in the order of the timestamps of the pictures to be read.

[0068] Based on any of the above embodiments, Figure 6 FIG. 2 is a flow chart of the auxiliary reading method provided by the present invention. Figure 6 As shown, the method includes: After receiving the auxiliary reading instruction, the model is run to extract the to-be-read picture from the video, and the model is run to perform the text generation step. The text generation step includes generating the to-be-read text (Once upon a time, there was a little rabbit and a big rabbit in the forest, and they...) based on the auxiliary reading instruction and the to-be-read picture, and storing the to-be-read text in the cache.

[0069] While the model is running to execute the text generation step, the monitoring model is also monitoring the task monitoring step, which includes monitoring whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time picture to be read, and all the text to be read in the cache. Among them, the real-time picture to be read is the real-time picture extracted from the video by the monitoring model.

[0070] At this time, the monitoring model detects that the real-time picture to be read is different from the picture to be read, and if not, the real-time picture to be read is used as the picture to be read and the monitoring model is used as the running model, and the text generation step is returned to execute. At this time, the monitoring model generates the text to be read (One day, the little rabbit and the big rabbit went out to play...), and the running model is used as the monitoring model, and the task monitoring step is returned to execute.

[0071] The generated text to be read is stored in the cache, and the text to be read in the cache is subjected to streaming speech synthesis to obtain the corresponding audio file, and the audio file is played synchronously to assist reading.

[0072] The auxiliary reading device provided by the present invention is described below. The auxiliary reading device described below and the auxiliary reading method described above can be referred to each other.

[0073] Based on any of the above embodiments, Figure 7 is a schematic diagram of the structure of the auxiliary reading device provided by the present invention, such as Figure 7As shown, the device comprises: An operation unit 710 is used to operate the model to execute a text generation step, wherein the text generation step includes generating a to-be-read text based on the auxiliary reading instruction and the to-be-read picture, and storing the to-be-read text in a cache; The monitoring unit 720 is used to monitor the execution of the task monitoring step by the model while the model is executing the text generation step. The task monitoring step includes monitoring whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time to-be-read picture and all the to-be-read texts in the cache; If not, the real-time to-be-read picture is used as the to-be-read picture and the monitoring model is used as the running model, and the text generation step is returned to be executed. Meanwhile, the running model is used as the monitoring model, and the task monitoring step is returned to be executed; In the process of executing the text generation step and the task monitoring step, the audio file corresponding to the text to be read in the cache is played synchronously to assist reading.

[0074] Based on any of the above embodiments, based on the auxiliary reading instruction, the real-time picture to be read, and all the text to be read in the cache, monitoring whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction includes: Based on the auxiliary reading instruction, determine the historical to-be-read text from all the to-be-read texts in the cache, and the timestamp of the to-be-read picture corresponding to the historical to-be-read text is earlier than the timestamp of the real-time to-be-read picture; Based on the semantic information of the real-time to-be-read pictures and the semantic information of each historical to-be-read text, monitor whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction.

[0075] Based on any of the above embodiments, based on the semantic information of the real-time to-be-read picture and the semantic information of each historical to-be-read text, monitoring whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction includes: Determine the semantic similarity between the real-time image to be read and the historical texts to be read based on the semantic information of the real-time image to be read and the semantic information of the historical texts to be read; If the semantic similarity between the real-time to-be-read image and each historical to-be-read text is less than the threshold, it is determined that the running model has not completed the auxiliary reading task indicated by the auxiliary reading instruction; If the semantic similarity between the real-time to-be-read image and any historical to-be-read text is greater than or equal to a threshold, it is determined that the running model completes the auxiliary reading task indicated by the auxiliary reading instruction.

[0076] Based on any of the above embodiments, generating a to-be-read text based on the auxiliary reading instruction and the to-be-read picture includes: Perform image recognition on the image to be read to obtain multiple image elements; Perform text recognition on the image to be read to obtain multiple text elements; A to-be-read text is generated based on the auxiliary reading instruction, the plurality of image elements, and the plurality of text elements.

[0077] Based on any of the above embodiments, generating a to-be-read text based on the auxiliary reading instruction, the plurality of image elements, and the plurality of text elements includes: Based on the auxiliary reading instruction, determining a target image element from a plurality of image elements, and determining a target text element from a plurality of text elements; Based on the target image elements and the target text elements, a text to be read is generated.

[0078] Based on any of the above embodiments, after determining that the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction, the method further includes: Check whether all the to-be-read texts in the cache have been played. If not, the monitoring model returns to the task monitoring step at a preset time interval.

[0079] Based on any of the above embodiments, the step of determining the audio file corresponding to the to-be-read text in the cache includes: Convert the newly added text to be read in the cache into a phoneme sequence in real time, and generate a speech signal based on the phoneme sequence; Convert the speech signal into an audio file corresponding to the text to be read in the buffer.

[0080] Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the auxiliary reading method.

[0081] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0082] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the auxiliary reading methods provided by the above methods.

[0083] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the auxiliary reading methods provided by the above methods.

[0084] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0085] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assisting reading, characterized in that: include: The running model executes a text generation step, wherein the text generation step includes generating a to-be-read text based on the auxiliary reading instruction and the to-be-read picture, and storing the to-be-read text in a cache; In the process of the running model executing the text generation step, the monitoring model simultaneously executes the task monitoring step, wherein the task monitoring step includes monitoring whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time to-be-read picture and all the to-be-read texts in the cache; If not, the real-time to-be-read picture is used as the to-be-read picture and the monitoring model is used as the running model, and the text generation step is returned to be executed. Meanwhile, the running model is used as the monitoring model, and the task monitoring step is returned to be executed; Wherein, during the execution of the text generation step and the task monitoring step, the audio file corresponding to the to-be-read text in the cache is synchronously played to assist reading.

2. The auxiliary reading method according to claim 1, characterized in that: The monitoring of whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time picture to be read, and all the text to be read in the cache includes: Based on the auxiliary reading instruction, determine a historical to-be-read text from all to-be-read texts in the cache, wherein the timestamp of the to-be-read picture corresponding to the historical to-be-read text is earlier than the timestamp of the real-time to-be-read picture; Based on the semantic information of the real-time to-be-read pictures and the semantic information of each historical to-be-read text, it is monitored whether the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction.

3. The auxiliary reading method according to claim 2, characterized in that: The monitoring of whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction based on the semantic information of the real-time image to be read and the semantic information of each historical text to be read includes: Determining the semantic similarity between the real-time image to be read and each historical text to be read based on the semantic information of the real-time image to be read and the semantic information of each historical text to be read; If the semantic similarity between the real-time to-be-read picture and each historical to-be-read text is less than a threshold, it is determined that the operation model has not completed the auxiliary reading task indicated by the auxiliary reading instruction; If the semantic similarity between the real-time to-be-read picture and any historical to-be-read text is greater than or equal to the threshold, it is determined that the running model completes the auxiliary reading task indicated by the auxiliary reading instruction.

4. The auxiliary reading method according to any one of claims 1 to 3, characterized in that: The generating of the to-be-read text based on the auxiliary reading instruction and the to-be-read picture includes: Performing image recognition on the picture to be read to obtain multiple image elements; Performing text recognition on the to-be-read picture to obtain a plurality of text elements; The to-be-read text is generated based on the auxiliary reading instruction, the plurality of image elements, and the plurality of text elements.

5. The auxiliary reading method according to claim 4, characterized in that: The step of generating the to-be-read text based on the auxiliary reading instruction, the plurality of image elements, and the plurality of text elements includes: Based on the auxiliary reading instruction, determining a target image element from the plurality of image elements, and determining a target text element from the plurality of text elements; The to-be-read text is generated based on the target image element and the target text element.

6. The auxiliary reading method according to any one of claims 1 to 3, characterized in that: After determining that the running model completes the auxiliary reading task pointed to by the auxiliary reading instruction, the method further includes: It is detected whether all the to-be-read texts in the cache have been played. If not, the monitoring model returns to execute the task monitoring step at a preset time interval.

7. The auxiliary reading method according to any one of claims 1 to 3, characterized in that: The step of determining the audio file corresponding to the to-be-read text in the cache comprises: Converting the newly added to-be-read text in the cache into a phoneme sequence in real time, and generating a speech signal based on the phoneme sequence; The voice signal is converted into an audio file corresponding to the text to be read in the buffer.

8. An auxiliary reading device, characterized in that: include: An operation unit, configured to operate the model to execute a text generation step, wherein the text generation step includes generating a to-be-read text based on the auxiliary reading instruction and the to-be-read picture, and storing the to-be-read text in a cache; A monitoring unit, used for monitoring the execution of a task monitoring step by the model during the execution of the text generation step by the running model, wherein the task monitoring step includes monitoring whether the running model completes the auxiliary reading task indicated by the auxiliary reading instruction based on the auxiliary reading instruction, the real-time picture to be read, and all the text to be read in the cache; If not, the real-time to-be-read picture is used as the to-be-read picture and the monitoring model is used as the running model, and the text generation step is returned to be executed. Meanwhile, the running model is used as the monitoring model, and the task monitoring step is returned to be executed; Wherein, during the execution of the text generation step and the task monitoring step, the audio file corresponding to the to-be-read text in the cache is synchronously played to assist reading.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the auxiliary reading method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the auxiliary reading method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the auxiliary reading method according to any one of claims 1 to 7 is implemented.

Images